Priors are important for achieving proper posteriors with physically meaningful covariance structures for Gaussian random fields (GRFs) since the likelihood typically only provides limited information about the covariance structure under in-fill asymptotics. We extend the recent penalized complexity prior framework and develop a principled joint prior for the range and the marginal variance of one-dimensional, two-dimensional, and three-dimensional Matérn GRFs with fixed smoothness. The prior is weakly informative and penalizes complexity by shrinking the range toward infinity and the marginal variance toward zero. We propose guidelines for selecting the hyperparameters, and a simulation study shows that the new prior provides a principled alternative to reference priors that can leverage prior knowledge to achieve shorter credible intervals while maintaining good coverage. We extend the prior to a nonstationary GRF parameterized through local ranges and marginal standard deviations, and introduce a scheme for selecting the hyperparameters based on the coverage of the parameters when fitting simulated stationary data. The approach is applied to a dataset of annual precipitation in southern Norway and the scheme for selecting the hyperparameters leads to conservative estimates of nonstationarity and improved predictive performance over the stationary model. Supplementary materials for this article are available online.