The unknown parameters (variance, smoothness, and covariance length) of a spatial covariance function can be estimated by maximizing the joint Gaussian log-likelihood function. To overcome cubic complexity in the linear algebra, the discretized covariance function is approximated in the hierarchical (H-) matrix format. The H-matrix format has a log-linear computational cost and O(knlogn) storage, where the rank k is a small integer, and n is the number of locations. The H-matrix technique can approximate general covariance matrices (also inhomogeneous) discretized on a fairly general mesh that is not necessarily axes-parallel, and neither the covariance matrix itself nor its inverse has to be sparse. It is investigated how the H-matrix approximation error influences the estimated parameters. Numerical examples with Monte Carlo simulations, where the true values of the unknown parameters are given, and an application to soil moisture data with unknown parameters are presented. The C, C++ codes and data are freely available.