Differential semblance optimization (DSO) is a popular method for the construction of a kinematically accurate velocity model for imaging applications. However, its gradient contains artifacts which affect the DSO performance. To resolve such a problem, a pseudo inverse operator under the assumption of laterally homogeneous background was used to generate the extended images for DSO. The laterally inhomogeneous version required extended images in the vertical offset direction, which is costly. We derive an extension of the approach to handle laterally inhomogeneous background media efficiently, and show that the term required for the laterally inhomogeneity treatment has a relatively small impact on the pseudo inverse operator for many cases.