Current-induced spin transfer torque and magnetization dynamics in the presence of spin diffusion in disordered magnetic textures is studied theoretically. We demonstrate using tight-binding calculations that weak, spin-conserving impurity scattering dramatically enhances the nonadiabaticity. To further explore this mechanism, a phenomenological drift-diffusion model for incoherent spin transport is investigated. We show that incoherent spin diffusion indeed produces an additional spatially dependent torque of the form ∼∇2[m×(u⋅∇)m]+ξ∇2[(u⋅∇)m], where m is the local magnetization direction, u is the direction of injected current, and ξ is a parameter characterizing the spin dynamics (precession, dephasing, and spin-flip). This torque, which scales as the inverse square of the domain wall width, only weakly enhances the longitudinal velocity of a transverse domain wall but significantly enhances the transverse velocity of vortex walls. The spatial-dependent spin transfer torque uncovered in this study is expected to have significant impact on the current-driven motion of abrupt two-dimensional textures such as vortices, skyrmions, and merons.