We investigate the stability properties of a collisionless Vlasov plasma in a weakly inhomogeneous magnetic field using non-modal stability analysis. This is an important topic in a physics of tokamak plasma rich in various types of instabilities. We consider a thin tokamak plasma in a Maxwellian equilibrium, subjected to a small arbitrary perturbation. Within the framework of kinetic theory, we demonstrate the emergence of short time scale algebraic instabilities evolving in a stable magnetized plasma. We show that the linearized governing operator (Vlasov operator) is non-normal leading to the transient growth of the perturbations on the time scale of several plasma periods that is subsequently followed by Landau damping. We calculate the first-order distribution function and the electric field and study the dependence of the transient growth characteristics on the magnetic field strength and perturbation parameters of the system. We compare our results with uniformly magnetized plasma and field-free Vlasov plasma.