We study an unconventional quantum Hall effect for the surface states of ultrathin Floquet topological insulators in a perpendicular magnetic field. The resulting band structure is modified by photon dressing and the topological property is governed by the low-energy dynamics of a single surface. An exchange of symmetric and antisymmetric surface states occurs by reversing the lights polarization. We find a novel quantum Hall state in which the zeroth Landau level undergoes a phase transition from a trivial insulator state, with Hall conductivity αyx = 0 at zero Fermi energy, to a Hall insulator state with αyx = e2/2h. These findings open new possibilities for experimentally realizing nontrivial quantum states and unusual quantum Hall plateaus at (±1/2,±3/2,±5/2, ...)e2/h. © 2016 IOP Publishing Ltd Printed in the UK.