The three-dimensional premixed H2-O2 detonation propagation in rectangular ducts is simulated using an in-house parallel detonation code based on the second-order space–time conservation element and solution element (CE/SE) scheme. The simulation reproduces three typical cellular structures by setting appropriate cross-sectional size and initial perturbation in square tubes. As the cross-sectional size decreases, critical cellular structures transforming the rectangular or diagonal mode into the spinning mode are obtained and discussed in the perspective of phase variation as well as decreasing of triple point lines. Furthermore, multiple cellular structures are observed through examples with typical aspect ratios. Utilizing the visualization of detailed three-dimensional structures, their formation mechanism is further analyzed.