All-inorganic perovskite nanocrystals (NCs) have emerged as a class of low-cost and high-efficiency light-emitting materials. However, the structure and phase transition behavior of these NCs remain poorly understood. Here, we provide unambiguous evidence that all these properties are associated with structural defects in NCs. Using CsPbCl3 NCs as a model system, we find that the cubic subdomains in highly defective NCs gradually convert to the orthorhombic upon cooling room-temperature quasi-cubic NCs, whereas high-quality cousins, with mixed cubic and orthorhombic subdomains at room temperature, exhibit a significant resistance for such a phase transition. Car-Parrinello molecular dynamics simulations unveil the important role of point defects in triggering a stepwise, reversible phase transition of CsPbCl3 NCs. Importantly, we show that the defect-triggered phase transition also exists in other all-inorganic perovskite NCs. Our findings provide new insight into the structure and phase transition of CsPbX3 NCs and highlight the important role of defects in impacting these properties.