The formation of different types of structural polymorphs of poly(3-hexyl-thiophene) (P3HT) affects the performance of organic photovoltaic (OPV) devices that use thermally annealed P3HT:PCBM blend films as a photoactive layer. Here it is demonstrated that when densely packed and nondensely packed P3HT polymorphs coexist in the P3HT:PCBM layer, nongeminate charge recombination is fast; however, in a device nongeminate recombination is effectively overruled by efficient and fast charge carrier extraction. In stark contrast, when only a less densely packed P3HT polymorph is present in the blend, nongeminate charge recombination losses are less pronounced, and the charge carrier extraction efficiency is lower. The antagonistic nongeminate charge recombination and charge carrier extraction processes in these systems are monitored by time-delayed collection field (TDCF) and ultrafast transient absorption (TA) experiments. Furthermore, resonance Raman spectroscopy reveals that in the absence of the densely packed P3HT polymorph the energetic disorder present in the P3HT:PCBM blend is higher. High-resolution atomic force microscopy imaging further identifies pronounced differences in the layer morphology when the polymorph distribution varies between unimodal and bimodal. These results indicate that less densely packed P3HT polymorphs increase disorder and impede charge collection, leading to a reduction of the device fill factor.