Non-fullerene acceptor (NFA) based organic solar cells have outperformed fullerene-based devices, yet their photophysics is less well understood. Herein, changes in the donor polymer backbone side-chain substitution and backbone fluorination in benzodithiophene (BDT)-thiophene copolymers are linked to the photophysical processes and performance of bulk heterojunction (BHJ) solar cells, using ITIC as NFA. Increased geminate recombination is observed when the donor polymer is alkoxy-substituted in conjunction with faster non-geminate recombination of free charges, limiting both the short circuit current and device fill factor. In contrast, thienylsubstitution reduces geminate recombination, albeit non-geminate recombination remains significant, leading to improved short circuit current density, yet not fill factor. Only the combination of thienyl-substitution and polymer backbone fluorination yields both efficient charge separation and significantly reduced non-geminate recombination, resulting in fill factors (FFs) in excess of 60 %. Time-delayed collection field measurements ascertain that charge generation is field-independent in the thienyl-substituted donor polymer:ITIC systems, while weakly field dependent in the alkoxy-substitued polymer:ITIC blend, indicating the low FFs are primarily caused by non-geminate recombination. This work provides insight into the interplay of donor polymer structure, BHJ photophysics, and device performance for a prototypical NFA, namely ITIC. More specifically, it links the donor polymer chemical structure to quantifiable changes of kinetic parameters and the yield of individual processes in ITIC-based BHJ blends.