All-polymer solar cells lag behind the state-of-the-art in small molecule nonfullerene acceptor (NFA) bulk heterojunction (BHJ) organic solar cells (OSCs) for reasons still unclear. Herein, the efficiency-limiting processes in all-polymer solar cells are investigated using blends of the common donor polymer PBDT-TS1 with different acceptor polymers, namely P2TPD[2F]T and P2TPDBT[2F]T. Combining data from steady-state optical spectroscopy and time-resolved photoluminescence, transient absorption, and time-delayed collection field experiments, provides not only a concise but also quantitative assessment of the losses due to limited photon absorption, geminate and nongeminate charge carrier recombination, field-dependent charge generation, and inefficient carrier extraction. Although both systems exhibit a similar charge separation efficiency in the absence of external bias, charge separation is significantly enhanced in P2TPDBT[2F]T-based blends when biased. Kinetic parameters obtained via pulsed laser spectroscopy are used to reproduce the experimentally measured device current–voltage (J–V) characteristics and indicate that low fill factors originate either from nongeminate recombination competing with charge extraction, or from a pronounced field dependence of charge generation, depending on the acceptor polymer. The methodology presented here is generic and can be used to quantify the loss processes in BHJ OSCs including both all-polymer and small molecule NFA systems.