The reported power conversion efficiencies (PCEs) of nonfullerene acceptor (NFA) based organic photovoltaics (OPVs) now exceed 14% and 17% for single-junction and two-terminal tandem cells, respectively. However, increasing the PCE further requires an improved understanding of the factors limiting the device efficiency. Here, the efficiency limits of single-junction and two-terminal tandem NFA-based OPV cells are examined with the aid of a numerical device simulator that takes into account the optical properties of the active material(s), charge recombination effects, and the hole and electron mobilities in the active layer of the device. The simulations reveal that single-junction NFA OPVs can potentially reach PCE values in excess of 18% with mobility values readily achievable in existing material systems. Furthermore, it is found that balanced electron and hole mobilities of >10−3 cm2 V−1 s−1 in combination with low nongeminate recombination rate constants of 10−12 cm3 s−1 could lead to PCE values in excess of 20% and 25% for single-junction and two-terminal tandem OPV cells, respectively. This analysis provides the first tangible description of the practical performance targets and useful design rules for single-junction and tandem OPVs based on NFA materials, emphasizing the need for developing new material systems that combine these desired characteristics.