Magnetic nanowires have emerged as essential components for a broad range of applications. In many cases, a key property of these components is the switching field, which is studied as a function of the angle between the field and the nanowire. We found remarkable differences of up to 100% between the switching fields of different nanowires from the same fabrication batch. Our experimental results and micromagnetic simulations indicate that the nanowires exhibit a single domain behavior and that the switching mechanism includes vortex domain wall motion across the nanowire. The differences between the switching fields are attributed to different cross-sections of the nanowires, as found by electron microscopy. While a circular cross-section yields the smallest switching field values, any deviation from this shape results in an increase of the switching field. The shape of the nanowires' cross-sections is thus a critical parameter that has not been previously taken into account.