In this paper, we introduced a comparative study of Sm-Tm-substituted Sr-Ba nanohexaferrites (NHFs), Sr0.5Ba0.5TmxSmxFe12-2xO19 with x = 0.00–0.05, manufactured via both citrate sol-gel auto-combustion and ultrasonication approaches. The phase formation of M-type hexaferrite (HF) for both compositions was confirmed by X-ray diffraction (XRD) powder pattern, Fourier-transform infrared (FT-IR) spectra, scanning and transmission electron microscopy (SEM and TEM) micrographs, energy dispersive X-ray (EDX) spectra, and elemental mappings. The magnetic properties at room temperature (RT) and low temperature (T = 10 K) were also investigated. M-H loops revealed ferrimagnetic nature for various prepared nanohexaferrites via sol-gel and ultrasonication routes. The Ms (saturation magnetization) and Mr (remanence) values increased with increasing Tm-Sm substituting contents. Ms and Mr reached their maximum values at x = 0.04 in the case of samples prepared using the sol-gel technique and at x = 0.03 for those prepared via ultrasonication route. M-H loops were very broad in samples prepared via ultrasonication route in comparison to those produced by means of the sol-gel approach, implying that the products synthesized via ultrasonication route have greater values of coercivity (Hc). The variations in Hc values with respect to Tm-Sm substitutions were governed by the evolutions in the magneto-crystalline anisotropy. Diffuse reflectance spectra (DRS) were employed to estimate the direct band gap of pristine and co-substituted Sr0.5Ba0.5Fe12O19 hexaferrites. Optical energy band gaps (Eg) of pristine samples were significantly tuned by co-substitution of Tm3+ and Sm3+ ions. Eg values of the Sr0.5Ba0.5Fe12O19 sample, which was synthesized using the sol-gel method, decreased almost linearly from 1.75 to 1.45 eV by increasing co-doped ion content. However, we observed a sharp drop from 1.85 eV to an average of 1.50 eV for the samples, which were synthesized using the ultrasonication approach.