Air Gap Membrane distillation (AGMD) is a thermally driven separation process capable of treating challenging water types, but its low productivity is a major drawback. Membrane fouling is a common problem in many membrane treatment systems, which exacerbates AGMD’s low overall productivity. In this study, we investigated the direct application of low-power ultrasound (8–23 W), as an in-line cleaning and performance boosting technique for AGMD. Two different highly saline feedwaters, namely natural groundwater (3970 μS/cm) and RO reject stream water (12760 μS/cm) were treated using Polytetrafluoroethylene (PTFE) and polyvinylidene fluoride (PVDF) membranes. Theoretical calculations and experimental investigations are presented, showing that the applied ultrasonic power range only produced acoustic streaming effects that enhanced cleaning and mass transfer. Attenuated Total Reflection Fourier-Transform Infrared Spectroscopy (ATR FT-IR) analysis showed that ultrasound was capable of effectively removing silica and calcium scaling. Ultrasound application on a fouled membrane resulted in a 100% increase in the permeate flux. Cleaning effects accounted for around 30–50% of this increase and the remainder was attributed to mass transfer improvements. Contaminant rejection percentages were consistently high for all treatments (>99%), indicating that ultrasound did not deteriorate the membrane structure. Scanning Electron Microscopy (SEM) analysis of the membrane surface was used to confirm this observation. The images of the membrane surface demonstrated that ultrasound successfully cleaned the previously fouled membrane, with no signs of structural damage. The results of this study highlight the efficient and effective application of direct low power ultrasound for improving AGMD performance.