Unmanned aerial vehicles (UAVs) have been extensively utilized in various scenarios due to their flexible mobility and rapid on-demand deployment. It becomes necessary and urgent to examine the secrecy performance of the UAV communication systems due to the open characteristics of the transmission technologies employed in UAV systems. In this paper, we investigate the physical layer security of the UAV communication system, in which the ground base station sends secret messages to the UAV hovering in the air, while a random number of eavesdroppers are randomly located around the ground base station. Using the Poisson point process theory, we obtain the exact and approximate expressions of the cumulative distribution function for the signal-to-noise ratio of the ground-to-air link, and then the analytical expressions of the secrecy outage probability and the average secrecy rate are derived. Finally, the accuracy of the analytical results is validated by Monte Carlo simulation.