As the demand for medical cares has considerably expanded, the issue of managing patient flow in hospitals and especially in emergency departments (EDs) is certainly a key issue to be carefully mitigated. This can lead to overcrowding and the degradation of the quality of the provided medical services. Thus, the accurate modeling and forecasting of ED visits are critical for efficiently managing the overcrowding problems and enable appropriate optimization of the available resources. This paper proposed an effective method to forecast daily and hourly visits at an ED using Variational AutoEncoder (VAE) algorithm. Indeed, the VAE model as a deep learning-based model has gained special attention in features extraction and modeling due to its distribution-free assumptions and superior nonlinear approximation. Two types of forecasting were conducted: one- and multi-step-ahead forecasting. To the best of our knowledge, this is the first time that the VAE is investigated to improve forecasting of patient arrivals time-series data. Data sets from the pediatric emergency department at Lille regional hospital center, France, are employed to evaluate the forecasting performance of the introduced method. The VAE model was evaluated and compared with seven methods namely Recurrent Neural Network (RNN), Long short-term memory (LSTM), Bidirectional LSTM (BiLSTM), Convolutional LSTM Network (ConvLSTM), restricted Boltzmann machine (RBM), Gated recurrent units (GRUs), and convolutional neural network (CNN). The results clearly show the promising performance of these deep learning models in forecasting ED visits and emphasize the better performance of the VAE in comparison to the other models.