Gabor holography is an amazingly simple and effective approach for three-dimensional imaging. However, it suffers from a DC term, twin-image entanglement, and defocus noise. The conventional approach for solving this problem is either using an off-axis setup or compressive holography. The former sacrifices simplicity, and the latter is computationally demanding and time-consuming. To cope with this problem, we propose a model-based holographic network (MB-HoloNet) for three-dimensional particle imaging. The free-space point spread function, which is essential for hologram reconstruction, is used as a prior in the MB-HoloNet. All parameters are learned in an end-to-end fashion. The physical prior makes the network efficient and stable for both localization and 3D particle size reconstructions.