Polarization, which represents the vector nature of electromagnetic waves, plays a fundamental role in optics. Fast, simple and broadband polarization state characterization is required by applications such as polarization communication, polarimetry and remote sensing. However, conventional polarization detection methods face great difficulty in determining the phase difference between orthogonal polarization states and often require a series of measurements. Here, we demonstrate how polarization-dependent holography enables direct polarization detection in a single measurement. Using a multiplexed Pancharatnam-Berry phase metasurface, we generate orthogonally polarized holograms that partially overlap with a spatially varying phase difference. Both amplitude and phase difference can be read from the holographic image in the circular polarization basis, facilitating the extraction of all Stokes parameters for polarized light. The meta-hologram detects polarization reliably at the near-infrared experimental wavelength of 798 nm and simulations predict broadband operation in the 580–940 nm spectral range. This method enables fast and compact polarization analyzing devices, e.g. for spectroscopy, sensing and communications.