The extraordinary electronic, optical, and mechanical characteristics of 2D materials make them promising candidates for optoelectronics, specifically in infrared (IR) detectors owing to their flexible composition and tunable optoelectronic properties. This review presents the recent progress in IR detectors composed of 2D materials and their hybrid structures, including graphene, black phosphorous, transition metal dichalcogenides, halide perovskite as well as other new layered materials and their heterostructures. The focus is on the short-wave, mid-wave, and long-wave infrared regimes, which pose a grand challenge for rational materials and device designs. The dependence of the device performance on the optical and electronic properties of 2D materials is extensively discussed, aiming to present the general strategies for designing optoelectronic devices with optimal performance. Furthermore, the recent results on 2D material-based heterostructures are presented with an emphasis on the relationship between band alignment, charge transfer, and IR photodetection. Finally, a summary is given as well as the discussion of existing challenges and future directions.