Reliable quantification of cellular treatment effects in many bioassays depends on the accuracy of cell colony counting. However, colony counting processes tend to be tedious, slow, and error-prone. Thus, pursuing an effective colony counting technique is ongoing, and varies from manual approaches to partly automated and fully automated techniques. Most fully automated techniques were developed using deep learning (DL). A significant problem in applying DL to this task is the lack of sizeable collections of annotated plate images. For this reason, here we propose an application of Transfer Learning to cell colony counting that can overcome this problem by exploiting models trained for other tasks. To demonstrate this idea’s feasibility, we show how a small dataset can be used to transform a DL model designed for counting objects in congested scenes into a specialized cell colony counting model and achieve better performance than existing, more widely-used models.