Point-of-interest (POI) recommendation has great significance in helping users find favorite places from a large number of candidate venues. One challenging in POI recommendation is to effectively exploit geographical information since users usually care about the physical distance to the recommended POIs. Though spatial relevance has been widely considered in recent recommendation methods, it is modeled only from the POI perspective, failing to capture user personalized preference to spatial distance. Moreover, these methods suffer from a diversity-deficiency problem since they are often based on collaborative filtering which always favors popular POIs. To overcome these problems, we propose in this paper a personalized geographical influence modeling method called PGIM, which jointly learns users' geographical preference and diversity preference for POI recommendation. Specifically, we model geographical preference from three aspects: user global tolerance, user local tolerance, and spatial distance. We also extract user diversity preference from interactions among users for diversity-promoting recommendation. Experimental results on three real-world datasets demonstrate the superiority of PGIM.