Image-based optical methods have been widely used for noncontact structural displacement measurements, due to their prominent advantages over conventional contact sensors. However, existing optical methods usually require complicated and expensive imaging systems, and have difficulties to accurately measure in-plane displacements when the optical axis is not perpendicular to surface of the object (i.e., off-axis measurement). In this work, we develop a low-cost and portable smartphone-based optical method for accurately measuring off-axis structural displacements. The theoretical equations of the in-plane physical displacements on the object surface are derived based on the smartphone gyroscope data that can be used to determine the rotation matrix between the defined world coordinate system and the camera coordinate system. Simple calibration tests are performed to validate the accuracy of the smartphone gyroscope in detecting the rotation angles. The effectiveness and accuracy of this method in off-axis structural displacement measurement are then verified by two in-plane translation tests in laboratory condition. Finally, the noise from the smartphone camera is evaluated, and the effectiveness of the proposed method in continuous displacement measurement is further confirmed the vibration measurement of a hanging light under random wind load.