In this letter, we analyze the spectral and energy efficiency of an intelligent reflecting surface (IRS)-assisted multiple-input single-output (MISO) downlink system with hardware impairments. An extended error vector magnitude (EEVM) model is utilized to characterize the impact of radio-frequency (RF) impairments at the access point (AP) and phase noise is considered for the imperfect IRS. We show that the spectral efficiency is limited due to the hardware impairments even when the numbers of AP antennas and IRS elements grow infinitely large, which is in contrast with the conventional case with ideal hardware. Moreover, the performance degradation at high SNR is shown to be mainly affected by the AP hardware impairments rather than the phase noise of IRS. We further obtain in closed form the optimal transmit power for energy efficiency maximization. Simulation results are provided to verify the obtained results.