We address the capacity of a discrete-time memoryless Gaussian channel, where the channel state information (CSI) is neither available at the transmitter nor at the receiver. The optimal capacityachieving input distribution at low signal-to-noise ratio (SNR) is precisely characterized, and the exact capacity of a non-coherent channel is derived. The derived relations allow to better understanding the capacity of non-coherent channels at low SNR. Then, we compute the noncoherence penalty and give a more precise characterization of the sublinear term in SNR. Finally, in order to get more insight on how the optimal input varies with SNR, upper and lower bounds on the non-zero mass point location of the capacity-achieving input are given.