The capacity of a discrete-time memoryless Gaussian channel, where the channel state information (CSI) is neither available at the transmitter nor at the receiver, is addressed. A closed form expression of the optimal capacity-achieving input distribution at low signal-to-noise ratio (SNR) is derived, and the exact capacity of a non-coherent Single Input Single Output (SISO) channel is given. The derived relations allow to better understanding the capacity of non-coherent channels at low SNR. Then, we compute the non-coherence penalty and give a more precise characterization of the sub-linear term in SNR. Finally, upper and lower bounds on the capacity of a Multiple Input Multiple Output (MIMO) channel are derived in termes of its counterpart SISO channel capacity. We show that these bounds are sufficient to characterize the MIMO channel capacity at low SNR.