Due to the underlying sparse structure of the mmWave channels, which indeed makes the exact closed-form capacity expressions inherently hard to derive, there has been less research on the ergodic capacity of mmWave systems. To overcome this problem, by means of the majorization theory, this paper analyzes the ergodic capacity of point-to-point mmWave communication systems under finite-dimensional channel model. In particular, we derive several closed-form ergodic capacity approximations, which exhibit excellent tightness in spite of whether the steering matrices are singular or not. Then, several Jensen's approximations and bounds of the ergodic capacity are also derived. The results indicate that the ergodic capacity seems to increase logarithmically with the number of antennas, the transmit SNR per antenna, and the eigenvalues of the steering matrix products. Besides, the DFT matrices can effectively characterize the spatial directions of mmWave channels when the number of antennas grows large. After that, high-SNR ergodic capacity, high-SNR slope, and power offset are also analyzed. It indicates that for a finite-dimensional channel, the maximum multiplexing gain increases with the number of paths instead of the number of antennas in Rayleigh channels. Numerical simulations are performed to validate the results.