In many real-world systems with multiple sources of data, data are often missing in a block-wise way. For example, in the diagnosis of Alzheimer's disease, doctors may collect patients data from MRI images, PET images and CSF tests, while some patients may have done the MRI scan and the PET scan only, while other patients may have done the MRI scan and the CSF test only. Despite various data imputation technologies exist, in general, they neglect the correlation among multi-sources of data and thus may lead to sub-optimal performances. In this paper, we propose a model called regularized multi-source matrix factorization (RMSMF) to alleviate this problem. Specifically, to model the correlation among data sources, RMSMF firstly uses non-negative matrix factorization to factorize the observed multi-source data into the product of subject factors and feature factors. In this process, we assume different subjects from the same data source share the same feature factors. Furthermore, similarity constraints are forced on different subject factors by assuming for the same subject, the subject factors are similar among all sources. Moreover, self-paced learning with soft weighting strategy is applied to reduce the negative influence of noise data and to further enhance the performance of RMSMF. We apply our model on the diagnosis of the Alzheimer's disease. Experimental results on the ADNI data set have demonstrated its effectiveness.