Vector GARCH model and vector SV model, which are based on the low frequency data, are difficult to be estimated. So it isn't easy to apply them to forecast the covariance matrix of many assets. Vector ARFIMA model can be used to model the covariance matrix of many assets computed by using high frequency financial data, but it is difficult to be estimated when the mention grows. This paper puts forward orthogonal ARFIMA model, which changes the modeling of the covariance matrix of many assets to the modeling of the variances of their principal factors by principal factor analysis. Orthogonal ARFIMA model lowers the mention of many assets, and it is easy to be estimated. It is meaningful to asset pricing, asset allocation, risk management and so on.