Background: Although the effect of pre-determined beam orientation on dose distribution of intensity modulated radiotherapy (IMRT) has been well-documented, its impacts on dose prediction are less investigated. In this study, the direction map of beam orientation was incorporated into our proposed deep-learning network and utilized in dose prediction of IMRT plans consisting of multiple static fields. Methods: The direction map was used to characterize the radiation path through region of interest along a beam orientation. Besides, the distance map was used to characterize the spatial distribution between organs at risk (OARs) and planning target volume (PTV). The input of prediction model consisted of CT image, mask image (for PTV and OARs), distance map, and direction map. The output of prediction model was the estimated dose distribution in three dimensions. A 3D fully-connected network composed of a down-sampling encoder and an up-sampling pyramid decoder was trained based on the calculated 3D dose distributions obtained from a treatment planning system. The voxel-level mean absolute error (MAE), dosimetric metrics, and dose-volume histogram were employed to assess the quality of the estimated dose distribution. Performance of the prediction model was evaluated in two aspects. First, the effectiveness of the new features, direction map, distance maps, and pyramid decoder on prediction accuracy of model were assessed. Second, the proposed model was compared with the other three published prediction models, 3D UNet, ResNet-anti-ResNet, U-ResNet-D for inter-model evaluation. Results: The improvement of prediction accuracy was 0.38 with the input of direction map and 0.43 with the input of distance map. Our proposed model achieved the least MAE (3.97 +/- 1.42) compared with the other three models: (5.37 +/- 1.51) for ResNet-anti-ResNet, (4.45 +/- 1.52) for U-ResNet-D, and (4.53 +/- 1.72) for Unet-3D. Conclusions: The preliminary result demonstrated that the prediction accuracy of the proposed model was higher than those of the other three state-of-the-art prediction models. The introduction of direction maps, distance map, and pyramid decoder can effectively improve the performance of the current deep-learning network-based prediction models.