Infrared polarization imaging is advantageous for its ability to enhance image contrast and identify true-false targets. In order to improve detection and identification probability, it is necessary to accurately obtain the infrared radiation polarization properties of the targets. However, the traditional analytical model of infrared radiation polarization ignores the shadowing effect caused by rough surfaces. Based on the surface microelement bidirectional reflectance distribution function and by using a Muller matrix, the stocks analytical model of the infrared radiation polarization degree of the rough surface is constructed with a shadowing function. The effects of the incident angle and the surface roughness on the polarization of the metallic and nonmetallic targets are analyzed quantitatively. The analysis results show that the polarization degree of infrared spontaneous radiation decreases with an increase in the roughness of both the metal and the nonmetal, and the decrease of polarization degree of nonmetallic is greater than that of metal. Under the same roughness and temperature, the degree of polarization of the infrared radiation of the metal is always greater than that of the nonmetal. The polarization degree of infrared radiation firstly increases with the incident angle, and reaches a peak value within a specific range of incident angle, and then decreases dramatically. The difference in the degree of polarization between the metallic and nonmetallic infrared radiation reaches a maximum within a certain range of incident angle. This property is useful for distinguishing the metal and nonmetal. Finally, a long-wave infrared micro-polarization imaging system and near-infrared polarization imaging system are used to collect different images. The infrared radiation polarization properties of the targets are reasonably consistent with the results of the theoretical analysis. This research is of great significance for analyzing the polarization properties of real targets, designing infrared polarization systems and processing polarization images.