Low-rank decomposition and visual saliency feature have shown great potential in fabric defect detection, however, there are still two shortcomings as follows: First, the non saliency information of fabric image will interfere with low-rank decomposition; Second, if the defect regions are very small, the defect priors, which are obtained by similarity measure, will cause a lot of false detections. To solve these problems, in this paper, we propose a detection model via low-rank decomposition with multi-priors and visual saliency features by the following three steps: (1) an initial saliency map is designed as the input of the low-rank decomposition model to enlarge the distance between the defect and the defect-free regions; (2) local and global priors are combined to highlight the location of defect; and (3) the residual difference between the initial saliency map and the low-rank part is expressed as a dual quaternion image. Here, we like to point out that the multiple saliency maps used in the paper are generated by quaternion Fourier transform and spectral scale space to ensure the integrity of the detection results. Our proposed new method is evaluated based on the standard database against with other recent six detecting methods, and the experimental results show that our method has better results, i.e., the overall TPR of star-, box- and dot-pattern is up to 87.22%.