The work aims to improve the defect detection accuracy of yarn-dyed fabric. A defect detection method of yarn-dyed fabric based on Gaussian pyramid and visual saliency was proposed. Firstly, the yarn-dyed fabric image to be detected was preprocessed to reduce the uneven illumination and the environmental influence. Then, the preprocessed image was grayed and layered based on Gaussian pyramid, and the layered images were processed based on saliency to obtain saliency maps. Finally, the saliency maps were segmented by iterative threshold segmentation to obtain the defect region of the yarn-dyed fabric image. Compared with other defect detection methods of yarn-dyed fabric, the detection effect of the proposed method was significantly better than other methods, and the accuracy of defect detection reached 91.25%. The proposed method can effectively detect and extract the defects of the yarn-dyed fabric, which provides useful information for the processing in the subsequent production of the yarn-dyed fabric.