The work aims to propose an end-to-end practical algorithm for plain cloth defect detection in order to solve the problems in the current cloth detection algorithm including incomplete coverage of defect types, low defect detection accuracy and poor positioning accuracy. Firstly, the number of samples was expanded by image enhancement, the Cascade-RCNN network with Resnet50 as the backbone was used, and the method of adding deformable convolution and feature fusion network to increase the number of anchor frames realized the defect detection of plain cloth. The experimental comparison showed that the algorithm could detect 20 kinds of cloth defects, the accuracy of detecting whether the cloth was defective was 97%, the average detection accuracy of defect location was 65%, and the average time for detecting each sample was 80 ms. This algorithm effectively improves the accuracy and precision of cloth defect detection and detects more defect categories, and can obtain defect locations and categories, thus meeting industrial production needs.