The work aims to solve the problems of current paper defect classification algorithm, including poor real-time ability and difficulty in adapting to the requirements of on-line detection of the production line. An on-line paper defect classification method based on background subtraction and support vector machine (SVM) was proposed. Firstly, background subtraction method was used to determine whether the paper contained defects. Then, the paper with defects was marked by the marking machine and the images were stored. The eigenvectors of enclosing rectangle in the paper defect area were extracted. Finally, the paper defects were classified by SVM. Based on the comparison of the proposed method and the existing BP neural network as well as the naive Bayesian method, the classification accuracy was higher than that of the existing classification method. The four kinds of paper defects with classification accuracy of over 90% and good real-time ability were more suitable for on-line detection. The proposed method can effectively classify paper defects and meet the requirements of real-time detection and classification of the production line.