Two-dimensional Gabor filter contains many parameters, and the effect of two-dimensional Gabor filter using different parameters to enhance image features is quite different in printing overprint defects detection. The paper aims to obtain the optimal parameters of two-dimensional Gabor filter in printing overprint defects detection. In the process of overprint defects detection, a PSO-Gabor-CNN algorithm was proposed. Sobel operator was used to detect the edge of printed images. Particle swarm optimization (PSO) was used to optimize the maximum central frequency 'kmax', bandwidth 'σ' and template 'window' of two-dimensional Gabor filter. The weighted Euclidean distance between the processed images and the template images was evaluated, and then the optimized Gabor filter was used to filter the images. Finally, the Convolution Neural Network (CNN) was used to detect and classify the printing overprint defects. The maximum center frequency of two-dimensional Gabor was 6.0476, the bandwidth was 0.1444 and the window of template was 27×27 by particle swarm optimization. At this time, the weighted Euclidean distance was 1.1927×10-33. The mean square error of convolution neural network after 70 times of training was 0.0035, and the accuracy of test samples was 96.93%. Compared with BP neural network (BPNN) without data preprocessing, Sobel BP neural network (Sobel-BPNN) with Sobel preprocessing, convolutional neural network (CNN) without data preprocessing and Sobel convolutional neural network (Sobel-CNN) with data preprocessing, this method showed better recognition effect. This method can obtain the optimal parameters of two-dimensional Gabor filter and obtain good filtering effect. It has certain application value in overprint defects detection.