The work aims to propose a surface defect detection algorithm improved based on convolutional neural network, so as to solve the problems of low precision, slow speed and large image noise of weldment surface defect detection in complex industrial scenes. The model was established based on the YOLOv4 algorithm. Firstly, considering the limitation of storage and computational resources, the lightweight network GhostNet was used to replace the YOLOv4 backbone feature extraction network (Backbone) CSPDarknet53. Secondly, an improved channel attention mechanism was embedded in the GhostNet network structure, which improved the learning ability of the model and reduced the parameter quantity. Finally, the K-means++ clustering algorithm was introduced to cluster the width and height of the labeled frames to be detected in the weldment surface defect dataset, so that the network model could detect the defects in the samples. From the experimental results, the improved YOLOv4 algorithm had an average precision (mean Average Precision, mAP) of 91.07%, a detection speed of 48.11 frame/s, and a model size of 43.2 MB. Compared with the original YOLOv4 algorithm, the detection precision was increased by 4.61%, the detection speed was improved by 26.59% frame/s and the model size was reduced by 82.37%. The proposed model improves the detection precision and speed of weldment surface defect, which is of practical significance in industrial surface defect detection.