The work aims to propose a surface defect detection method based on YOLOv5 network by combining attention mechanism and Ghost convolution to solve problem of low detection accuracy of small size defects on metal workpiece surface. First, the SE channel attention module was added to the original network. The weight of the defect-related information was increased and the interference of useless features was reduced to improve the detection accuracy of the target. Then, the maxpool module of the spatial pyramid pooling module in the network was replaced with Softpool so as to retain more feature information in the down sampling activation map and obtain a better classification accuracy. Finally, Ghost convolutional blocks were used to replace the conventional convolutional modules in the backbone network to extract rich and redundant features and improve the efficiency of the model. The mean average accuracy of the improved network reached 0.997 8, increased by 7.07% over the original network. The proposed network significantly improves the accuracy of surface defect detection in metal workpieces.