The work aims to detect the surface defects of pharmaceutical hollow capsules in quality inspection accurately and quickly. Based on YOLOv5 algorithm and aiming at the problems of large amount of model network parameters and weak learning ability of long-distance dependence, GhostNet module and Coordinate attention mechanism were introduced into the backbone network to make the network effectively capture the relationship between data location information and channel information. The experimental results showed that the improved network structure could accurately detect five kinds of defects such as damage, printing error, hole, scratch and depression on the surface of pharmaceutical capsule on the premise of decreasing to 57% of the original parameters. The average accuracy of each defect was 96.9%, which was increased by 2.4 percentage points. The detection speed was increased by 12 FPS. The proposed method can effectively classify and locate the surface defects of pharmaceutical capsules, and improve the accuracy of defect detection.