The work aims to improve the surface defect detection accuracy and automation level of the pharmaceutical hollow capsules in the process of quality inspection. A high-quality image acquisition scheme was designed to avoid light spots on the capsule surface. A dataset of defects in pharmaceutical hollow capsules was constructed. Based on the YOLOv4 algorithm, a deep learning detection model with multi-scale feature extraction and training strategies was developed to enhance the robustness of defect detection for small objects. The K-means++ clustering algorithm was used to update the initial values of the anchor frame to improve the performance of the model in detecting defects on the capsule surface. The experimental results showed that the proposed capsule defect detection method can accurately detect five types of defects which include dent, hole, scratch, stain and gap on its surface. Furthermore, the mean average precision of whether the capsule was defective or not was 99.05%, the average precision of each defect type was 91.81%, and the detection speed was 22 FPS. The method had certain advantages in comparison to other typical target detection methods in terms of detection speed and precision. The proposed YOLOv4-based defect detection method achieves the classification and localization of multiple types of defects in pharmaceutical hollow capsules, and has a better detection effect and stability, which can significantly reduce labor costs while fulfilling production quality control requirements.