The work aims to propose a lightweight method based on YOLOv4 to detect recyclable garbage, so as to address the problems of slow detection speed and large model weight files in the current garbage detection methods used by smart garbage sorting devices. The MobileNetV2 lightweight network was used as the backbone network of YOLOv4 and the depth-separable convolution was used to optimize the neck and head networks to reduce the parameters and computation to accelerate detection. The CBAM attention module was incorporated into the neck network to improve the sensitivity of the model to the target feature information. The K-means algorithm was used to re-cluster to get suitable self-built recyclable data with a priori frame for focused detection of targets. The experimental results showed that:the parameters were reduced to 17.0% of the original YOLOv4 model. The detected mAP reached 96.78%. The model weight file size was 46.6 MB, which was about 19.1% of the YOLOv4 model weight file. The detection speed was 20.46 frames/s, which was improved by 25.4%. Both the detection accuracy and the detection speed met the real-time detection requirements. The improved YOLOv4 model can guarantee high detection accuracy and good real-time performance in detection of recyclable garbage.