The work aims to design an apple grading packaging system based on machine vision with apple sorting as the research object, in order to improve the accuracy of fruit grading. According to the actual grading function requirements, an overall scheme of apple grading system was designed based on the size of apple, including DSP+ARM controller, industrial camera, transmission belt, light source, etc. By focusing on the image processing module, image background segmentation was firstly completed. Median filter was used to realize image smoothing. Canny operator was used to sharpen image edge. Based on image processing, the apple grading model was established. Finally, an experimental study was carried out. The experimental results showed that the anastomosis rate of grading by manual sorting and automatic recognition reached 96.8%, which proved the effectiveness and feasibility of the system. The accuracy of automatic apple grading based on machine vision is close to the level of manual sorting, which can be extended to the automatic grading and packaging of apple fruits.