Aiming at the problem that the traditional fruit and vegetable grading equipment and technology are complex and not suitable for small-scale fruit growers, a tomato size and shape grading detection algorithm based on machine vision was designed. In this algorithm, the improved gray level calculation based on color difference and Otsu algorithm was used to solve the problem of the influence of shadow on contour. Then, the top view of tomato image was obtained by distortion correction based on control point transform, and contour noise was removed by area threshold and Fourier transform. Finally, the physical size of tomato was calculated by comparing the pixel size of reference object, and automatic grading and calibration were carried out. The experimental results show that the accuracy of tomato size measurement with shadow interference is more than 95%, and the measurement deviation is negative. The algorithm uses software to replace the hardware needed for traditional grading, and solves the key problems such as shadow, distortion, noise, etc.