The work aims to design a more efficient automatic garbage classification system based on the lightweight network model to solve the problems of the traditional machine vision network model, such as large number of references, low efficiency and difficult landing. The innovative design of the structure could realize the automatic switching of four kinds of garbage classification and storage with different proportions and the working mode of the garbage bin. The STM32 control mechanism motor and a variety of sensors were used to communicate with the Raspberry PI 4B serial port to realize garbage classification and delivery. The cloud server realized the Internet of Things communication at the small program side to improve management efficiency. The MobileViT lightweight model was used to train on the self-built data set, and the training speed and accuracy of the model were improved by combining transfer learning. The feasibility was verified by comparing the model with the mainstream model. The accuracy of MobileViT model could reach 98.01%, the average reasoning time of a single image in the actual test was only 17.8 ms, and the number of model parameters was only 5.6×106. The accuracy was 9.25% higher than that of lightweight network MobileNetV3 under the similar parameters. The performance indexes were better than those of traditional ResNet50 and AlexNet models. The design of automatic garbage classification system based on MobileViT lightweight visual model can complete the task of automatic garbage classification more efficiently. The accuracy and speed of the model meet the actual demand, and it is very friendly to the edge equipment in the field of garbage classification.