As an important parameter determining the gear performance, the tooth side clearance affects the friction, vibration, noise and stiffness of the gear transmission system, so it is necessary to study the influence of different tooth side clearance on the dynamic characteristics of the helical gear. A rigid-to-flexible coupling model of the transmission system of the offset press with tooth side clearance was established by Adams software. The variation trend of the angular velocity, meshing force and torque of the gear under different tooth side clearance was studied by simulation, and the parameter changes before and after applying 200 N load were compared. The experimental results showed that the increase of tooth side clearance would reduce the rigidity and impact force of the gear transmission system, reduce the vibration and noise of the system, and also lead to the decline of transmission efficiency. Moderate tooth side clearance was conducive to the balance of transmission efficiency and system stability. After loading, the system showed higher stability, and the tooth clearance was still the main factor affecting the dynamic characteristics of the system. This study provides a theoretical basis for the design and optimization of gear systems.