The work aims to propose an improved meshing stiffness calculation method to enhance the accuracy and efficiency of meshing stiffness calculation of gear system in dynamic analysis. By accurately modeling the time-varying meshing stiffness, the dynamic behavior of gear system is deeply discussed, thus providing theoretical basis for optimizing gear design and improving system performance. A meshing stiffness calculation method for helical gears based on the improved potential energy method was proposed and then analyzed and compared with the traditional potential energy method, finite element analysis, and ISO 6336-1 standard. A dynamic model of the gear system was established in MATLAB, and the dynamic meshing force and radial and axial vibration characteristics of helical gears were analyzed. The dynamic meshing force of the rubber cylinder and plate cylinder gear pair was similar to that of the plate cylinder and impression cylinder gear pair. The radial vibration index of rubber cylinder transmission helical gear was small, but the axial vibration was large. The radial vibration of plate cylinder and impression cylinder transmission helical gear was close, with opposite directions and the axial vibration direction was the same. In summary, the improved potential energy method significantly improves the calculation efficiency and reduces the workload, and the resulting meshing stiffness is closer to the actual working conditions. In addition, through the dynamic analysis, the change of meshing force and vibration characteristics of the gear operation is understood comprehensively, which provides a powerful guidance for optimizing the gear design and improving the performance of the transmission system.