The work aims to optimize the structure of cycloid wheel according to its calculation results of modal analysis. Pro/E was applied to establish the 3D model for cycloid gear of RV reducer. The completed entity model was imported into ANSYS to build dynamic simulation analysis model for the finite element modal analysis, thus obtaining the inherent characteristics of cycloid gear under free and constrained boundaries. The results of the modal calculation were compared with the prototype's natural frequency. The position of the maximum rigid body displacement of the cycloid gear was analyzed according to the vibration mode, and the structure optimization was designed and verified. The natural frequencies of the 7th and 8th orders under free boundary and the 2nd and 3rd orders under constrained boundary were close to those of the complete machine, and the natural frequency range was 844.7~1163.7 Hz, which was easy to cause resonance. The position of the largest rigid body displacement of cycloid gear was the cycloid gear profile outside of the through hole, and the maximum displacements under the free and constrained boundaries were respectively 44.349 mm and 59.484 mm; after structure optimization, the maximum displacements were respectively 37.581 mm and 44.066 mm. The cycloid gear profile is the weak link of the structure, and the inherent characteristics and natural vibration mode of the optimized cycloid gear model are effectively improved.