The work aims to improve the encoder detection accuracy, study the frequency topology optimization of encoder mounting bracket to achieve lightweight structure and improve its dynamic stiffness characteristic. According to the actual working condition of the encoder mounting bracket, the finite element pre-processing was conducted on the encoder mounting bracket based on HyperMesh. With the loading node frequency response displacement of encoder mounting bracket minimized as the objective function of optimization, and the structural volume ratio before and after optimization as the constraint condition, the frequency response optimization model of the encoder mounting bracket was constructed and solved based on OptiStruct. Meanwhile, the OSSmooth was adopted to extract and reanalyze the CAD model of optimized results. The comparison of results before and after the optimization showed that, the dynamic stiffness of the encoder mounting bracket could be ignored in x direction and the value increased from 2564.25 to 5932.11, and from 62 500.36 to 109 792.54 respectively in y and z directions. The first order natural frequency was significantly increased from 3860.664 Hz to 8380.32 Hz. The proposed method not only improves the dynamic property of encoder mounting bracket in the working process, but also reduces its manufacturing material, thus achieving the lightweight structure.