The work aims to propose an extraction method based on the sub-pixel edge, in order to improve the extraction precision of the calibration board feature, as the camera calibration precision is the decisive factor in the field of vision measurement. Firstly, the calibration board image was acquired and processed to obtain the edge at pixel level regarding the dot calibration board. Then, 3×3 digital window was taken to calculate the gradient direction, taking the edge pixel as the center. The edge at sub-pixel level was obtained by gray level fitting in the gradient direction based on hyperbolic tangent. Finally, the center coordinates were acquired by the sub-pixel edge fitting according to the circle. The experiment showed that the resolution of the algorithm reached 0.03 pixels and the precision could reach 0.1 pixels. Characterized by stability, reliability, high precision and fast computing speed, the proposed algorithm can be applied in the field of image splicing and segmentation, feature extraction and camera calibration.