The work aims to solve such problems as block effect and discontinuity effect in the restored image induced by the current image inpainting method that mainly determines the restoration process with the normal vector of pixels to be repaired, so that the repairing order is not guaranteed from the periphery to the center of the damaged area. A new image inpainting method was designed based on guided factor coupled with curvature penalty model. The distance between the center pixel of the damaged area and other arbitrary pixels to be repaired was used to construct the guided factor, and the guided factor was combined with the confidence degree and data item to form the priority model for the selection of the priority repair block. The smoothness of the block to be repaired was judged by its gradient feature to determine the search range of the optimal matching block corresponding to such block, and the optimal matching block was searched through the sum of squared differences (SSD) function, so that the pixel point diffusion in the optimal matching block was filled to the block to be repaired. Finally, the curvature penalty model was constructed based on the curvature of the equal illumination line between pixels, which was used to update the confidence term and thus realize image restoration. The test data showed that, compared with the existing image inpainting scheme, the proposed algorithm could better give consideration to restoration quality and efficiency. The proposed scheme has better restoration quality and can be used for the restoration of extensively damaged image.