The work aims to propose an image inpainting algorithm based on similarity sparsity coupled with local difference feature, to solve the problem that the restored image of the current image inpainting algorithm easily loses the local details when the damaged area is large, thus causing such deficiencies as the ringing effect and the incoherence effect of the restored image. Firstly, the similarity sparsity model was constructed by the mean square distance of the pixels in the block to be restored and its adjacent blocks, so as to form the priority measurement function, and the priority repair block was determined according to the priority calculated based on the said function. Then, the local difference factor was constructed by the gradient vector value corresponding to the sample block. The local difference of the sample block was calculated, and the size of the sample block was adjusted on the basis of the calculation results. Finally, the approximate function was constructed based on the color difference information of pixels, and the best matching blocks were selected to repair the blocks to be restored. The simulation results showed that, compared with the current image restoration algorithms, the proposed algorithm had higher restoration quality and efficiency. The restored image was not subject to ringing effect and incoherence effect, etc. The proposed algorithm has higher visual restoration quality and can be used for the restoration of extensively damaged image.