The work aims to propose an image inpainting algorithm based on matching adjustment rule and gradient constraint model, regarding such deficiencies as discontinuity effect and fuzzy effect in the restoration results caused by the situation that currently it is hard for major image inpainting algorithms to make self-adaptive adjustments of the restored block size according to different texture structures. Firstly, the confidence item was restrained by smoothing factor. The priority decision model was constructed to measure the priority of the block to be restored and determine the preferably restored block. Then, the SSD model was used to measure the matching results between sample blocks, and the matching adjustment rule was set out according to the matching results, so that the adaptive adjustment of sample block size could be made according to the matching degree, in order to improve the restoration quality. Finally, the gradient constraint model was constructed by combining the measurements of mean square distance of the pixels in the gradient modulus block with the pixels in the sample blocks, so as to obtain the best matching blocks for the filling and restoration of the blocks to be restored. The experimental results showed that, compared with the current image inpainting algorithm, the image restored by the proposed algorithm had better restoration quality and still had higher similarity value when the pixel loss rate was higher. The proposed algorithm has better visual restoration quality and can be used for the restoration of extensively damaged image.