The work aims to propose an image inpainting algorithm based on priority improvement and matching optimization measurement, with respect to the problem that the current image inpainting algorithm achieves the matching measurement between repair block and matching block to complete the image restoration mainly by the fixed single template size, thus causing such deficiencies as certain blurring effect and ringing effect. First, the smoothing factor was constructed by the data item, the priority model was established to measure the priority of the pixel to be repaired, and the priority repair block was selected. Then, the four level template size was formulated. By means of square error and functions, combined with the template size characteristics, the matching measurement model was constructed to achieve the dynamic matching of repair block and matching block. The best matching template was selected to fill and restore the block to be repaired. Finally, the neighborhood gray differential model was constructed by the gray of the pixels to be restored and their neighborhood pixels, which could be used to stitch the edges of the repaired region and optimize the restoration effect. With the best matching measure, a confidence updating model was constructed to update the confidence item and realize the image restoration. The simulation results showed that, compared with the current image inpainting algorithm, the proposed algorithm had higher inpainting quality, and its output image was subject to no blurring effect and ringing effect. The proposed algorithm can better restore the damaged image and it has certain reference value in the field of image information processing.