The work aims to propose a stitching restoration model and algorithm about the correlation coefficient based on the color image fragments divided into horizontal and vertical slices, in order to realize the stitching of the color image fragments in the horizontal and vertical slices. By obtaining each monochrome image of the color image fragments and extracting the gray value of the monochrome image edge, the fragments were automatically stitched according to the similarity between gray values of the image fragment edge. The stitching effects of the color image fragments resulting from the proposed algorithm were superior to the conventional algorithm. In the experiment of 100 color images, the 64 pieces of color image fragments with 300×300 pixels cut from each color image were restored in order. The success rate of the comprehensive stitching reached 100%, and the stitching took an average time of 1.59 s. In addition, the proposed algorithm was strongly experimental. It could not only stitch the images divided into vertical slices, but also stitch the color images divided into horizontal and vertical slices. The experimental results show that the proposed algorithm has good adaptability and repeatability for the stitching of color image fragments. No strict requirements are needed for the size and color of the image fragments. The algorithm is a complete set of effective automatic stitching scheme for the regular color image fragments.