The work aims to study the assessment algorithm of structural similarity between color images in the Matlab to evaluate the quality of color images. Based on the structural similarity index measurement (SSIM) of the grayscale image, in Matlab, the color images stored in RGB color space were converted into the YUV color space in which the brightness and color were measured respectively. Assessment algorithm was designed to measure values of the luminance comparison, the contrast comparison and the structure comparison. The structure similarity between color images could be acquired by integrating the three parts. The experimental results showed that this algorithm of color structural similarity could calculate the degree of similarity between color images and reference images and be used for quality evaluation. Moreover, the evaluation results are consistent with the evaluation of human visual perception (i.e. subjective evaluation). This method can be used as a new type of color image quality evaluation criteria.