Objective The spatial visual effects of image colors among different pixels are not considered when using the typical color difference formula to assess the quality of color images. Targeting at this problem, a color image quality assessment algorithm based on optimal color space and visual masking effects was presented. Methods In this algorithm, an orthogonal and opponent color space, determined by analyzing correlation coefficient matrices of color spaces, was selected as its optimal working space. Visual masking functions for each channel of the color space were then used to de-correlate the spatial correlation caused by the interplay among colors in human vision. Finally, an image color difference formulae was constructed to assess the quality of color images. Results In the testing experiment, Pearson Correlation Coefficient (PCC), Spearman Rank-Order Correlation Coefficient (SROCC) and Kendall Rank-Order Correlation Coefficient (KROCC) were used to figure out the relationship between the objective assessment and the subjective visual assessment. In relation to the subjective assessment values, the PCC, SROCC and KROCC values of the objective assessment algorithms were 0.3948, 0.5840 and 0.4814, respectively, which were higher than those for other assessment algorithms. Conclusion The final results of the experiment showed that the proposed algorithm was essentially consistent with the subjective visual assessment.