When the noise exists, especially the relatively serious noise level, the visual quality of color image will be reduced. The work aims to propose a color image denoising method based on depth residual learning, in order to remove noise effectively and make the denoised image have better visual effect. Firstly, a residual layer consisting of several residual unit modules was designed, and then the skip connection was added between residual unit modules to form the non-linear mapping from noise image to denoised image. The number of residual units was optimized, so that the network could learn more image details to improve the denoising performance. At the same time, the activation function of each residual unit module is moved to the front of the convolution layer to accelerate the network convergence. Compared with common denoising algorithms, the proposed method had better effects in subjective visual score MOS values and objective indicators (PSNR and SSM) on Kodak24 and CBSD100 datasets. The proposed color image denoising method based on depth residual learning can effectively remove the noise in the image, especially when the noise is serious, and obtain satisfactory visual effect, which shows that the proposed method has good denoising performance.