This study aimed to improve the visual effect and the quality of the reconstructed images. Combining the characteristics of human visual system and spectrum coefficients in the transformation domain, a compensation matrix was constructed to improve the quality of decompressed image, which was to compensate the inverse quantization coefficients during the decompression. And the simulation experiments were carried out for three color images based on JPEG compression algorithm. The experimental results showed that, under the six compression ratios, compared with JPEG technology, SSIM and PSNR average values that reflected improved image quality were increased respectively by 2.5275% and 11.8977% by adding the compensation matrix in the reconstruction process of the decompressed image. The experiment showed the compensation matrix can enhance effectively the quality of the decompressed image, and compensates the quality degradation due to quantization during compression.