The restored images by the existing inverse halftoning methods still suffer either halftone artifacts or fine detail losses. The paper aims to improve the quality of inverse halftone image in smooth area and texture detail and propose a method of inverse halftoning based on multi-scale convolutional neural network combined with attention mechanism. Firstly, according to the multi-frequency distribution of halftone artifacts, a deep learning network based on multi-scale convolutional neural network is designed to suppress the noise and restore the image information at different scales. Secondly, inverse halftone images are generated by fusing different reconstructed information with attention mechanism. Finally, multi-task loss functions are presented to accelerate network optimization. Experimental results show that the inverse halftone images obtained by the proposed method are more similar to the original images and the image details are recovered better in vision. In terms of objective evaluation, the restored images by the proposed method have significantly higher peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) values compared to the state-of-the-art methods. The average PSNR increased by 0.562-10.95 dB, and the average SSIM increased by 0.01-0.171. This method can achieve high quality restoration of halftone images.