Objective According to the correlation between inter frames of video sequence images, a video image reconstruction algorithm based on inter-frame difference was proposed. Methods The background image obtained by the first two reconstructed frame image was used as the prior knowledge for reconstruction of the next frame image. Results The method obviously improved the quality of the reconstructed image and reduced the reconstruction time. Conclusion Simulation results showed that the method had obvious advantages in image quality and reconstruction speed by combining the inter-frame difference method and the compressive sensing reconstruction algorithm.