In order to restore the panorama with the problems of translation, color difference, rotation and deformation, a color panorama stitching algorithm combining SIFT (scale invariant feature transform) and RBF neural network is proposed. The SIFT algorithm was used to match the corresponding feature points in the two subgraphs, and the affine transformation was used to solve the rotation and deformation problems between images. The RBF neural network was used to correct the color difference of the subgraphs. Finally, the weight matrix fusion technique was used to realize the fusion of overlap areas. The proposed algorithm was superior to other algorithms in the stitching effect. The DoEM value of the stitching effect was 0.902, and the image overlap area was excessively smooth, effectively avoiding the luminance block or the luminance line of the merged area. The proposed algorithm has good effect and can solve many problems in the panorama restoration process.