In order to realize the presence of longitudinal displacement and brightness difference, a panoramic image stitching technology was put forward based on HSV color space. This technology utilized a variable feature vector matching algorithm and conducted traversal of corresponding feature matching vectors by combining the maximum criterion of correlation coefficient for feature matching vectors of overlapping areas to find out the overlapping areas of adjacent images and achieve stitching by weight fusion for the overlapped area. This new algorithm had better stitching effect than traditional algorithm, with DoEM at 0.999 dB and average time consumption of 499 ms. The gradual fade-out effect effectively solved the problems of displacement and sudden change in brightness. This algorithm for stitching color panorama has good flexibility and repeatability and has no strict requirements on the size and color of the sub-graph. With fast stitching and high precision, this is a fully automatic stitching scheme for color panorama with longitudinal displacement and brightness difference.