In order to overcome shortcomings such as the low accuracy and the high complexity of the traditional approach of video shot boundary detection, an algorithm using spectral residual saliency map combined with sub-block gray histogram to calculate likehood ratio was proposed in this paper. First, to extract effective structure information as the feature to compare differences between frames, the spectral residual approach was used to obtain the structure saliency map of frames. Second, the frames were segmented into necessary subblocks to calculate all the histogram data which further improved the precision. The likehood radio of each frame was calculated then the frame-to-frame difference index proposed in this paper was computed to calculate and compare frame-to-frame differences. The detection precision of the proposed method can reach 93.82%, and the similarity of unchanged neighboring frames maintained as more than 99%. Experimental results showed that the method was more effective which had a higher precision and relatively simpler process， and it had a strong robustness against common influencing factors.