The work aims to solve the defects as unsatisfactory sequence autocorrelation that causes low cipher security induced by neglecting time delay generated by the random sequences in current chaotic image encryption technology and difficulty in overcoming the periodicity of its own iteration. The cascade-coupled chaotic semiconductor ring laser was introduced to design an image encryption algorithm based on physical random bit generator and chaotic pixel cross interchange. Firstly, the SHA-256 hash function was introduced and a 256-bit key was generated with the value of the plaintext pixels, and then the 256-bit key was divided into a series of 8-bit sub-keys. A group of chaotic sequences were outputted by calculating the initial conditions of Logistic-Sine composite map with these sub-keys. Then, the pixel cross interchange mechanism was defined according to the chaotic sequence and the input plaintext was preprocessed to eliminate the correlation between adjacent pixels. The physical random bit generator was designed based on the cascade-coupled chaotic semiconductor ring laser which made full use of its own time delay and cross feedback to synchronously output the control matrix and stochastic bit stream that considered the time delay. The pixel confusion mechanism was defined by transforming the chaotic sequence outputted by the Logistic-Sine composite map into a transition matrix and jointing the control matrix to completely change the location of plaintext pixels. Finally, the pixel diffusion function was designed with random bit stream to change the pixel value of the image. The experimental results showed that the proposed algorithm had higher security and robustness which could effectively resist against the plaintext attack compared with the current chaotic encryption technology, and its corresponding cipher entropy was about 7.9958, and the values of NPCR (Number of Pixels Change Rate) and UACI (Unified Average Changing Intensity) were 99.50% and 33.46%, respectively. With higher security and anti-attack capability, the proposed encryption algorithm can protect the safe transmission of images in the network and it has better application value in the field of information security and anti-counterfeiting.