The work aims to solve such defects as easy occurrence of color information loss in the decryption process induced by the current optical image encryption algorithm that mainly directly applies the monochromatic beam to the plaintext. An optical image encryption algorithm based on hybrid amplitude-phase retrieval technique and two-dimensional coupled chaotic map was designed herein. Firstly, the R, G and B components of the color image were extracted. Then, the two-dimensional composite map was formed by nonlinearly coupling the Logistic map and Sine map. The three chaotic sequences were obtained by the pixel information of color image that iterated the composite map, so that the R, G and B components were scrambled by constructing the position engine obfuscation mechanism. Based on Logistic map, the initial value conditions generated by the plaintext pixels were used for the iteration of the Logistic map to output one chaotic random mask. Finally, the hybrid amplitude-phase retrieval technique was designed based on the amplitude-phase truncation method and Gyrator transform. The scrambled R, G and B components were encrypted by the unidirectional binary phase function and random mask to obtain the corresponding detection amplitude. Then, these components were combined to form the encrypted ciphertext of real-valued function. The experimental results showed that, with higher security and decryption quality, the proposed algorithm had stronger ability to resist plaintext attack compared with the current optical image encryption mechanism. The proposed encryption algorithm has higher anti-attack ability and can protect the safe transmission of images in the network. It has better application value in the field of information security and anti-counterfeiting.