Objective In order to obtain a stronger key encryption system and greater sensitivity to key space to improve the performance of the anti-attacks. Methods In this paper, a novel block encryption algorithm based on chaotic neural networks was proposed. The permutation and diffusion of the encryption system consisted of two 3- neurons layers, which were permutation neuron layer and chaotic neuron layer, and the two layers were controlled by a chaotic key generator block through corresponding weights and biases. In the chaotic neuron layer, three chaotic systems were used to generate the weights matrices and biases matrices, and through standardization and nonlinear Bit-wise XOR operation for non-linear combination, as well as a chaotic tent map is employed as the activation function of this layer; In the permutation neurons layers, and the key generation block was used to generate chaotic scrambling matrix which was applied to obtain diffused information, then a two-dimensional Cat chaotic map was applied to the data to obtain three-dimensional permutation. Results Compared with the existing encryption algorithm, the algorithm proposed in this paper was more secure, with an average entropy of 7.9991, the key space of this encryption algorithm was as large as 2160× 1060, and the sensitivity was high, the cipher text error rate between the wrong and correct keys was 99.765%. Conclusion Simulation results showed that the image encryption algorithm was highly secure and can effectively resist various attacks.