In this paper, a novel chaotic image encryption algorithm based on extended Zigzag confusion and RNA operation is proposed. Firstly, a novel chaotic system is designed, which can be regarded as an improvement of traditional Logistic map and Sine map. Then it can be used to generate chaotic sequence combined with SHA 512 hash function. Secondly, extended Zigzag confusion scheme is proposed. It not can only process non square matrix, but also can choose from four directions to determine the position where encryption starts. This scheme can solve the problem of some elements keep in the same position in each encryption round. Thirdly, RNA operation is performed. In RNA computing, the choice of operators is determined according to the category of amino acids, and the coding rules of RNA matrix are completely controlled by chaotic sequence, which makes the results of operation more unpredictable. Finally, this encryption method can extend to color image. The data from experimental simulation and performance analysis show that this encryption algorithm has higher safety performance.