Effect of Self-Invertible Matrix on Cipher Hexagraphic Polyfunction

A cryptography system was developed previously based on Cipher Polygraphic Polyfunction transformations, C-ixj((t)) equivalent to A(ixi)(t)P(ixj) mod N where C-i x j, P-i x j, A(i x i) are cipher text, plain text, and encryption key, respectively. Whereas, (t) is the number of transformations of plain text to cipher text. In this system, the parameters (A(i x i), (t)) are kept in secret by a sender of messages. The security of this system, including its combination with the second order linear recurrence Lucas sequence (LUC) and the Ron Rivest, Adi Shamir and Leonard Adleman (RSA) method, until now is being upgraded by some researchers. The studies found that there is some type of self-invertible A(4x4) should be not chosen before transforming a plain text to cipher text in order to enhance the security of Cipher Tetragraphic Trifunction. This paper also seeks to obtain some patterns of self-invertible keys A(6x6) and subsequently examine their effect on the system of Cipher Hexagraphic Polyfunction transformation. For that purpose, we need to find some solutions L-3x3 for L-3x3(2) equivalent to A(3x3) mod N when A(3x3) are diagonal and symmetric matrices and subsequently implement the key L-3x3 to get the pattern of A(6x6).