I am trying to implement generalization of Hill cipher. My idea is very simple: the size of key matrix should be arbitrary number not only three. All steps of this cipher is trivial except computation of determinant.
Formally we have square matrix of order n under the finite ring of order r. There is the trivial algorithm which always gives the answer, but it's quite impractical. Gaussian elimination method is completely clear for me only in the case of matrix under real numbers. For example, we have $$ A= \begin{bmatrix} 6 & 1 & 3 \\ 2 & 5 & 6 \\ 3 & 4 & 5 \\ \end{bmatrix} $$ According to Gaussian method I should multiple the first row by -2/6 and add to the second row. In result I will have 0 instead of 2 in the second row, and so on. But how does Gaussian method work in the following case: $$ B= \begin{bmatrix} 6 & 1 & 3 \\ 2 & 5 & 6 \\ 3 & 4 & 5 \\ \end{bmatrix} mod r$$
I can multiple the first row by -2/6 only when r is prime number, in this case we have finite field, so multiplicative inverse exists for all nonzero elements. But the determinant of this matrix certainly exists in the all cases even if modulus r is composite number and we don't have finite field.
How can I efficiently compute the determinant of matrix under finite ring of arbitrary order r?