Hill Cipher - Unknown plaintext attack, known key

Question

I was wondering if I have the key and the encrypted Hill Cipher message(s). I can definitely figure out the charset involved. But the order is missing. Assuming that I use 37 modulo or higher. How would I go about cracking it?

Answer 1

You don't. You just decrypt it. I can't think of a language that uses more than 36 letters apart from the Chinese family but then again I'm not a linguist.

Decrypt to a reasonable $n$ and then use the character frequencies to infer the language (and hence the modulo). Some trial and error may be required untill you get it right but for values of $n$ less than $2^{32}$ it shouldn't give you any practical problems on a desktop computer.