ROT encryption with hash functions

Question

I am not trying to create my own encryption (I am just interested in the math and possibility of this form of "encryption"). I apoligize if this post does not fit in, it is my first time posting.

A Caesar cipher is probably the most basic form of encryption, but what if we were to step it up?

1. We take a password key

2. We hash it through a SHA256 algorithm and save the result

3. We hash the result multiple times saving the result each time

4. We take all of the results and stack them (using simple numbers for example)

   123 < Hash
   321 < Hash 2
   221 < Hash 3

5. We add up the values of each column

   1 2 3 < Hash 1
   3 2 1 < Hash 2
   2 2 1 < Hash 3
   (6)(6)(5) < Final result

6. We apply the resulting numbers to our text to be encrypted through a Caesar cipher

7. When we get to the last digit in the ROT, we continue steps 3-6 untill complete

Decryption would be as simple as entering a password to hash over and over and undo the Caesar cipher

What is the practicality of this; does it exist/what would be potential flaws?

Here is an example of this encryption in python. Run the program and you will see what I am trying to explain.

Answer 1

What you are describing is a strange version of OFB mode, using a hash function instead of a block cipher and modular addition instead of XOR to combine the data with the keystream. The algorithm you describe appears vulnerable to known plaintext attacks. A simple tweak to make it more similar to true OFB mode would improve security. OFB mode is a stream cipher mode:

\begin{align*} O_i = E_K(O_{i-1}) \quad\text{where}\quad O_0 = IV \end{align*}

The keystream is $O$, so encryption and decryption is implemented as in any stream cipher:

\begin{align*} C_i &= P_i \oplus O_i\\ P_i &= C_i \oplus O_i \end{align*}

This technique can be visualized:

The block cipher, $E$, can be replaced with any keyed hash function. The hash function must be keyed and the initial input must be an IV, not a key, otherwise it is vulnerable to a known plaintext attack where an attacker who can derive one of the outputs can use it to calculate the rest of the keystream. Your current construction is $O_i = H(O_{i-1})$ where $O_0 = H(K)$, with multiple blocks of keystream $O$ being used to encrypt a single block of plaintext. If I understand this correctly, this does not resist known plaintext attacks. Instead, you need to use a keyed hash and the first input should be an IV. One of the most common keyed hash constructions is an HMAC:

\begin{align*} \operatorname{HMAC}_K(m) = H((K \oplus \mathrm{opad}) \mathbin\| H((K \oplus \mathrm{ipad}) \mathbin\| m)) \end{align*}

The $\oplus$ represents the XOR operation. Any operation that can be reversed with the same input can be used for this purpose, not just XOR. In your scenario, a Caesar cipher functions as modular addition, so if I understand what you are describing correctly, it should have similar security to OFB mode, including its pitfalls (short permutation cycles, for example), as long as you replace your hash function with a proper keyed hash like an HMAC.

Answer 2

If I understood your description correctly, your scheme is to generate a pad with $H(k) \mathbin\| H(H(k)) \mathbin\| H(H(H(k))) \mathbin\| \cdots$ to combine with the plaintext with xor (the binary equivalent of the Caear cipher's addition-modulo-26), where $k$ is your secret.

Unfortunately, this is completely insecure: an adversary who can guess a single block of plaintext can decrypt the rest of the message.

Specifically, break the plaintext $P$ into blocks $P_1, P_2, \ldots$; then the corresponding ciphertext blocks are $C_1 = P_1 \oplus H(k), C_2 = P_2 \oplus H(H(k)), \ldots$, or, generally, $C_i = P_i \oplus H^i(k)$, where $H^i(k)$ is the $i$-fold iteration of $H$: $H^1(k) = H(k)$, $H^{i+1}(k) = H(H^i(k))$.

In any attack model, the adversary knows $C_i$ but not $k$. If the adversary knows or can guess $P_i$—for instance, if they correctly guess that the message starts with the text ‘Dear Bucephalus,’—then from the above equation relating $P_i$ and $C_i$ they can recover $H^i(k) = P_i \oplus C_i$, and from there they can compute $H^{i+1}(k) = H(H^i(k))$ and proceed to recover $P_{i+1} = C_{i+1} \oplus H^{i+1}(k)$, and so on, decrypting everything else in the message.