Using a SHA512 hash to encrypt data

Question

How can I judge the level of security with the following algorithm:

I create a 64 byte hash using SHA512 via some input. I use this hash to iterate over the plaintext, byte by byte, and similarly iterate over the hash byte by byte XORing each byte of the plaintext with a respective byte from the hash like this (in C#):

for (int i = 0; i < data.Length; i++)
{
  if (j == key.Length) j = 0;
  data[i] = (byte)(data[i] ^ key[j++]);

}

where key is a byte array containing the hash. Issues of how the code acquires the key aside, is this secure? (Apologies if I am not framing this with the proper terminology). I am not using a salt. The input which is used to create the hash can be of any length (up to being reasonable; i.e., not a huge block of text).

Answer 1

No it's not. It is really bad.

Basically, this is a stream cipher, where your keystream is $key | key| key|key....$. This is really bad, similar to the level of "using OTP twice". As If you take a block of size keylength, and XOR two such blocks, then you get the XOR of the plaintexts. Depending on the nature of the plaintext, this can be really easy to guess.

If you want an easy encryption scheme, like XOR the plaintext with a keystream, use a stream cipher or a block cipher in OFB mode of operation. If you want a construction with SHA512, that is also possible, see e.g.

• Is it feasible to build a stream cipher from a cryptographic hash function?
• Can I use HMAC-SHA1 in counter mode to make a stream cipher?

However, since hash functions have different security properties than stream ciphers, it might be better to use a construction which is actually intended to be used this way. In a similar context, Bruce Schneier wrote (about ciphers based on hash functions):

While these constructions can be secure, they depend on the choice of the underlying one-way hash function. A good one-way hash function does not necessarily make a secure encryption algorithm. Cryptographic requirements are different. For example, linear cryptanalysis is not a viable attack against one-way hash functions, but works against encryption algorithms. A one-way hash function such as SHA could have linear characteristics which, while not affecting its security as a one-way hash function, could make it insecure in an encryption algorithm such as MDC. I know of no cryptanalytic analysis of particular one-way hash functions as block ciphers; wait for such analysis before you trust any of them.