Is there an increased risk of nonce reuse when using GCM with a nonce more than 96 bits?

Question

GCM is designed to use a 96-bit nonce, which can be generated either randomly or deterministically. If you use a nonce that isn't 96 bits then it is padded and hashed with GHASH to create a pseudorandom nonce (and block counter) instead. For example, suppose that you generated uniformly random 128-bit nonces, these would be hashed with GHASH and converted into a 96-bit nonce and 32-bit random block counter. Given that GHASH is not collision resistant, is this significantly more likely to result in nonce reuse than generating a random 96-bit nonce directly?

Answer 1

If you insist on random nonces, 128 bit random nonces are safer than 96 bit random nonces (but not as good as 96 bit counter-based nonces - I presume that is infeasible in your scenario).

The 128 bit random nonce is transformed (via GHash) into what is effectively a 96 bit nonce and a 32 bit counter; in the 96 bit case, the 96 bit nonce is used directly, and we use a fixed 32 bit counter value value $1$. If this (96, 32) bit value is $(X, Y)$, then an encryption of an $N$ block message uses the values $(X, Y)$ through $(X, Y+N+1)$ (and the addition is done modulo $2^{32}$)

Now, GCM runs into trouble if either a used range contains the value $(0, 0)$ (which can't happen with 96 bit nonces), or two separate encryptions (with their nonces) use a value in common.

As for the first possible case (using a value $(0, 0)$ somewhere in the range), that happens with probability $2^{-128}(N+2)$ for an $N$ block message - obviously, 96 bit nonces are better there.

As for the second possible case (using two ranges that interest), 128 bit nonces are clearly better. After all, for two ranges to interest, they must share the same $X$ component; for 96 bit nonces, that is sufficient; for 128 bit nonces, the $Y$ components also need to intersect, and unless $N$ is near the maximum, that has a probability less than 1.

If we model the prohibition on the $(0, 0)$ value as a single message that just happens to use that single value, then that falls in line with the 'no two messages can use intersecting ranges'. Then, it is easy to see that encrypting $M$ messages with 128 bit random nonces must be at least as secure as encryption $M+1$ messages with 96 bit random nonces, and actually, because of the observation that two 128 bit nonces that happen to map to the same 96 bit space will generally not intersect, it is actually considerably better.

In addition, you claimed that GHASH was not collision resistant, and Woodstock claimed that GHASH output was likely not uniform. Actually, if $H \ne 0$, then with 128 bit inputs, GHASH is a permutation, hence it is collision resistant, and the output is uniform.