I believe this shows that the scheme is a lot less secure than the original hash function $H$, with a security parameter only proportional to the square root of the output size of $H$, in the ideal case. It is a relatively straight forward adaptation of Wagner2002. It shows that it is not, as I was hoping, the use of modular groups that make the simple additive methods ineffective. Non modular addition is just as vulnerable, it seems.
Let $\mathcal{Y}$ be the set of triplets of the form $(\mathcal{h},\mathcal{A},\mathcal{B})$ where:
- $\mathcal{A}$, $\mathcal{B}$ are subsets of $X$
- $H^*(\mathcal{A}) \ge H^*(\mathcal{B})$
- $\mathcal{h} = H^*(\mathcal{A}) - H^*(\mathcal{B})$, a non negative integer
We define a combining binary operator $\oplus \colon \mathcal{Y} \times \mathcal{Y} \to \mathcal{Y}$.
Given two elements of $\mathcal{Y}$, say $y_r = (\mathcal{h}_r,\mathcal{A}_r,\mathcal{B}_r)$ and $y_s = (\mathcal{h}_s,\mathcal{A}_s,\mathcal{B}_s)$, with $\mathcal{A}_r$, $\mathcal{B}_r$, $\mathcal{A}_s$ and $\mathcal{B}_s$ all disjoint:
- if $\mathcal{h}_r < \mathcal{h}_s$, then $y_r \oplus y_s \to y_s \oplus y_r$
- otherwise $y_r \oplus y_s \to (\mathcal{h}_r - \mathcal{h}_s, \mathcal{A}_r \cup \mathcal{B}_s, \mathcal{A}_s \cup \mathcal{B}_r)$
The idea is that the elements of $\mathcal{Y}$ are collision candidates for $H^*$, and the $\oplus$ operator combines them to form a new candidate.
We create a stream $S^0$ of elements from $\mathcal{Y}$ by enumerating elements of $X$ in some order, and setting the $i^{th}$ element of $S^0$ to
$ S^0_i = (H(x_i), \{x_i\}, \emptyset)$.
We build a new stream $S^1$ by consuming elements of $S^0$ and storing them in a hash table indexed by the $l$ lower order bits of the hash difference.
When a collision is found, say $S^0_i = (\mathcal{h}^0_i,\mathcal{A}^0_i,\mathcal{B}^0_i)$ and $S^0_j = (\mathcal{h}^0_j,\mathcal{A}^0_j,\mathcal{B}^0_j)$, where $\mathcal{h}^0_i$ coincides with $\mathcal{h}^0_j$ in their $l$ low-order bits, we output $S^0_i \oplus S^0_j$ and erase $S^0_i$ and $S^0_j$ from the hash table.
We build new streams, $S^k$, by consuming elements of $S^{k-1}$ and storing them in a new hash table indexed by the $k\cdot l$ lower order bits of the hash difference.
When a collision is found, say $S^{k-1}_i$ coincides with $S^{k-1}_j$ in the $k\cdot l$ lower order bits, we output $S^{k-1}_i \oplus S^{k-1}_j$ and erase $S^{k-1}_i$ and $S^{k-1}_j$ from the hash table.
Note that if $k > 0$, $S^k$ is a stream of collision candidates, made of triples $(\mathcal{h},\mathcal{A},\mathcal{B})$ where $\mathcal{A}$ and $\mathcal{B}$ each have cardinality $2^{k-1}$.
If $k\cdot l \ge b + k$, $S^k$ is a stream of actual collisions.
To get the first real collision, you need to fill in all $k$ hash tables. This will consume on average about $2^{k+l}$ elements of $S^0$.
We therefore chose $l = \lceil \sqrt{b} \rceil$ and $k = \lceil (b + k) / l \rceil$. The running time is $\mathcal O(2^{2\sqrt{b}})$ invocations of $H$.
Thereafter, to get the next collision, you need to consume an average of $2^k$ elements of $S^0$. The running time is only $\mathcal O(2^\sqrt{b})$, just enough time to calculate $H^*$ on the next colliding pair.
The storage requirement is $\mathcal O(\sqrt{b} 2^\sqrt{b})$.
[ I expect this is not very intelligible. Am willing to update and clarify. ]
