Consequence of improper validation in point decompression?

Question

Assume a standard ECC curve in a prime field $\mathbb F_p$ with $p\equiv3\pmod 4$, such as secp256k1; and code turning a bytestring for a compressed ECC public key into an Elliptic Curve point, that does as specified except in the following two emphasized spots:

2.1. Parse $M=Y\mathbin\|X$ as a single octet $Y$ followed by $\lceil(\log_2 q)/8\rceil$ octets $X$.

2.2. Convert $X$ to a field element $x_P$ of $\mathbb F_q$ using the conversion routine specified in Section 2.3.6. Output “invalid” and stop if the routine outputs “invalid”.

2.3. If $Y=\mathtt{02_{16}}$, set $\tilde y_p=0$, and if $Y=\mathtt{03_{16}}$, set $\tilde y_p=1$. Otherwise output “invalid” and stop.

2.4. Derive from $x_P$ and $\tilde y_p$ an elliptic curve point $P=(x_P,y_P)$, where:

• 2.4.1. If $q=p$ is an odd prime, compute the field element $\alpha={x_P}^3+a\,x_P+b\bmod p$, and compute a square root $\beta$ of $\alpha$ modulo $p$. Output “invalid” and stop if there are no square roots of $\alpha$ modulo $p$, otherwise set $y_P=\beta$ if $\beta\equiv\tilde y_p\pmod 2$, and set $y_P=p −\beta$ if $\beta\equiv\tilde y_p\pmod 2$.

Specifically, assume that, as in the code of this answer (on bitcoin-SE):

• in 2.2, $x_P\ge p$ is not caught;
• in 2.4.1, $\beta$ is computed as $\beta\gets\alpha^{(p+1)/4}\bmod p$, but it's checked neither that $\beta^2\bmod p=\alpha$ (direct check that the computed $\beta$ is a square root), nor that $\alpha^{(p-1)/2}\bmod p=1$ (Euler's criterion).

Therefore, when decompressing an invalid compressed public key, we can end up with $P=(x_P,y_P)$ not on the curve (even after reducing the coordinates modulo $p$), but still a very specific relation between $x_P$ and $y_P$. Also we can end up with $x_P=0$, or $x_P=p$, or $x_P>p$, or perhaps (I did not check) $y_P=0$ or $y_P=p$. A few more oddities are possible when $\log_2p\not\equiv7\pmod8$ (e.g. for secp521r1).

What can be the consequences, in various common primitives (ECDSA, ECIES, ECDH, ECMQV…), protocols, contexts? In particular

• Assuming a certification authority with that specific blunder, and no further sanity check of the public key, could a Certificate Signing Request for the invalid public key still pass the signature check of the CSR?
• Is there some protocol where these blunders could allow/has allowed an attack?

Answer 1

I'm not mathematically expertized, so this is provided as foundation for further reasoning, development, and refutation. My feeling right now is that, exploitation is difficult, but may be viable.

Effect of Unvalidated X (little)

Because operands are processed in modular arithmetic, an out-of-range X poses little threat. But it can nonetheless lead to an invalid Y.

Key Recovery Attack

Referencing https://safecurves.cr.yp.to/twist.html for background knowledge, here's a list of preconditions that are required to carry out a key recovery attack on an elliptic-curve decryption oracle:

1. the decryption oracle accepts an ECC ciphertext in the exact fashion as described in the OP's question - that is, a compressed point for key-consensus, and a symmetric-key encrypted ciphertext payload.

2. the key-consensus ECC point is on an "invalid" curve, where the so called "invalid curve attack" in the referenced URL is carried out. The point may be valid on that curve, but will not pass proper validation on the decryption oracle's "real" curve.

Such invalid curve attack would've been otherwise impossible if the implementation properly validate both uncompressed and compressed ECC points, but it's the assumption in the OP's question that parsing of compressed ECC points is improperly implemented.

Notice that I didn't say ECIES and I only said elliptic-curve decryption oracle. This is because ECIES uses authenticated encryption for the ciphertext payload, which constitutes an extra layer of defense. ECDHE may be vulnerable, but for the major curves in widespread use listed in the referenced URL, the difficulty of combined mathematical-theoretic analysis (small subgroup + invalid curve) is close to being as difficult as brutal-force - which is almost twice as difficult as plain Pollard-rho.

Digital Signature Forgery (Conjectured)

Most forms of DSS over Weierstrass involve generating an ephemeral key-pair:

• $k$ a scalar
• $R = [k]G$

Attacker may construct an invalid ephemeral key-pair to produce a valid signature that can be verified using another invalid public ECC point. Right now, I have no idea how this may be worked, but I conjecture that it's possible, at least in some implementations^[1] and for some formula of ECDSA (e.g. ANSI/SEC#1 ECDSA, Chinese SM2-ECDSA, etc.)

This attack is very limited.

1. when reconstructing $v$ for check with $r$, the verification public key must be an invalid one provided by the attacker - otherwise, the right hand of the verification equation (i.e. the side involving base point $G$ and verification key $Q_A$) will revert to a valid point in the group generated by $G$.

2. the attacker must convince the verification agent to use the invalid public key - for software agents, the attacker might as well request a certificate using a valid public key; it may be only limitedly meaningful if the target is some kind of HSM.

---

[1]: For example, implementations using the SEC#1 spec's simple formula may be more vulnerable than implementations using some kind of complete formula, as the simple formula doesn't involve $b$ - the constant term in the right-hand of the curve equation.