In ISO/IEC 9796-2 (signature with message recovery), why are some message representatives bigger than the modulus?

Question

I have implemented signature scheme 2 from ISO/IEC 9796-2 in C using OpenSSL for the underlying crypto operations.

To generate the message representative (F in the standard), I just call PKCS1_MGF1() (defined here), and then XOR that to the data to be signed, excluding the hash and the trailer, as the standard says.

Then, I get that blob and run it through RSA_private_encrypt(), setting the padding flag to RSA_NO_PADDING (docs here).

It turns out that, for some inputs, the resulting message representative is bigger than the modulus when converted to a number, which causes RSA_private_encrypt() to fail.

NOTE: the message representative is computed according to the standard, it is always 128 bytes in length (just like the modulus), and it matches the samples provided in Appendix E, so I presume there are no bugs in the way I generate F.

My key is a 1024-bit RSA private key generated with openssl genrsa, with e = 65537, which is an odd number. To the best of my understanding, given that e is odd (which the standard calls v, but they're otherwise the same thing) I can use the "alternative signature production function" (defined in Appendix B.6), which is, well, basically textbook RSA.

Is there anything I have misunderstood that is causing this issue?

If my implementation is correct, then how could I deal with these Fs that are bigger than the modulus?

Answer 1

ISO/IEC 9796-2 is a digital signature scheme giving message recovery. That is: able to embed some of the message in the signature, thus lowering communication or/and storage requirements. There are a lot of options:

• Total recovery is when the message is fully embedded in the fixed-size signature; this limits the message size to a certain capacity. This opposes to Partial recovery were the message size exceeds the capacity: the recoverable part $M_1$ of the message (up to capacity) is embedded in the signature, and the (non-empty) non-recoverable part $M_2$ must somewhat be made available to the verifier independently from the signature (often: along).
• There are 3 schemes:
  • #1 is the original, simplest, but is at least theoretically broken in a chosen-messages setup. It is used in in EMV, Passport, and Digital Tachograph.
  • #2 is inspired by RSASSA-PSS, and uses random salt and MGF1.
  • #3 is simply #2 with no salt.
• The hash used for signature. A common choice for 1024-bit modulus would have been SHA-1, but both are obsolete. A modern choice is SHA-256.
• If that hash is assumed known by the parties (Trailer field option 1) or encoded in the message representative trailer (rare: the motivation for ISO/IEC 9796-2 is compact signed messages, so the overhead of explicitly encoding the hash is unwelcome, plus that's not a good idea from security and simplicity standpoints).
• For schemes #2 and #3: which hash is used for MFG1. Two popular choices are SHA-1, and the same hash as in as the signature.
• Salt size for scheme #2. The more salt, the simpler/stronger the security argument, but the lesser the capacity.
• Odd or even public verification exponent $v$, that is RSA or Rabin. $v=2$ provides the fastest standardized signature verification around.
• Use of "Alternative" signature (RSASP1 and RSAVP1 in PKCS#1, basically big-endian textbook RSA with left-padding with zeroes up to a fixed size), or non-alternative which saves one bit in the signature (I have only seen that with even public exponent, and in test vectors).
• Modulus size: many implementations restrict to modulus a multiple of 8 bits, or a larger power of two.
• Message alignment: the standard supports any size, but many implementations restrict to bytes.

When using byte-aligned message, $k$-bit modulus (multiple of 8), $L_h$-bit hash (multiple of 8), salt size $L_s$ (multiple of 8, possibly 0), Trailer field option 1, the capacity is $k-L_h-L_s-16$ bits.

There are variations in usage:

• How the data to be signed is split in recoverable and non-recoverable parts of the message: the standard defines that the recoverable part of the message is at the beginning, but sometime it is desirable that the start of the payload data is readable before signature verification (e.g. because it includes a key index), therefore the recoverable part of the message sometime is gathered from the end of the payload data.
• Signature before or after the non-recoverable message in signed data.
• Tolerating use of partial message recovery mode when the message size is exactly the capacity.
• Perhaps, tolerating a recoverable message $M_1$ shorter than the capacity including with a non-empty non-recoverable message $M_2$.

---

It turns out that, for some inputs, the resulting message representative is bigger than the modulus when converted to a number.

That's a clear indication that the message representative is not computed correctly.

To generate the message representative ($F$ in the standard), I just call PKCS1_MGF1() and then XOR that to the data to be signed, excluding the hash and the trailer, as the standard says.

Based on that oversimplified description, the problem likely is that the output of MFG1 is considered to be bytes, when the standard wants it one bit less than a multiple of 8 (when everything is byte-aligned). You want to zero the first/left bit of what MFG1 produces before the XOR, or equivalently zero the high-order bit of the message representative after the XOR. That won't lose message data, since that bit is part (or on the left of) the header.

I can use the "alternative signature production function"

Yes. That's by far the most common when public exponent is odd.