I'm trying to understand the different between AES-CCM and AES-CCM* (as defined in IEEE Std 802.15.4™‐2011, Annex B). I see that there are more restrictions on L and M and also M=0 is allowed:
Is it correct that the max message length is 2^16 = 64K (since L=2)? Isn't it too short?
The encoding of M is quite different since M=0 is allowed. Are there other differences in the calculation?
Is it secure for variable-length messages? How?
CCM* as defined by IEEE Std 802.15.4-2011 is a generalization of CCM as defined by RFC 3610. The generalization is that where RFC 3610 allows the tag lengths M {4, 6, 8, 10, 12, 14, 16} which are encoded in a 3-bit field M' where the value 0 is reserved, CCM* also allows the tag length M=0 and encodes it using the value M'=0 in the tag length field.
There are other specifications of CCM with slightly different constraints. For example, NIST allows the 16-byte header to be formatted differently.
The length L of the message length field can range from 2 to 8. There is no difference between RFC 3610 and CCM* in this respect.
Allowing a zero-length tag makes CCM* a dual-family algorithm: CCM* with a non-empty tag is an AEAD algorithm, but CCM* with an empty tag is an unauthenticated cipher.
CCM* as used in the 802.15.4 LR-WPAN protocol has additional restrictions which are enumerated in §B.3.2 of IEEE Std 802.15.4-2011: that the underlying block cipher is AES, that the plaintext length is encoded in 2 bytes, and that the authentication tag length is one of {0, 4, 8, 16}. These restrictions only apply to the use in 802.15.4, not to CCM* in general.
Since the plaintext length is encoded in 2 bytes, the plaintext cannot be more than 65535 bytes long. This limitation would be too restrictive for some applications, but it's perfectly fine for a 802.15.4 frame, which is limited by the underlying physical transport. The maximum size depends on the transport, and I don't know what the highest-capacity transport is, but Wikipedia states that “most IEEE 802.15.4 PHYs only support frames of up to 127 bytes”.
(Similarly RFC 5116 specifies a 12-byte nonce for CCM, meaning L=3, which is fine for the TLS protocol since its record size field itself is a 3-octet field.)
As for security, 802.15.4 §B.4.3 discusses it, and I'll just summarize the relevant points here. The original security proof for CCM assumes a fixed tag length for a given key. CCM* allows the use of different tag lengths with the same key, which has a known weakness (Rogaway and Wagner, A Critique of CCM, §3.4). CCM* avoids this weakness by ensuring that the encryption mask depends on the authentication tag length (which the general definition of CCM doesn't mandate, but RFC 3610 and CCM* do).
