quicproquo/docs/src/protocol-layers/hybrid-kem.md

Hybrid KEM: X25519 + ML-KEM-768

quicprochat implements a hybrid Key Encapsulation Mechanism that combines classical X25519 Diffie-Hellman with post-quantum ML-KEM-768 (FIPS 203). The hybrid construction ensures that the system remains secure even if one of the two components is broken: X25519 protects against failures in ML-KEM, and ML-KEM protects against quantum computers breaking X25519.

The implementation lives in quicprochat-core/src/hybrid_kem.rs. It is fully implemented and tested but not yet integrated into the MLS ciphersuite -- integration is planned for the M5 milestone. Currently, the module can be used as a standalone envelope encryption layer to wrap MLS payloads in an outer post-quantum-resistant encryption before they transit the network.

Design approach

The hybrid KEM follows the combiner approach from draft-ietf-tls-hybrid-design. The core idea:

1. Perform both a classical key exchange (X25519) and a post-quantum key encapsulation (ML-KEM-768) against the recipient's public keys.
2. Combine the two shared secrets into a single AEAD key using HKDF.
3. Encrypt the payload with ChaCha20-Poly1305 using the derived key.

This ensures:

• IND-CCA2 security if either X25519 or ML-KEM-768 is secure.
• No reliance on a single hardness assumption.
• Graceful degradation: if ML-KEM is found to have a flaw, classical X25519 still protects the data.

Component algorithms

Component	Algorithm	Size	Security Level
Classical KEM	X25519 ECDH	32-byte keys, 32-byte shared secret	128-bit classical
Post-quantum KEM	ML-KEM-768 (FIPS 203)	1184-byte EK, 2400-byte DK, 1088-byte CT, 32-byte SS	NIST Level 3 (128-bit quantum)
Key derivation	HKDF-SHA256	32-byte output key, 12-byte output nonce	256-bit PRF security
Symmetric encryption	ChaCha20-Poly1305	32-byte key, 12-byte nonce, 16-byte tag	256-bit security

ML-KEM-768 constants

These constants are defined in hybrid_kem.rs and match FIPS 203:

Constant	Value	Description
MLKEM_EK_LEN	1,184 bytes	Encapsulation (public) key size
MLKEM_DK_LEN	2,400 bytes	Decapsulation (private) key size
MLKEM_CT_LEN	1,088 bytes	Ciphertext size
Shared secret	32 bytes	Output of encapsulate/decapsulate

ML-KEM-768 was chosen over ML-KEM-512 (NIST Level 1) for a stronger security margin and over ML-KEM-1024 (NIST Level 5) because the additional key/ciphertext sizes are not justified for 128-bit target security.

Wire format

Every hybrid-encrypted payload is packaged as a self-describing envelope:

┌─────────┬──────────────────┬──────────────────┬──────────────┬──────────────────┐
│ version │ x25519_eph_pk    │ mlkem_ct         │ aead_nonce   │ aead_ct          │
│ (1 B)   │ (32 B)           │ (1088 B)         │ (12 B)       │ (variable)       │
└─────────┴──────────────────┴──────────────────┴──────────────┴──────────────────┘

Field	Offset	Size	Description
version	0	1 byte	Envelope version. Currently 0x01.
x25519_eph_pk	1	32 bytes	Ephemeral X25519 public key (generated fresh per encryption).
mlkem_ct	33	1,088 bytes	ML-KEM-768 ciphertext (encapsulation of the PQ shared secret).
aead_nonce	1,121	12 bytes	ChaCha20-Poly1305 nonce (derived from HKDF).
aead_ct	1,133	variable	ChaCha20-Poly1305 ciphertext + 16-byte authentication tag.

The total header (HEADER_LEN) is 1 + 32 + 1088 + 12 = 1,133 bytes. The minimum valid envelope is HEADER_LEN + 16 = 1,149 bytes (16 bytes for the AEAD tag on an empty plaintext).

The version byte enables future format evolution. Decryption rejects any version other than 0x01 with HybridKemError::UnsupportedVersion.

Key derivation

The two shared secrets are combined via HKDF-SHA256 with domain separation:

ikm  = X25519_shared_secret(32 bytes) || ML-KEM_shared_secret(32 bytes)
salt = [] (empty)

key   = HKDF-SHA256(salt, ikm, info="quicprochat-hybrid-v1",       L=32)
nonce = HKDF-SHA256(salt, ikm, info="quicprochat-hybrid-nonce-v1", L=12)

The implementation in derive_aead_material():

fn derive_aead_material(x25519_ss: &[u8], mlkem_ss: &[u8]) -> (Key, Nonce) {
    let mut ikm = Zeroizing::new(vec![0u8; x25519_ss.len() + mlkem_ss.len()]);
    ikm[..x25519_ss.len()].copy_from_slice(x25519_ss);
    ikm[x25519_ss.len()..].copy_from_slice(mlkem_ss);

    let hk = Hkdf::<Sha256>::new(None, &ikm);

    let mut key_bytes = Zeroizing::new([0u8; 32]);
    hk.expand(b"quicprochat-hybrid-v1", &mut *key_bytes).unwrap();

    let mut nonce_bytes = [0u8; 12];
    hk.expand(b"quicprochat-hybrid-nonce-v1", &mut nonce_bytes).unwrap();

    (*Key::from_slice(&*key_bytes), *Nonce::from_slice(&nonce_bytes))
}

Key design decisions:

• Concatenation order: X25519 shared secret first, ML-KEM shared secret second. This is consistent with the draft-ietf-tls-hybrid-design convention.
• Separate info strings: The key and nonce are derived with different HKDF info strings to ensure domain separation. Using the same info string for both would be a cryptographic error.
• Zeroization: The concatenated IKM and the derived key bytes are wrapped in Zeroizing to ensure they are cleared from memory when dropped.
• Empty salt: HKDF is used in extract-then-expand mode with no salt. The IKM already has high entropy from both DH operations.

HybridKeypair

Each peer holds a HybridKeypair combining classical and post-quantum key material:

pub struct HybridKeypair {
    x25519_sk: StaticSecret,        // 32 bytes
    x25519_pk: X25519Public,        // 32 bytes
    mlkem_dk: DecapsulationKey<MlKem768Params>,  // 2400 bytes
    mlkem_ek: EncapsulationKey<MlKem768Params>,  // 1184 bytes
}

Generation

pub fn generate() -> Self {
    let x25519_sk = StaticSecret::random_from_rng(OsRng);
    let x25519_pk = X25519Public::from(&x25519_sk);
    let (mlkem_dk, mlkem_ek) = MlKem768::generate(&mut OsRng);
    // ...
}

Both key pairs are generated from the OS CSPRNG (OsRng). The X25519 key uses x25519-dalek's StaticSecret (not EphemeralSecret) because the keypair is long-lived and must be stored.

Serialisation

For persistence, HybridKeypairBytes provides a serialisable form:

pub struct HybridKeypairBytes {
    pub x25519_sk: [u8; 32],
    pub mlkem_dk: Vec<u8>,   // 2400 bytes
    pub mlkem_ek: Vec<u8>,   // 1184 bytes
}

Round-trip: keypair.to_bytes() serialises, HybridKeypair::from_bytes(&bytes) reconstructs. The ML-KEM keys are reconstructed using DecapsulationKey::from_bytes() and EncapsulationKey::from_bytes(), which accept Array types converted from slices.

Public key extraction

The public portion is extracted for distribution to peers:

pub struct HybridPublicKey {
    pub x25519_pk: [u8; 32],
    pub mlkem_ek: Vec<u8>,   // 1184 bytes
}

HybridPublicKey can be serialised to a single byte blob: x25519_pk(32) || mlkem_ek(1184) = 1,216 bytes total. This is uploaded to the server via the uploadHybridKey RPC and fetched by peers via fetchHybridKey.

Encryption flow: hybrid_encrypt

pub fn hybrid_encrypt(
    recipient_pk: &HybridPublicKey,
    plaintext: &[u8],
) -> Result<Vec<u8>, HybridKemError>

Step-by-step:

1. Ephemeral X25519 DH: Generate a fresh EphemeralSecret, compute the X25519 shared secret with the recipient's static public key. The ephemeral secret is consumed (moved) by diffie_hellman() and cannot be reused.

2. ML-KEM-768 encapsulation: Reconstruct the recipient's EncapsulationKey from the public key bytes, then call encapsulate(&mut OsRng). This produces a ciphertext (1,088 bytes) and a shared secret (32 bytes).

3. Key derivation: Call derive_aead_material() with both shared secrets to produce a 32-byte ChaCha20-Poly1305 key and a 12-byte nonce.

4. AEAD encryption: Encrypt the plaintext with ChaCha20Poly1305::encrypt(). The output includes the 16-byte authentication tag.

5. Envelope assembly: Concatenate version || x25519_eph_pk || mlkem_ct || nonce || aead_ct.

Decryption flow: hybrid_decrypt

pub fn hybrid_decrypt(
    keypair: &HybridKeypair,
    envelope: &[u8],
) -> Result<Vec<u8>, HybridKemError>

Step-by-step:

1. Envelope parsing: Verify minimum length (HEADER_LEN + 16), check version byte (0x01), then extract the five fields by offset.

2. X25519 DH: Compute the shared secret using the recipient's static private key (keypair.x25519_sk) and the sender's ephemeral public key from the envelope.

3. ML-KEM-768 decapsulation: Convert the ciphertext bytes to the Array type expected by DecapsulationKey::decapsulate(), then decapsulate to recover the shared secret.

4. Key derivation: Same derive_aead_material() call as encryption, producing the same key and nonce (the nonce from the envelope is used for AEAD decryption, not the derived one -- actually, both are identical because the derivation is deterministic from the same shared secrets).

5. AEAD decryption: Decrypt and authenticate the ciphertext with ChaCha20Poly1305::decrypt().

Error handling

The HybridKemError enum covers all failure modes:

Variant	Meaning
EncryptionFailed	AEAD encryption failed (should not happen with valid inputs)
DecryptionFailed	AEAD decryption failed -- wrong recipient key or tampered ciphertext
UnsupportedVersion(u8)	Envelope version byte is not 0x01
TooShort(usize)	Envelope is shorter than HEADER_LEN + 16 bytes
InvalidMlKemKey	ML-KEM encapsulation key bytes are malformed
MlKemDecapsFailed	ML-KEM decapsulation failed -- tampered ciphertext or wrong key

The tests in hybrid_kem.rs verify:

• Round-trip encrypt/decrypt with correct keys.
• Decryption with wrong key fails (DecryptionFailed).
• Tampered AEAD ciphertext fails (DecryptionFailed).
• Tampered ML-KEM ciphertext fails (either MlKemDecapsFailed or DecryptionFailed).
• Tampered X25519 ephemeral public key fails (DecryptionFailed).
• Unsupported version is rejected.
• Too-short envelope is rejected.
• Keypair and public key serialisation round-trip.
• Large payloads (50 KB) round-trip successfully.

Current status and roadmap

The hybrid KEM module is:

• Implemented: All types, encryption, decryption, serialisation, and key management are complete.
• Tested: Comprehensive unit tests cover all success and failure paths.
• Server-supported: The NodeService RPC interface includes uploadHybridKey and fetchHybridKey methods. The server stores hybrid public keys in its FileBackedStore.
• Not yet integrated into MLS: The MLS ciphersuite (MLS_128_DHKEMX25519_AES128GCM_SHA256_Ed25519) uses classical DHKEM(X25519). Replacing it with a hybrid KEM requires either:
  • A custom openmls ciphersuite that uses the hybrid KEM for HPKE (complex, requires forking openmls).
  • An outer encryption layer that wraps MLS messages in a hybrid envelope before delivery (simpler, less tightly integrated).

The M5 milestone will integrate the hybrid KEM, likely as an outer encryption layer. Until then, MLS application data is protected by classical X25519 ECDH (128-bit security against classical computers, vulnerable to quantum computers).

The post-quantum gap in the transport layer (QUIC + TLS 1.3) is a separate concern -- TLS 1.3 key exchange uses classical ECDHE and does not yet include post-quantum key agreement.

Security analysis

Hybrid security guarantee

The combiner construction ensures that an attacker must break both X25519 and ML-KEM-768 to recover the plaintext. Specifically:

• A classical attacker cannot break X25519 (ECDLP is hard on Curve25519) and therefore cannot derive the AEAD key, regardless of whether they can break ML-KEM.
• A quantum attacker with a cryptographically relevant quantum computer could break X25519 via Shor's algorithm but cannot break ML-KEM-768 (based on the Module-LWE problem, believed to be quantum-resistant).
• An attacker who discovers a flaw in ML-KEM still faces X25519, which provides 128-bit classical security.

Key reuse

The X25519 component of the hybrid keypair is a StaticSecret (long-lived), not an EphemeralSecret. This is safe because:

• Each encryption uses a fresh EphemeralSecret for the sender's X25519 contribution.
• The static secret is only used in the DH computation with the ephemeral public key; it never appears in the wire format.
• The ML-KEM encapsulation also generates fresh randomness per encryption.

Nonce handling

The AEAD nonce is derived deterministically from the shared secrets via HKDF. Since each encryption uses a fresh ephemeral X25519 key and fresh ML-KEM randomness, the shared secrets (and therefore the derived nonce) are unique per encryption with overwhelming probability. Nonce reuse would require both:

• The same ephemeral X25519 key (probability 2^{-256}).
• The same ML-KEM encapsulation randomness (probability 2^{-256}).

Crate dependencies

Crate	Version	Role
ml-kem	0.2	ML-KEM-768 (FIPS 203) implementation
x25519-dalek	2	X25519 ECDH (with static_secrets feature)
chacha20poly1305	0.10	AEAD symmetric encryption
hkdf	0.12	HKDF-SHA256 key derivation
sha2	0.10	SHA-256 (used by HKDF)
zeroize	1	Secure memory clearing for key material
rand	0.8	OsRng for CSPRNG
serde	1	Serialisation of keypair and public key types

Further reading

• Post-Quantum Readiness -- Broader discussion of quicprochat's PQ strategy.
• MLS (RFC 9420) -- The MLS layer that the hybrid KEM will wrap.
• Key Lifecycle and Zeroization -- How hybrid key material is managed and cleared.
• Threat Model -- Where hybrid KEM fits in the overall threat model.
• Milestone Tracker -- M5 milestone for hybrid KEM integration into MLS.