ML-KEM, Explained: How Kyber Got Into Your TLS Handshake

From encryption to encapsulation: a different way to deliver a key

Establishing a symmetric TLS session key always comes back to one problem: how do two parties agree on a secret over a public channel? Classical answers came in two generations — the RSA era encrypted a pre-master secret with the server's public key; the ECDH era had both sides publish ephemeral keys and run a Diffie-Hellman exchange. Both generations share a foundation — factoring and elliptic-curve discrete log — that collapses under Shor's algorithm (background in the previous post).

ML-KEM is the third generation: a key encapsulation mechanism. Three steps:

1. Keygen: produce a public and private key (as before)
2. Encapsulate: anyone holding the public key runs the encapsulation algorithm, which outputs a ciphertext and a fresh random shared secret
3. Decapsulate: the private-key holder recovers the same shared secret from the ciphertext

The difference from RSA-OAEP's "pick a key, then encrypt and send it" is who controls the key's birth: in a KEM the shared secret is derived inside the algorithm, so you never choose a key and then figure out how to ship it — eliminating an entire class of "the key wasn't random enough" engineering incidents.

Where Kyber came from: an eight-year competition

ML-KEM's predecessor, Kyber, emerged from NIST's post-quantum standardization competition (launched 2016), was selected in July 2022 as the basis for the KEM standard, and was published in August 2024 as FIPS 203. The ML stands for module lattice — the mathematical ground shifts from factoring to noisy linear systems over lattices, where quantum algorithms currently hold no structural advantage. You don't need lattice math to remember its engineering profile: fast (software performance in the same league as ECDH), big keys (an order of magnitude above X25519), quantum-resistant.

Three parameter sets and their sizes: the root of all friction

TLS deployments today mostly run ML-KEM-768. Public key plus ciphertext totals ~2.3KB — over thirty times X25519. That number drives nearly all the engineering friction below.

The hybrid handshake: not a transition, a design

What browsers and servers run today is not "pure ML-KEM" but the X25519MLKEM768 hybrid group:

1. The ClientHello's key_share carries both an X25519 public key (32B) and an ML-KEM-768 public key (1,184B)
2. The server completes key exchange for both algorithms, yielding two shared secrets
3. The TLS 1.3 key schedule (HKDF-based) derives the session key from the two secrets concatenated together

The key property: an attacker must break both the lattice problem and the elliptic-curve problem to recover the session. If quantum breaks the classical side, ML-KEM holds; if ML-KEM turns out to have a weakness, X25519 holds. "Hybrid" is often misread as a transitional compromise — it's a redundancy structure that won't be retired anytime soon.

The ClientHello-bloat middlebox trap

The extra 1.1KB pushes the ClientHello past a single TCP segment. Chrome hit this for real while rolling out X25519Kyber768 (the hybrid's predecessor): some enterprise firewalls dropped oversized or "unfamiliar" handshakes outright. Hence the deployment order — canary at the CDN/edge first, watch handshake success rates, then full rollout. That's why the migration checklist lists "canary + handshake monitoring" as mandatory.

How to turn it on (October 2026)

• Browsers: Chrome and Firefox already support it by default — nothing to configure
• OpenSSL 3.5+: ships the X25519MLKEM768 group
• nginx (linked against OpenSSL 3.5+): ssl_ecdh_curve X25519MLKEM768:X25519; (keep X25519 as fallback)
• CDNs: Cloudflare and peers expose toggles; if your origin sits behind a CDN, upgrading the origin buys little — negotiation happens at the edge
• To verify: check the key-exchange group name in your browser's developer tools Security panel, or inspect the negotiated group with a hybrid-capable openssl s_client. Your certificate itself (visible in the certificate decoder) is still RSA/ECDSA for now — moving certificates to PQC is the slower timeline, because signatures wait on the ML-DSA ecosystem

Implementer's note

There's a concrete reason this site has no PQC tool yet: WebCrypto still has no ML-KEM interface (the crypto.subtle roster remains RSA/ECDH/AES), so running ML-KEM in a browser means a wasm build of liboqs — multi-megabyte payloads, which is hostile to "open and use" online tools. Writing the RSA tool, I left a comparison point: encrypting a 32-byte key with RSA-OAEP costs hundreds of bytes of overhead, and ML-KEM-768 encapsulates a similar-sized secret into 1,088 bytes — ciphertext size is the most tangible difference between KEM and DH families. Once WebCrypto ships native support, ML-KEM goes into the tool matrix and that comparison becomes playable.

Related reading

• Harvest now, decrypt later: whose data moves first
• Post-quantum migration self-check: 12 items
• RSA encryption, explained properly
• HMAC and key derivation