Skip to content

aead: require &mut self for seal/open and add TLS AEAD variants - #514

Merged
kornelski merged 2 commits into
cloudflare:masterfrom
janrueth:aead-mut-self-tls-variants
Aug 14, 2026
Merged

aead: require &mut self for seal/open and add TLS AEAD variants#514
kornelski merged 2 commits into
cloudflare:masterfrom
janrueth:aead-mut-self-tls-variants

Conversation

@janrueth

Copy link
Copy Markdown
Contributor

Change all seal/open methods on AeadCtxRef from &self to &mut self.

The TLS-specific AEADs (aes_gcm_tls12, aes_gcm_tls13) are stateful and internally mutate the EVP_AEAD_CTX during seal operations (nonce counter tracking). Using &self allows LLVM to optimize under the assumption that the context is read-only, which can corrupt nonce state and cause cryptographic failures. This was the root cause of cloudflare/quiche#2383.

Using &mut self universally (for both generic and TLS AEADs) is the simplest fix and prevents this class of bug entirely.

Also adds Algorithm constructors for the four TLS GCM variants:

  • aes_128_gcm_tls12 / aes_256_gcm_tls12
  • aes_128_gcm_tls13 / aes_256_gcm_tls13

These enforce strictly monotonic nonces at the AEAD level, matching the TLS 1.2 and TLS 1.3 nonce construction requirements.

Tests cover:

  • Round-trip seal/open for all TLS variants
  • TLS nonce monotonicity enforcement (reuse and backwards rejected)
  • Contrast: generic GCM accepts non-monotonic nonces

@kornelski

Copy link
Copy Markdown
Collaborator

Have you reported this to Google?

The C header uses const EVP_AEAD_CTX *ctx and Google's own bssl-crypto uses &self, so it looks like UB everywhere.

BTW, &mut self does help avoid concurrent access by construction, but I don't think it matters from LLVM optimization perspective in this case, because mutation happens behind a C pointer which doesn't inherit restrictions of references.

@janrueth

Copy link
Copy Markdown
Contributor Author

no, I haven't reported this to google.

Not sure about the optimizer, I followed the argument in cloudflare/quiche#2383

janrueth and others added 2 commits August 14, 2026 16:56
Change all seal/open methods on AeadCtxRef from &self to &mut self.

The TLS-specific AEADs (aes_*_gcm_tls12, aes_*_gcm_tls13) are stateful
and internally mutate the EVP_AEAD_CTX during seal operations (nonce
counter tracking). Using &self allows LLVM to optimize under the
assumption that the context is read-only, which can corrupt nonce state
and cause cryptographic failures. This was the root cause of
cloudflare/quiche#2383.

Using &mut self universally (for both generic and TLS AEADs) is the
simplest fix and prevents this class of bug entirely.

Also adds Algorithm constructors for the four TLS GCM variants:
- aes_128_gcm_tls12 / aes_256_gcm_tls12
- aes_128_gcm_tls13 / aes_256_gcm_tls13

These enforce strictly monotonic nonces at the AEAD level, matching the
TLS 1.2 and TLS 1.3 nonce construction requirements.

Tests cover:
- Round-trip seal/open for all TLS variants
- TLS nonce monotonicity enforcement (reuse and backwards rejected)
- Contrast: generic GCM accepts non-monotonic nonces
@kornelski
kornelski force-pushed the aead-mut-self-tls-variants branch from 83251b9 to eaf9df4 Compare August 14, 2026 15:56
@kornelski

Copy link
Copy Markdown
Collaborator

I've made it backwards-compatible, because upgrade to v5 is causing us so much pain, I don't want to throw v6 into the mix.

@janrueth

Copy link
Copy Markdown
Contributor Author

yeah I wasn't sure about that! sweet, thanks.

@kornelski
kornelski merged commit c270a40 into cloudflare:master Aug 14, 2026
@janrueth
janrueth deleted the aead-mut-self-tls-variants branch August 17, 2026 13:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants