Skip to content

sigstore: Cache verify_image results across policy evaluations#3136

Merged
simonbaird merged 1 commit into
conforma:mainfrom
arewm:ec-1545/sigstore-verify-image-cache
Jul 7, 2026
Merged

sigstore: Cache verify_image results across policy evaluations#3136
simonbaird merged 1 commit into
conforma:mainfrom
arewm:ec-1545/sigstore-verify-image-cache

Conversation

@arewm

@arewm arewm commented Feb 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add sync.Map + singleflight.Group cache for ec.sigstore.verify_image results keyed by ref|hash(opts)
  • OPA's Memoize only deduplicates within a single Eval() call; this cache persists across component evaluations
  • Prevents redundant signature verification when many components share task bundles (e.g., 100 components using git-clone)
  • Follows the caching pattern established by manifestCache/descriptorCache in internal/rego/oci/oci.go

Test plan

  • go build ./internal/rego/sigstore/ compiles cleanly
  • Verify caching behavior with multiple components sharing task bundles
  • Benchmark signature verification with/without cache on a multi-component snapshot

Ref: EC-1545

🤖 Generated with Claude Code

@arewm arewm force-pushed the ec-1545/sigstore-verify-image-cache branch from 3950ea8 to b0710ba Compare February 28, 2026 02:25
@github-actions github-actions Bot added size: L and removed size: S labels Feb 28, 2026
@arewm arewm force-pushed the ec-1545/sigstore-verify-image-cache branch from b0710ba to d029230 Compare February 28, 2026 02:30
@codecov

codecov Bot commented Feb 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

Flag Coverage Δ
acceptance 53.46% <77.77%> (+0.03%) ⬆️
generative 17.35% <0.00%> (-0.03%) ⬇️
integration 28.63% <0.00%> (-0.05%) ⬇️
unit 71.80% <100.00%> (+0.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
internal/rego/sigstore/sigstore.go 93.41% <100.00%> (+0.46%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arewm arewm marked this pull request as ready for review February 28, 2026 02:51
@qodo-code-review

Copy link
Copy Markdown
Contributor

Review Summary by Qodo

Cache verify_image results across policy evaluations

✨ Enhancement

Grey Divider

Walkthroughs

Description
• Add caching layer for verify_image results using sync.Map and singleflight.Group
• Cache key combines image reference and SHA256 hash of verification options
• Prevents redundant signature verification across multiple policy evaluations
• Add comprehensive test coverage for cache hit/miss scenarios
Diagram
flowchart LR
  A["sigstoreVerifyImage called"] --> B{"Cache hit?"}
  B -->|Yes| C["Return cached result"]
  B -->|No| D["Use singleflight dedup"]
  D --> E["doVerifyImage execution"]
  E --> F["Store result in cache"]
  F --> G["Return result"]
  C --> H["Result"]
  G --> H
Loading

Grey Divider

File Changes

1. internal/rego/sigstore/sigstore.go ✨ Enhancement +45/-1

Implement verify_image caching with singleflight deduplication

• Add sync.Map and singleflight.Group package imports for caching infrastructure
• Introduce global verifyImageCache and verifyImageFlight variables for cross-evaluation caching
• Refactor sigstoreVerifyImage to check cache first, then use singleflight for deduplication
• Extract core verification logic into new doVerifyImage function
• Add buildCacheKey function that combines ref and SHA256 hash of options
• Add ClearCaches utility function for testing purposes

internal/rego/sigstore/sigstore.go


2. internal/rego/sigstore/sigstore_test.go 🧪 Tests +86/-0

Add comprehensive cache behavior test coverage

• Add ClearCaches() call in existing TestSigstoreVerifyImage test setup
• Add new TestVerifyImageCache test function with four test cases
• Test cache hit when same ref and opts are used multiple times
• Test cache miss when different ref is used with same opts
• Test cache miss when different opts are used with same ref
• Test that ClearCaches forces re-verification of previously cached results

internal/rego/sigstore/sigstore_test.go


Grey Divider

Qodo Logo

@qodo-code-review

qodo-code-review Bot commented Feb 28, 2026

Copy link
Copy Markdown
Contributor

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider


Action required

1. Caches failed verifications 🐞 Bug ⛯ Reliability
Description
verify_image stores results in the global cache unconditionally, including failure results
(success=false) returned via signatureFailedResult. This can turn transient verification errors
into persistent failures for the lifetime of the process, preventing retries in later policy
evaluations/workers.
Code

internal/rego/sigstore/sigstore.go[R141-150]

+	resultIface, err, _ := verifyImageFlight.Do(cacheKey, func() (interface{}, error) {
+		return doVerifyImage(bctx, logger, uri, optsTerm)
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	result := resultIface.(*ast.Term)
+	verifyImageCache.Store(cacheKey, result)
+	return result, nil
Evidence
sigstoreVerifyImage stores whatever doVerifyImage returns into verifyImageCache without
checking if verification succeeded. doVerifyImage converts errors into a normal result term via
signatureFailedResult (not a Go error), so failures will be cached too. In contrast, other global
caches (OCI) only store successful results and return nil on errors, avoiding caching transient
failures.

internal/rego/sigstore/sigstore.go[131-150]
internal/rego/sigstore/sigstore.go[156-174]
internal/rego/sigstore/sigstore.go[350-379]
internal/rego/oci/oci.go[570-623]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`ec.sigstore.verify_image` unconditionally caches the returned result term, including failure results. This can make transient verification failures persist across later policy evaluations within the same process.

### Issue Context
`doVerifyImage` converts errors into a normal `{success:false, errors:[...]}` term via `signatureFailedResult`, so the cache sees failures as valid values to store.

### Fix Focus Areas
- internal/rego/sigstore/sigstore.go[131-150]
- internal/rego/sigstore/sigstore.go[350-379]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Truncated opts hash key 🐞 Bug ⛨ Security
Description
Cache key uses only the first 8 bytes (64 bits) of SHA-256 over optsTerm.String(). While
collisions are unlikely in typical runs, a collision would cause results computed with different
verification options to be reused incorrectly.
Code

internal/rego/sigstore/sigstore.go[R180-184]

+func buildCacheKey(ref string, optsTerm *ast.Term) string {
+	h := sha256.Sum256([]byte(optsTerm.String()))
+	optsHash := hex.EncodeToString(h[:8]) // First 8 bytes is enough for dedup
+	return ref + "|" + optsHash
+}
Evidence
buildCacheKey intentionally truncates SHA-256 to 8 bytes before hex encoding, reducing the
collision resistance of the cache key and risking incorrect cache hits across different option sets.

internal/rego/sigstore/sigstore.go[180-184]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`buildCacheKey` truncates SHA-256 to 8 bytes, which increases the chance of key collisions and incorrect cache hits across different verification options.

### Issue Context
Even if collisions are rare in practice, a collision would reuse a verification result computed under different trust/verification settings.

### Fix Focus Areas
- internal/rego/sigstore/sigstore.go[180-184]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Error-return contract break 🐞 Bug ⛯ Reliability
Description
This file documents that rego builtins must not return Go errors, but the new wrapper returns `nil,
err when singleflight.Do` returns an error. This risks inconsistent behavior vs the rest of the
file’s patterns and may cause evaluation failures in unexpected error cases.
Code

internal/rego/sigstore/sigstore.go[R141-146]

+	resultIface, err, _ := verifyImageFlight.Do(cacheKey, func() (interface{}, error) {
+		return doVerifyImage(bctx, logger, uri, optsTerm)
+	})
+	if err != nil {
+		return nil, err
+	}
Evidence
The file header explicitly states these builtins should not return errors. The new wrapper adds an
explicit error-return branch if singleflight returns an error, which conflicts with the documented
behavior and with the rest of the function’s approach of returning structured failure terms.

internal/rego/sigstore/sigstore.go[17-19]
internal/rego/sigstore/sigstore.go[141-146]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`sigstoreVerifyImage` now has a code path that returns a non-nil Go error, contradicting the file’s documented builtin contract.

### Issue Context
This codebase intentionally returns structured `{success:false, errors:[...]}` terms instead of Go errors for rego builtins.

### Fix Focus Areas
- internal/rego/sigstore/sigstore.go[17-19]
- internal/rego/sigstore/sigstore.go[141-150]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

ⓘ The new review experience is currently in Beta. Learn more

Grey Divider

Qodo Logo

@arewm arewm force-pushed the ec-1545/sigstore-verify-image-cache branch from d029230 to c504a19 Compare February 28, 2026 02:55
Comment thread internal/rego/sigstore/sigstore.go Outdated
@joejstuart

Copy link
Copy Markdown
Contributor

Here's the performance stats I ran. We should try and look for improvements. If we can't, I think we'll have to send out communication on having to up memory requirements.

Current PR

Peak RSS (MB)

Comp w=1 w=35
1c 433.7MB
50c 711.3MB 5574.2MB
191c 5648.3MB

Wall Time (s)

Comp w=1 w=35
1c 33.5s
50c 1155.9s 103.0s
191c 363.6s

main branch

Peak RSS (MB)

Comp w=1 w=35
1c 358.2MB
50c 613.7MB 3136.7MB
191c 3988.8MB

Wall Time (s)

Comp w=1 w=35
1c 33.0s
50c 1005.4s 84.2s
191c 288.4s

@arewm

arewm commented Mar 2, 2026

Copy link
Copy Markdown
Contributor Author

@joejstuart , thanks for that analysis. Is this increase just if we are adding the signature verification? So users would not need to increase memory requirements unless they used that functionality?

If we were trying to keep ~200 components to finish evaluation in under 5 minutes, then we also fail that goal with the associated policy changes.

@joejstuart

Copy link
Copy Markdown
Contributor

@joejstuart , thanks for that analysis. Is this increase just if we are adding the signature verification? So users would not need to increase memory requirements unless they used that functionality?

If we were trying to keep ~200 components to finish evaluation in under 5 minutes, then we also fail that goal with the associated policy changes.

I tested it with the corresponding policy change. Should I test without it?

@arewm

arewm commented Mar 2, 2026

Copy link
Copy Markdown
Contributor Author

I think we needed the test with the policy change to understand the impact of that change. But it would be good to test it without as well to ensure that we don't have regression when that policy isn't being used.

@joejstuart

Copy link
Copy Markdown
Contributor

Without the policy change

Peak RSS (MB)

Comp w=1 w=35
1c 345.4MB
50c 621.9MB 3003.0MB
191c 4111.2MB

Wall Time (s)

Comp w=1 w=35
1c 29.3s
50c 928.7s 81.6s
191c 276.2s

@arewm

arewm commented Mar 3, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, @joejstuart . These are just single runs or averages over multiple runs?

@arewm

arewm commented Mar 5, 2026

Copy link
Copy Markdown
Contributor Author

@joejstuart @simonbaird , would you be okay with this change along with conforma/policy#1680 ?

@joejstuart

joejstuart commented May 1, 2026

Copy link
Copy Markdown
Contributor

Now I'm not seeing much of a performance difference between this and main.

@simonbaird

Copy link
Copy Markdown
Member

/ok-to-test

@simonbaird simonbaird force-pushed the ec-1545/sigstore-verify-image-cache branch from c8a13b9 to c97787f Compare May 14, 2026 19:56
@coderabbitai

coderabbitai Bot commented May 14, 2026

Copy link
Copy Markdown

Caution

Review failed

The head commit changed during the review from 6692db6 to e145df4.

📝 Walkthrough

Walkthrough

The PR adds verification result caching to sigstore.VerifyImage builtin. A SHA-512 hash of the options term combined with the image reference forms the cache key. Concurrent requests for the same key are deduplicated via singleflight to avoid redundant verifications. The ClearCaches() function enables test isolation, and comprehensive tests validate cache behavior across reference changes, option variations, and cache clearing.

Changes

Sigstore verification caching

Layer / File(s) Summary
Cache infrastructure and exports
internal/rego/sigstore/sigstore.go
Imports for hashing (crypto/sha512, encoding/hex) and singleflight are added. Package-level state includes verifyImageCache (sync.Map) for results and verifyImageFlight (singleflight.Group) for deduplication. Exported ClearCaches() resets the cache for test isolation.
Cache key generation and builtin caching
internal/rego/sigstore/sigstore.go
buildCacheKey() helper hashes the options term with SHA-512 and appends to image ref. sigstoreVerifyImage() builtin is reworked to compute cache keys, return cached results on hit, use singleflight to deduplicate concurrent requests for the same key, and store newly computed results back into cache.
Test isolation and cache validation
internal/rego/sigstore/sigstore_test.go
Existing test clears cache before subtests to prevent result leakage. New TestVerifyImageCache() validates cache hits on repeated ref+options (single VerifyImageSignatures call), misses when ref or options differ, and forced re-verification after ClearCaches() via mock call counts.

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and concisely describes the main change: adding caching for verify_image results across policy evaluations.
Description check ✅ Passed The description is directly related to the changeset, explaining the caching mechanism, motivation, and test plan for the sigstore verification cache implementation.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@simonbaird

Copy link
Copy Markdown
Member

Rebased to see if it gets closer to green.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
internal/rego/sigstore/sigstore.go (1)

480-483: ⚡ Quick win

ClearCaches reassigning the sync.Map value is racy.

verifyImageCache is a package-level value (not a pointer); reassigning it while other goroutines may still be calling Load/Store on it is a data race. It happens to work in serial test runs, but go test can run tests from other packages or the same package in parallel, and the singleflight callback above also touches it from inner goroutines. Prefer clearing in place.

-func ClearCaches() {
-	verifyImageCache = sync.Map{}
-}
+func ClearCaches() {
+	verifyImageCache.Range(func(k, _ any) bool {
+		verifyImageCache.Delete(k)
+		return true
+	})
+}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@internal/rego/sigstore/sigstore.go` around lines 480 - 483, ClearCaches
currently reassigns the package-level sync.Map verifyImageCache which is racy;
instead clear the map in place to avoid races: in ClearCaches iterate the
existing verifyImageCache (use verifyImageCache.Range) and delete each key
(verifyImageCache.Delete) so any concurrent Load/Store operations remain safe;
update the ClearCaches function to perform in-place deletion rather than
reassigning verifyImageCache.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/rego/sigstore/sigstore_test.go`:
- Around line 224-241: The newFakeContext helper misses a mock for
fake.FakeClient.HasBundles which doVerifyImage calls before
VerifyImageSignatures; update newFakeContext to add an expectation on c (the
*fake.FakeClient) for HasBundles (use mock.Anything for args) and return the
same boolean/error used in other tests (e.g., false, nil) so the HasBundles call
is explicitly verified before VerifyImageSignatures is exercised.

In `@internal/rego/sigstore/sigstore.go`:
- Around line 140-149: The current wrapper unconditionally stores the term
returned by doVerifyImage in verifyImageCache via verifyImageCache.Store,
causing transient verification failures to become permanently cached; change the
flow so only cacheable (deterministic/success) results are stored: have
doVerifyImage indicate cacheability (e.g., return a second bool or make the
ast.Term include a "success"/"cacheable" flag) and update the
verifyImageFlight.Do callback/result handling to inspect that flag and call
verifyImageCache.Store only when cacheable/successful; keep verifyImageFlight.Do
and verifyImageCache usage but skip storing non-cacheable failure terms so
transient errors are not persisted.

---

Nitpick comments:
In `@internal/rego/sigstore/sigstore.go`:
- Around line 480-483: ClearCaches currently reassigns the package-level
sync.Map verifyImageCache which is racy; instead clear the map in place to avoid
races: in ClearCaches iterate the existing verifyImageCache (use
verifyImageCache.Range) and delete each key (verifyImageCache.Delete) so any
concurrent Load/Store operations remain safe; update the ClearCaches function to
perform in-place deletion rather than reassigning verifyImageCache.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 2a574443-5227-40c2-9e2c-e9fffd56ad98

📥 Commits

Reviewing files that changed from the base of the PR and between 3743934 and c97787f.

📒 Files selected for processing (2)
  • internal/rego/sigstore/sigstore.go
  • internal/rego/sigstore/sigstore_test.go

Comment thread internal/rego/sigstore/sigstore_test.go
Comment thread internal/rego/sigstore/sigstore.go
@simonbaird

Copy link
Copy Markdown
Member

/ok-to-test

@st3penta st3penta marked this pull request as draft June 10, 2026 12:42
@arewm

arewm commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

@st3penta , did you move this to draft because there is something conflicting or just because there hasn't been much review volume?

Are there any concerns with merging this?

@st3penta

Copy link
Copy Markdown
Contributor

hi @arewm, i moved it to draft because it seemed a bit stale, but talking with the team we agreed we should instead get it green and reviewed, and merge it

@st3penta st3penta marked this pull request as ready for review June 16, 2026 08:02
@simonbaird

simonbaird commented Jun 26, 2026

Copy link
Copy Markdown
Member

I rebased on all the commits from #3357 to see if it goes green. Would probably want to rebase again on updated main once 3357 is merged.

Update: Done now.

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 26, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 7:59 PM UTC · Completed 8:12 PM UTC
Commit: 47d3320 · View workflow run →

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added go Pull requests that update Go code docker Pull requests that update Docker code dependencies Pull requests that update a dependency file labels Jun 26, 2026
@simonbaird

Copy link
Copy Markdown
Member

/ok-to-test

@simonbaird simonbaird force-pushed the ec-1545/sigstore-verify-image-cache branch from 975e565 to 0109754 Compare July 6, 2026 13:26
@github-actions github-actions Bot added size: L and removed size: XXL labels Jul 6, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Jul 6, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 1:29 PM UTC · Completed 1:34 PM UTC
Commit: 47d3320 · View workflow run →

@fullsend-ai-review fullsend-ai-review Bot dismissed stale reviews from themself July 6, 2026 13:34

Superseded by updated review

@simonbaird

Copy link
Copy Markdown
Member

/ok-to-test

@simonbaird simonbaird force-pushed the ec-1545/sigstore-verify-image-cache branch from 0109754 to 6692db6 Compare July 6, 2026 17:37
@simonbaird

Copy link
Copy Markdown
Member

Addressing fullsend's comments:

  • Added a comment about the possibility of caching a transient network problem. I don't think it's worthwhile to do the work to actually decide what errors should be cached and which shouldn't, though it is a reasonable idea.
  • Added the missing ClearCache() in some tests.
  • Removed the sha512 cache key in favor of sha256.

@simonbaird

Copy link
Copy Markdown
Member

/ok-to-test

@simonbaird

Copy link
Copy Markdown
Member

Should be good to merge. If it goes green, I'll seek an ack and aim to get it merged.

@simonbaird simonbaird marked this pull request as ready for review July 6, 2026 17:44
Add a sync.Map + singleflight.Group cache for ec.sigstore.verify_image
results keyed by ref + opts hash. OPA's Memoize only deduplicates within
a single Eval() call, but components are validated in separate Eval()
calls. Since bundles are pinned by digest, verification results are
stable and can be safely cached for the process lifetime.

This prevents redundant signature verification when many components
share the same task bundles (e.g., 100 components using git-clone).

Ref: EC-1545

Assisted-by: Claude Code (Sonnet 4.6)
Signed-off-by: arewm <arewm@users.noreply.github.com>
@simonbaird simonbaird force-pushed the ec-1545/sigstore-verify-image-cache branch from 6692db6 to e145df4 Compare July 6, 2026 17:44
@simonbaird

Copy link
Copy Markdown
Member

/ok-to-test

@simonbaird

Copy link
Copy Markdown
Member

I just noticed there were some other fullsend suggestions, largely related to explict cache expiry. I think they're reasonable, but not high value. Also I think we've spent enough energy on this now, so I closed the threads.

@simonbaird

Copy link
Copy Markdown
Member

Let's merge.

@simonbaird simonbaird merged commit 61a3f64 into conforma:main Jul 7, 2026
32 checks passed
@fullsend-ai-retro

fullsend-ai-retro Bot commented Jul 7, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 8:43 PM UTC · Completed 8:53 PM UTC
Commit: 7c8ccca · View workflow run →

@fullsend-ai-retro

Copy link
Copy Markdown

PR #3136 (conforma/cli) added caching for sigstore verify_image results. The fullsend review agent provided the highest-impact finding: missing ClearCaches() calls in bundle-related tests causing them to silently pass via cached results. This was fixed. The main friction: all three bots (qodo, coderabbit, fullsend) repeatedly flagged the same design tradeoff (caching transient failures), which the author had explicitly acknowledged. Fullsend re-raised this as [high] logic-error across 2 runs. The PR was merged with fullsend reviews DISMISSED and no review on the final code after fixes were applied.

Proposals filed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file docker Pull requests that update Docker code go Pull requests that update Go code size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants