Skip to content

[WIP] Add durable API-cost outbox and canonical usage tracking - #1376

Merged
groupthinking merged 3 commits into
mainfrom
claude/agentharden-api-cost-outbox
Aug 13, 2026
Merged

[WIP] Add durable API-cost outbox and canonical usage tracking#1376
groupthinking merged 3 commits into
mainfrom
claude/agentharden-api-cost-outbox

Conversation

@Claude

@Claude Claude AI commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Thanks for asking me to work on this. I will get started on it and keep this PR's description up to date as I form a plan and make progress.

This section details on the original issue you should resolve

<issue_title>Execution: durable API-cost outbox and canonical usage tracking (PR #869)</issue_title>
<issue_description>## Parent and dependency

Current state — CODE COMPLETE, PROTECTED PROOF PENDING

Agent login

groupthinking

Agent run ID

eventrelay-outbox-session-scope-20260721-0415z

Acceptance criteria

  • Durable outbox keyed by UTC date and alert type.
  • Conflict-safe atomic usage persistence, budget recomputation, and outbox enqueue in one transaction.
  • Bounded retry/backoff, token-fenced claims, stale/cancelled claim recovery, and stable idempotency headers.
  • SQLAlchemy session management with supported worker-thread in-memory SQLite tests.
  • Webhook networking remains asynchronous and does not block accounting.
  • Only successful conditionally finalized deliveries are counted.
  • Production delivery is kept out of FastAPI lifespan and uses the dedicated worker substrate.
  • Gemini provider token metadata is preserved.
  • The canonical v1/HybridProcessor path emits durable track_api_call telemetry without allowing telemetry failure to discard a paid result.

Exact-head evidence

  • CI 29810092858: passed
  • API cost PostgreSQL 29810092890: passed
  • Coverage 29810092885: passed
  • CodeQL 29810092856: passed
  • Security Scan 29810092921: passed
  • Secret Scan 29810092855: passed
  • Dependency Review 29810092888: passed
  • E2E 29810092846: repository-skipped
  • Focused Gemini/HybridProcessor tests: 248 passed locally
  • Review threads: 0 unresolved

Remaining gates — execute in this order

  1. Complete Execution: prove durable API-cost production substrate (PR #906) #868 / PR fix(ci): remediate PR #877 rollout and verification gaps #906 protected staging proof: the deployed database is at migration head and a pending row survives worker revision A → B and is visible to a distinct reader.
  2. Against the exact final fix: harden API-cost webhook outbox retries (MYX-79) #869 head, attach production-shaped evidence for overlapping workers, crash boundaries, timeouts, ordering, retry exhaustion, and multi-batch delivery.
  3. Review and pin the webhook secret/config change separately; prove rollback while delivery remains disabled until approval.
  4. Reverify the exact head, mark fix: harden API-cost webhook outbox retries (MYX-79) #869 ready, merge with authority, then close this issue.

There is no remaining known code blocker in #869. If protected proof exposes a defect, patch the same canonical branch; do not open another implementation.

Execution receipt

  • Agent login: groupthinking
  • Run ID: eventrelay-outbox-session-scope-20260721-0415z
  • Canonical PR/branch: fix: harden API-cost webhook outbox retries (MYX-79) #869 / agent/harden-api-cost-outbox
  • Claimed: 2026-07-21T04:15:00Z
  • Latest verified heartbeat: 2026-07-21T07:43:50Z
  • Exact head: 45edc01037d72e7d2d9a56e18b2d5c2f6bb4ba76
  • Artifact: green exact-head workflows, canonical Gemini tracking commits, and zero unresolved threads

Safety boundary

Keep #869 draft until the protected staging and production-shaped proof gates are attached. Do not expose secrets or mutate production from this issue.

Truth-gate reconciliation — 2026-07-21

  • Added the machine-readable Agent login and Agent run ID fields without fabricating a snapshot.
  • Normalized the existing verified checklist as Acceptance criteria.
  • Added copilot-rabbit authorization and triggered an exact-head reevaluation.
  • Replaced the generic invalid_payload failure with explicit evidence on run 29821174570.
  • Pre-dispatch intent snapshot exists: no.
  • Declared file scope and focused-test contract existed before implementation: no; do not backfill as historical intent.
  • Trusted terminal result from groupthinking / eventrelay-outbox-session-scope-20260721-0415z: pending.
  • Current-head Copilot review: pending.

Controller receipt

  • Controller login/run: groupthinking / eventrelay-blocker-watch-20260721-1005z
  • Canonical PR: fix: harden API-cost webhook outbox retries (MYX-79) #869
  • Claimed: 2026-07-21T10:05:35Z
  • Latest verified heartbeat: 2026-07-21T10:10:42Z
  • Exact head: 45edc01037d72e7d2d9a56e18b2d5c2f6bb4ba76
  • Artifact: truth-gate run 29821174570 with valid identity projection and explicit provenance/scope/review reasons

Next executable action

Keep #869 draft. If the accepted Copilot trigger opens a valid finding, patch the same branch. Do not invent retroactive intent, scope, test, or agent-result evidence. Protected deployment verification and an authorized legacy-provenance disposition remain the only later human boundaries.

Current controller reconciliation — 2026-07-22T10:23Z

  • Detected new intermediate PR head 1fd74ffc6fb791797428a071f75f12b0afa2a013 and unsafe ready state.
  • Returned canonical PR fix: harden API-cost webhook outbox retries (MYX-79) #869 to draft.
  • Diagnosed authoritative Coverage failure 29909459928: the branch synchronized main before fix(ci): make coverage and gh-aw canary authoritative #927 landed, retaining the obsolete 90% floor and pre-repair harness.
  • Fast-forwarded the same branch to fd7c82d268224709487cbf0f1c5865feeb77da2d using current main@5da61c595aa9dc848786e9e1fe99e40ad2a4fce0 plus exactly the eight declared implementation/test files.
  • CI 29911281817, Coverage 29911282072, PostgreSQL 29911282046, CodeQL 29911282154, Security 29911282064, Secret Scan 29911281875, and Dependency Review 29911282177 pass.
  • Coverage artifact 8526104756: 7,774 passed; 88.2371%; 2,033,370 bytes; exact-head bound.
  • Zero unresolved review threads.
  • Current-head Copilot review.
  • Protected staging migration and A→B durability proof.
  • Production-shaped overlap/crash/timeout/order/exhaustion/multi-batch proof and separately reviewed webhook activation/rollback.

Execution receipt

  • Agent login: groupthinking
  • Run ID: eventrelay-blocker-watch-20260722-1011z
  • Canonical branch/PR: agent/harden-api-cost-outbox / fix: harden API-cost webhook outbox retries (MYX-79) #869
  • Claimed: 2026-07-22T10:11:32Z
  • Latest heartbeat: 2026-07-22T10:23:12Z
  • Exact head: fd7c82d268224709487cbf0f1c5865feeb77da2d
  • Machine artifact: draft-state mutation, new fast-forward head, completed workflow suite, and exact coverage artifact.

Next executable action

Keep #869 draft. Obtain current-head automated review; patch only a verified finding on this branch. Protected staging/worker proof remains the later human boundary.</issue_description>

Comments on the Issue (you are @claude[agent] in this section)

@groupthinking Reopening after post-merge audit: one acceptance criterion remains unmet on `main` at `b71e2d60`.

process_outbox() can select pending/failed rows with retry_count < 5, but it is invoked only once by _trigger_delivery() when the alert is first claimed. After a transient delivery failure, no worker, timer, or backoff schedules another call; current tests only demonstrate retry by manually calling process_outbox() again.

Remaining completion scope:

  • automatically process due failed rows at startup and while the service is alive;
  • persist retry timing (for example next_attempt_at) and use bounded exponential backoff with jitter, max 5 attempts;
  • claim atomically so concurrent workers cannot double-send;
  • recover stale processing rows after restart and shut the worker down cleanly;
  • test transient failure → automatic later success, due-time gating, retry exhaustion, restart recovery, and concurrent no-duplicate delivery.

The optional AUTOCOMMIT SQLite VACUUM hardening observed in closed PR #866 is separate and should not be folded into this retry fix.</comment_new>
<comment_new>@groupthinking
Action taken from the post-merge audit:

#869 must remain draft. It is not production-safe on the current 0–10 instance Cloud Run deployment because storage is still instance-local /tmp SQLite. Two #861-owned gaps also remain: usage persistence and outbox enqueue are separate transactions, and the canonical v1/HybridProcessor path does not currently emit track_api_call events.

Exact next action: complete #868 / MYX-81, then revise #869 to make enqueue atomic (or add durable reconciliation) and prove a canonical request creates production-visible usage/outbox rows before marking it ready.</comment_new>
<comment_new>@groupthinking
Material execution update for canonical PR #869:

Next executable code unit: replace the four remaining manual outbox session lifecycles with the existing context-managed session contract, then make usage persistence and outbox enqueue atomic (or add durable reconciliation). Protected Cloud SQL/webhook secrets and real staging revision-replacement proof remain human-authority gates and must not be fabricated or posted.</comment_new>
<comment_new>@groupthinking

Execution artifact ready on canonical draft PR #869: the managed-session refactor is verified on exact head 7eacfdbce77aae5a38298c5b722118fadd675486. This is partial progress, not merge approval; the remaining issue acceptance gates stay open.</comment_new>
<comment_new>@groupthinking

Focused execution checkpoint

agent: groupthinking
run: eventrelay-hybrid-usage-20260721-0715z
canonical branch/PR: agent/harden-api-cost-outbox / #869
claimed: 2026-07-21T07:15:04Z
latest heartbeat: 2026-07-21T07:26:53.141Z
exact head: 45edc01037d72e7d2d9a56e18b2d5c2f6bb4ba76

  • Atomic usage/outbox persistence previously completed.
  • Canonical HybridProcessor Gemini calls now emit durable cost records from provider-reported token metadata.
  • Telemetry failure is non-fatal to completed provider responses.
  • 248 focused tests passed locally.
  • All exact-head required checks passed; zero review threads remain.
  • Canonical PR metadata no longer lists this completed code gate as pending.

Remaining gates are protected staging migration/revision-replacement evidence, webhook activation/rollback, and final production approval. Those gates do not invalidate the completed code artifacts.</comment_new>
<comment_new>@groupthinking
Controller reconciliation — 2026-07-23T05:39:03Z

New #955 copied this unit's implementation into a competing PR. It was returned to draft at exact head e39c495b16d02a281be7d32358900bb259355773, labeled duplicate, and marked evidence-only. Canonical execution remains this issue / #869 at fd7c82d268224709487cbf0f1c5865feeb77da2d.

Receipt: groupthinking / eventrelay-blocker-watch-20260723T0511Z; claimed 2026-07-23T05:11:49Z; heartbeat 2026-07-23T05:39:03Z.</comment_new>
<comment_new>@groupthinking
Controller state correction: noncanonical proof branch #926 was marked ready again and had advanced to 781c26b379a969cbddbdb5ea390477625febe8bc across 11 files. It is now back in draft with a current receipt. Canonical #869 remains the only executable implementation. #926's documentation/configuration claims do not satisfy protected staging, shared PostgreSQL, worker durability, webhook, real-credential, or rollback proof.</comment_new>
<comment_new>@groupthinking
Current-head review remediation on canonical #869:

  • Exact head c6d123b1a81ec7abc10472e2bd17759f7d505d51 adds an authoritative gemini-3.5-flash standard price from Google’s current pricing contract and fails closed for unknown models instead of charging the first unrelated table entry.
  • Gemini thoughts_token_count is now included in billed output tokens; the canonical HybridProcessor usage test covers prompt + candidate + thought tokens.
  • Every outbox claim now generates and persists a unique token; completion and recovery compare-and-swap predicates fence on it, and terminal/recovered rows clear it. A focused regression proves a stale owner cannot complete another claim.
  • Dependency Review and Secret Scan pass; E2E is repository-skipped. CI, Coverage, PostgreSQL, CodeQL, and Security are still running and are not counted as passes yet.
  • The three current review threads remain unresolved until exact-head workflows substantiate the patch. fix: harden API-cost webhook outbox retries (MYX-79) #869 stays draft.

Execution receipt:

This partial implementation progresses #898 and does not close #861.</comment_new>
<comment_new>@groupthinking

Controller receipt — CI compatibility correction

  • Agent login: OpenAI
  • Run ID: eventrelay-blocker-watch-20260725T0015Z
  • Canonical branch/PR: agent/harden-api-cost-outbox / fix: harden API-cost webhook outbox retries (MYX-79) #869
  • Claimed timestamp: 2026-07-25T00:15:19Z
  • Latest heartbeat: 2026-07-25T00:22:00Z
  • Exact head: 9276cf11e129cacc30d879dd9c47ba90a2971f12

The prior exact head failed CI because six established callers intentionally omit a model and therefore use the legacy sentinel model="default". The new explicit-unknown-model guard incorrectly treated that sentinel as a typo.

Applied the smallest focused correction on the canonical branch:

  • explicit unknown provider models still fail closed;
  • the legacy default sentinel retains service-default costing;
  • added a regression test for the compatibility contract.

This supersedes c6d123b1. Review threads remain unresolved until exact-head workflows complete.</comment_new>
<comment_new>@groupthinking

Controller receipt — exact-head completion

Completed and verified:

  • explicit unknown Gemini models fail closed; the intentional default sentinel preserves legacy unspecified-model calls;
  • Gemini 3.5 Flash pricing is normalized from the official per-million-token rate;
  • thought tokens are billed as output;
  • outbox claims are UUID-token fenced through claim, completion, recovery, and release;
  • CI 30136594736, Coverage 30136594795, API cost PostgreSQL 30136594742, CodeQL 30136594768, Security 30136594743, Secret Scan 30136594746, and Dependency Review 30136594731 pass;
  • E2E 30136594730 is repository-skipped;
  • all three current review findings are resolved with exact-head evidence; zero review threads remain unresolved.

The exact-head Vercel preview was canceled by unverified-commit policy. #869 was automatically promoted after checks and has been returned to draft because protected staging migration, revision-replacement durability, production-shaped worker scenarios, webhook activation review, and rollback proof remain incomplete.

This partial implementation progresses #898 and does not close #861.</comment_new>
<comment_new>@groupthinking

Controller reconciliation — 2026-07-27T21:53Z

  • Returned this unsafe ready-state transition to draft.
  • Advanced the existing canonical branch with focused commits 3093abf8 and cc6fe44a.
  • Exact head: cc6fe44ab2d6092b2f550d8c1bec74e1a90e6db1.
  • Restored the SQLite due index contract to (status, next_attempt_at, retry_count, id) and updated its focused regression assertion.
  • Changed stale-claim selection and compare-and-swap fencing to claimed_at, preserving the NULL legacy-claim branch and matching (status, claimed_at, id).
  • Both exact review findings are proven fixed and resolved; all 11 review threads are now resolved.
  • Exact-head CI, Coverage, CodeQL, Security, and API cost PostgreSQL workflows are currently running. Dependency Review and Secret Scan pass; E2E is repository-skipped.
  • Protected staging/worker, webhook activation, rollback, and current-head independent review gates remain incomplete. Keep draft.

Execution receipt

Next executable action: wait for the exact-head workflows, then record their terminal results. Do not merge or activate production delivery.</comment_new>
<comment_new>@groupthinking

Exact-head workflow completion — 2026-07-27T21:58Z

Exact head remains cc6fe44ab2d6092b2f550d8c1bec74e1a90e6db1.

Keep #869 draft. The two outbox findings remain fixed and all 11 threads resolved, but exact-head CI cannot become green until the separately owned TypeScript rollback blocker is repaired through a valid focused canonical unit.</comment_new>

@vercel

vercel Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
v0-uvai Ready Ready Preview, v0 Aug 4, 2026 8:28pm

@Claude Claude AI linked an issue Aug 4, 2026 that may be closed by this pull request
Execution: durable API-cost outbox and canonical usage tracking (PR #869) #861
Closed
27 tasks
@Claude @groupthinking
feat: harden API-cost outbox with CAS claims, bounded retry, atomic t…
c97b1aa 
…ransactions, and canonical usage tracking

- Add compare-and-swap claim/completion with UUID tokens to prevent double-sends
- Implement bounded exponential equal-jitter backoff (max 5 retries)
- Make usage persistence, budget recomputation, and outbox enqueue atomic
- Add fail-closed pricing for unknown models (ValueError instead of silent fallback)
- Add usage_metadata passthrough to GeminiResult for provider-reported token counts
- Add _track_gemini_usage to HybridProcessorService (non-fatal telemetry)
- Add comprehensive outbox worker test suite (test_api_cost_outbox_worker.py)
- Update existing tests for atomic transaction semantics and new pricing behavior

Co-Authored-By: Claude <noreply@anthropic.com>

Agent-Logs-Url: https://github.com/groupthinking/EventRelay/sessions/989c4fcc-0545-427d-887f-754827f12eeb

Co-authored-by: groupthinking <154503486+groupthinking@users.noreply.github.com>
@github-actions

github-actions Bot commented Aug 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Agent Completion Truth Gate: BLOCKED

Reasons: verdict_artifact_missing

Machine-readable verdict
{
  "verdict": "blocked",
  "reasons": [
    "verdict_artifact_missing"
  ],
  "details": {}
}

Workflow evidence

@Claude
Claude AI requested a review from groupthinking August 4, 2026 20:29
vercel[bot]
vercel Bot reviewed Aug 4, 2026
View reviewed changes

@vercel vercel Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional Suggestion:

A strict pricing lookup (calculate_cost raising ValueError for unknown models) runs inside the broad try/except of the paid provider calls in real_ai_processor.py, so a missing price turns an already-completed, successful, billed AI call into success=False with the content discarded.

Fix on Vercel

This was referenced Aug 5, 2026
Closed
Merged
Closed
@groupthinking
groupthinking marked this pull request as ready for review August 13, 2026 07:55
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

🔍 PR Validation

⚠️ PR title should follow conventional commits format
⚠️ Large PR detected (1343 lines changed)

@github-actions

github-actions Bot commented Aug 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Snapshot Warnings

⚠️: No snapshots were found for the head SHA 51db332.
Ensure that dependencies are being submitted on PR branches and consider enabling retry-on-snapshot-warnings. See the documentation for more information and troubleshooting advice.

Scanned Files

None

@groupthinking
groupthinking merged commit 6dc1f03 into main Aug 13, 2026
23 of 24 checks passed
@groupthinking
groupthinking deleted the claude/agentharden-api-cost-outbox branch August 13, 2026 07:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vercel vercel[bot] vercel[bot] left review comments
@groupthinking groupthinking Awaiting requested review from groupthinking groupthinking is a code owner

Assignees

@groupthinking groupthinking

@Claude Claude

Labels

python

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Execution: durable API-cost outbox and canonical usage tracking (PR #869)

2 participants

@Claude @groupthinking