Skip to content

fix(server): read the AI meter alone for a cost quota's pricing blackout - #1201

Merged
arantespp merged 2 commits into
mainfrom
claude/soat-missing-camelcase-ym00s3
Sep 3, 2026
Merged

fix(server): read the AI meter alone for a cost quota's pricing blackout#1201
arantespp merged 2 commits into
mainfrom
claude/soat-missing-camelcase-ym00s3

Conversation

@arantespp

@arantespp arantespp commented Sep 3, 2026

Copy link
Copy Markdown
Member

Summary

A cost_usd quota's pricing-blackout verdict counted every usage event for the project, whatever its meter type. A window holding only unpriced compute_execution events therefore read as a blackout, and an enforcing cap refused every generation with 409 QUOTA_UNENFORCEABLE.

That state could not recover. The refusal blocked the very generation that would have landed the first priced llm_tokens event, so the window stayed all-unpriced and the cap stayed dead. It is not an edge case: no prices ship by default, and every orchestration node execution meters compute_execution, so any deployment that has run an orchestration and then sets its first cost cap hits it.

The verdict now reads llm_tokens events alone.

Why the AI meter alone is the right line

A platform meter is priced by the operator from a soat SKU, not by a tenant's AI provider. A deployment that leaves compute unpriced has not lost the ability to measure AI spend, which is what the cap is about, so an unpriced platform meter is not evidence that the cap protects nothing.

Reading the AI meter alone also fixes the opposite error the whole-window verdict allowed: a single priced platform event masked a genuine AI blackout, because the old check only needed one priced event anywhere in the window to conclude all was well. Both directions are covered by unit tests.

The aggregate the cap compares against is unchanged. Every priced meter is real spend and still counts toward the limit, platform included. Only the blackout verdict is meter-specific.

Relationship to #1177

Same function, same underlying weakness, different bug — this does not close #1177.

That issue quotes this exact expression and objects that it is blind to a partially priced window: with model A priced and model B not, priced.length > 0, so the verdict reads 0 and neither the exception nor the refusal fires while B's spend goes uncounted. After this change that case is unchanged — both models are llm_tokens, so pricedAiEvents is non-empty and the verdict is still 0. The component-level sub-case in that issue is untouched too.

Two ways this bears on it, worth knowing before picking it up:

  • It is a prerequisite for that issue's preferred remedy, not a competitor. The per-model unpriced signal proposed there groups the window's events by model. Without a meter-type distinction that grouping would surface soat/compute-second as just another unpriced model on every deployment that does not price compute, which is precisely the noise this PR removes. The two changes compose: this one adds meterType to the aggregation query, that one adds model and probably aiProviderId.
  • The file:line references in the issue are pinned to bd853d8a and will not match after this merges. The expression moves and gains the meter filter.

A fuller recommendation for that issue is posted on it directly.

Changes

File Change
lib/quotaEnforcement.ts The cost_usd aggregate reads meterType and scopes unpricedEventCount to llm_tokens
tests/.../quotaGenerationEnforcement.test.ts Two tests, plus a meterType option on the seeding helper
openapi/v1/quotas.yaml on_unpriced describes which meter the verdict reads
docs/modules/quotas.md Same, in the Unpriced usage section
tests/smoke-tests.sh Asserts the blackout refusal still fires end to end

Where the coverage sits, and why

The two cases this PR changes are pinned by unit tests, because they need a controlled mix of meter types in one window. A platform-only window must not refuse; a priced platform event must not mask an unpriced AI window.

The smoke assertion covers the opposite regression — a verdict narrowed so far it never fires at all. Every price the smoke stack writes is dated 2099, so no llm_tokens event there is ever priced and the project's window is a genuine AI blackout that block must still refuse. That is the one thing this environment can prove about the change, and it now asserts the 409 rather than a success.

Worth recording, since it is the kind of thing that reads as a fix regression later: my first version of that smoke step asserted the generation would succeed, and CI rejected it. The premise was wrong, not the refusal — I had assumed AI usage in the smoke stack was priced. The failure output confirms the project accumulates well past the three-event blackout threshold, which is also what makes the corrected assertion stable.

Verification

Red/green, in that order.

  • Both new unit tests fail on the unfixed code for the right reason: the platform-only window returns a reason: "unpriced_usage" breach where null is expected. Then they pass.
  • pnpm test in packages/server: 239 suites, 6709 tests, all passing against a local Postgres 16 with pgvector.
  • pnpm typecheck: passes. eslint on the changed files: clean.
  • pnpm docs-lint: OK. SDK and CLI regenerated: no output change.
  • bash -n on the smoke script: clean. Smoke and tutorials need a live stack, so CI is the real run — the first one caught the bad assertion above.

Context

This surfaced as five failures in naturali's live e2e on its first run against v0.37.0, all of them this one cause. Worth noting for anyone bisecting: v0.37.0 did not introduce the bug. The quota enforcement file is byte-identical between v0.36.0 and v0.37.0, and compute metering is older still. The release changed how many node executions get recorded, which is what pushed an existing latent condition past the three-event blackout threshold.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DRwsGecUbNX8PcAzZ5L2ov

The blackout verdict counted every usage event for the project, so a
window holding only unpriced platform events read as a pricing blackout
and an enforcing cost cap refused every generation. That could not
recover: the refusal blocked the very generation that would have landed
the first priced AI event, so the window stayed all-unpriced. A
deployment that prices no compute hit it on its first cost cap, because
no prices ship by default and every orchestration node meters
compute_execution.

The verdict now reads llm_tokens events alone. A platform meter is priced
by the operator from a soat SKU rather than by a tenant's provider, so
leaving it unpriced says nothing about whether AI spend is measurable.
Reading the AI meter alone also stops a priced platform event from
masking a genuine AI blackout, which the whole-window verdict allowed.

The aggregate the cap compares against is unchanged: every priced meter
is real spend and still counts, platform included.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DRwsGecUbNX8PcAzZ5L2ov
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown

Deploy Outputs

Package Stack Output Key Output Value
@soat/website SoatWebsite-claude-soat-missing-camelcase-ym00s3 BucketWebsiteURL http://soatwebsite-claude-soat-missing-camel-staticbucket-49z7nas4k7hy.s3-website-us-east-1.amazonaws.com

…ed window

The added step asserted a generation would succeed under a live cost cap.
Every price this stack writes is dated 2099, so no llm_tokens event here
is ever priced and the window is a genuine AI blackout that block must
refuse — the premise was wrong, not the refusal.

Asserting the 409 covers the regression the meter scoping could actually
introduce: a verdict narrowed far enough to never fire. The platform-only
window and the priced-platform-masking cases are pinned by the unit tests,
which can control the meter mix that this stack cannot.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DRwsGecUbNX8PcAzZ5L2ov

Copy link
Copy Markdown
Member Author

Server Tests (4/4) failed, and it is not this PR's. Re-running the failed jobs once.

The failure is tests/lib/webhookOutbox.test.ts:195, in "a delivery stranded mid-flight by a restart is reclaimed once its lease expires" — expect(stranded.status).toBe('success') received 'pending'.

Why it is not this change:

  • The diff touches lib/quotaEnforcement.ts, its test, quotas.yaml, modules/quotas.md and tests/smoke-tests.sh. Nothing in the webhook outbox, and nothing it depends on.
  • All four Server Tests shards passed on the previous head, ea736ca0. The only delta to 4c82be87 is tests/smoke-tests.sh, which the Server Tests job does not execute — so the source under test here is byte-identical to a run that passed this same shard minutes earlier.
  • The suite passes locally on this branch: 8/8 in that file.

The flake has a real mechanism, not just bad luck. The test waits on the wrong condition:

await waitFor(() => {
  return callsToUrl(url).length > 0;   // the outbound call has been observed
});

await stranded.reload();
expect(stranded.status).toBe('success'); // ...but this is written after it resolves

waitFor returns as soon as the delivery attempt has been seen, while the row's transition to success is persisted after that call resolves. Any scheduling delay between the two lands the assertion in the gap. Waiting on the row status instead of the call count would close it.

I am not fixing that here: it is an unrelated module and would widen a quota fix into a webhook one. It is worth its own small change, and I would rather it be visible than silently re-run forever.


Generated by Claude Code

@arantespp
arantespp merged commit 4c51067 into main Sep 3, 2026
20 of 22 checks passed
@arantespp
arantespp deleted the claude/soat-missing-camelcase-ym00s3 branch September 3, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] A partially-priced cost_usd quota enforces on a fraction of real spend, and files no signal

2 participants