Skip to content

feat(measurement): record failed tasks and lost failover attempts in the usage log (#100) - #112

Open
7487 wants to merge 1 commit into
Jason-Vaughan:mainfrom
7487:feat/issue-100-log-failures
Open

feat(measurement): record failed tasks and lost failover attempts in the usage log (#100)#112
7487 wants to merge 1 commit into
Jason-Vaughan:mainfrom
7487:feat/issue-100-log-failures

Conversation

@7487

@7487 7487 commented Aug 28, 2026

Copy link
Copy Markdown

What

Failed tasks and lost failover attempts are now recorded in the usage log: a task that fails at every backend writes a kind: "failure" record carrying the per-backend attempt list, and a task served only after failover carries the attempts it lost in a new optional failures field. --stats shows a Tasks failed: N (lost failover attempts: M) line once there is anything to report.

Why

The log only recorded successes — a fully-failed task raised RouterError and wrote nothing, and a third-try success looked identical to a first-try one, so failures were invisible to --stats, the GUI, and any later analysis (gaps 1–2 in docs/design/observability.md).

Fixes #100

On the design questions the issue raises, this takes the smallest consistent answers:

  • Headline integrity: failure records are held out of the spend-avoided headline exactly the way delegate records already are (rollup counts them under failures / lost_attempts instead), and spend_avoided_usd is 0.0 on them — nothing was served, so nothing was avoided.
  • Record shape: stays one-line-per-task; lost attempts are a nested optional failures: [{entry, error}, …] field, written only when attempts were actually lost. Additive and optional, so pre-existing lines and readers predating the field stay correct, per the append-only forward-compat guarantee.
  • Quota consumed by failed attempts is left as recorded data (the attempt list), not folded into any cost figure — that valuation question stays open.

Mechanically, Router.route() surfaces last_failures alongside the existing last_served measurement seam, and the CLI's record_task calls (both run_once and run_once_stream) pass it through / write the failure record on RouterError. Not covered here: unlinked-delegate visibility (observability gap 1), which architecture.md also files under #100 — that seemed worth its own issue rather than riding along.

Test plan

make test — 500 tests, OK (17 skipped). New hermetic coverage: Router.last_failures on failover and total failure (test_router.py), the kind: "failure" record and the failures field on a failover success — plus its absence on first-try success (test_cli.py), and the rollup holding failures out of the headline while tallying lost attempts (test_measurement.py). All new tests fail against the unmodified code.

Checklist

  • make test passes (hermetic suite; HTTP is mocked).
  • New behavior has tests; bug fixes have a regression test.
  • Docs updated in this PR if behavior changed (docs/design/observability.md, data-model.md; closed gap admissions removed per CONTRIBUTING).
  • CHANGELOG.md [Unreleased] updated (under the right subsection).
  • No secrets committed — key_ref references only, never a raw key.
  • Change is consistent with the opt-in / bring-your-own-key posture (DISCLAIMER.md).

…son-Vaughan#100)

The usage log only recorded successes: a task that failed at every backend
raised RouterError and wrote nothing, and a request served after failover
looked identical to one served first try. For a tool whose value is routing
decisions, failures and their cost were invisible to --stats and the GUI.

- Router.route() now surfaces the (entry_id, error) attempts it lost as
  last_failures, mirroring the existing last_served measurement seam.
- record_task() takes an optional failures list, written as an optional
  failures field only when attempts were actually lost, and a kind="failure"
  record (spend_avoided_usd 0.0) is appended when every backend fails —
  additive optional fields, per the append-only forward-compat guarantee.
- rollup() holds failure records out of the spend-avoided headline the same
  way delegate records are held out, counting them under failures plus a
  lost_attempts total; format_rollup() shows a Tasks failed line when nonzero.

Docs: data-model/observability updated for the new kind and field; the two
closed gap admissions removed; CHANGELOG entry added.

Fixes Jason-Vaughan#100
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[feature] Record failed tasks and lost failover attempts in the usage log

1 participant