feat(measurement): record failed tasks and lost failover attempts in the usage log (#100) - #112
Open
7487 wants to merge 1 commit into
Open
feat(measurement): record failed tasks and lost failover attempts in the usage log (#100)#1127487 wants to merge 1 commit into
7487 wants to merge 1 commit into
Conversation
…son-Vaughan#100) The usage log only recorded successes: a task that failed at every backend raised RouterError and wrote nothing, and a request served after failover looked identical to one served first try. For a tool whose value is routing decisions, failures and their cost were invisible to --stats and the GUI. - Router.route() now surfaces the (entry_id, error) attempts it lost as last_failures, mirroring the existing last_served measurement seam. - record_task() takes an optional failures list, written as an optional failures field only when attempts were actually lost, and a kind="failure" record (spend_avoided_usd 0.0) is appended when every backend fails — additive optional fields, per the append-only forward-compat guarantee. - rollup() holds failure records out of the spend-avoided headline the same way delegate records are held out, counting them under failures plus a lost_attempts total; format_rollup() shows a Tasks failed line when nonzero. Docs: data-model/observability updated for the new kind and field; the two closed gap admissions removed; CHANGELOG entry added. Fixes Jason-Vaughan#100
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Failed tasks and lost failover attempts are now recorded in the usage log: a task that fails at every backend writes a
kind: "failure"record carrying the per-backend attempt list, and a task served only after failover carries the attempts it lost in a new optionalfailuresfield.--statsshows aTasks failed: N (lost failover attempts: M)line once there is anything to report.Why
The log only recorded successes — a fully-failed task raised
RouterErrorand wrote nothing, and a third-try success looked identical to a first-try one, so failures were invisible to--stats, the GUI, and any later analysis (gaps 1–2 indocs/design/observability.md).Fixes #100
On the design questions the issue raises, this takes the smallest consistent answers:
delegaterecords already are (rollupcounts them underfailures/lost_attemptsinstead), andspend_avoided_usdis0.0on them — nothing was served, so nothing was avoided.failures: [{entry, error}, …]field, written only when attempts were actually lost. Additive and optional, so pre-existing lines and readers predating the field stay correct, per the append-only forward-compat guarantee.Mechanically,
Router.route()surfaceslast_failuresalongside the existinglast_servedmeasurement seam, and the CLI'srecord_taskcalls (bothrun_onceandrun_once_stream) pass it through / write the failure record onRouterError. Not covered here: unlinked-delegate visibility (observability gap 1), whicharchitecture.mdalso files under #100 — that seemed worth its own issue rather than riding along.Test plan
make test— 500 tests, OK (17 skipped). New hermetic coverage:Router.last_failureson failover and total failure (test_router.py), thekind: "failure"record and thefailuresfield on a failover success — plus its absence on first-try success (test_cli.py), and the rollup holding failures out of the headline while tallying lost attempts (test_measurement.py). All new tests fail against the unmodified code.Checklist
make testpasses (hermetic suite; HTTP is mocked).docs/design/observability.md,data-model.md; closed gap admissions removed per CONTRIBUTING).CHANGELOG.md[Unreleased]updated (under the right subsection).key_refreferences only, never a raw key.DISCLAIMER.md).