Phase 7: Resilience & test hardening#5
Merged
Merged
Conversation
- LLMUnavailable raised on the three failure modes (timeout/network, malformed or non-schema JSON, empty/refused response) in ranking and postmortem generation - Orchestrator degrades honestly: 'degraded' terminal status, empty suspect list, raw diffs + runbook matches preserved, contract-valid payload; Slack card and dashboard badge surface the flag - contract: add 'degraded' status + degraded/raw_commits fields; allow null impact figures (real writes were already null — caught by test) - 18 backend tests (state transitions, degraded contract validation, diff-window extraction on a temp repo, runbook ranking, LLM failure modes); 3 dashboard smoke tests (Jest + RTL) - GitHub Actions CI: black/flake8/pytest + ESLint/jest; badge in README - black-formatted core/ and sandbox/; flake8 clean at 110 cols Verified live: server booted with a bad API key, real triggered incident finished cleanly as status=degraded with no fabricated diagnosis. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
LLMUnavailableis raised on all three failure modes (timeout/network, malformed/non-schema JSON, empty/refused). The orchestrator models this as a realdegradedterminal state: empty suspect list (nothing fabricated), raw git diffs + vector-matched runbooks preserved on the incident, contract-valid payload, warning surfaced in the Slack card and dashboard badge. Postmortem generation failure writes an honest placeholder instead of crashing.degradedstatus +degraded/degraded_reason/raw_commitsdiagnostics fields; impact figures now nullable — the contract-validation test caught that real runs were already writing nulls.contract/pipeline_schema.jsonwith jsonschema, diff-window extraction against a real temp git repo, runbook similarity ranking, all LLM failure modes. Fully offline/deterministic.Verification
pytest core/tests -q— 18 passednpm test— 3 passed;npm run lint— cleanblack --check+flake8— cleanOPENROUTER_API_KEY, real incident triggered → finished cleanly asstatus=degraded, suspects[], runbooks still matched, 3 raw commits attached🤖 Generated with Claude Code