[FEATURE] Chaos test scenarios for transient failure recovery

[koydas]

🎯 Goal

Validate that the retry and error classification logic holds under simulated failure conditions — and prevent regressions as resilience features are added.

📍 Context

• Repo: autonomous-dev-loop
• Domain: Robustness validation
• Component: scripts/tests/anthropic_client.test.mjs, scripts/tests/groq_client.test.mjs, scripts/tests/llm_client.test.mjs, scripts/tests/smoke.test.mjs

🚀 Description

The test suite is already comprehensive for happy-path and input validation scenarios. What is missing is targeted failure injection:

• groq_client.test.mjs covers 429 retry — good. No equivalent for 5xx.
• anthropic_client.test.mjs has no retry count verification tests (no check on exact number of fetch calls).
• llm_client.test.mjs does not verify that provider fallback is blocked on permanent 4xx errors.
• No test injects a network-level error (connection reset) to verify it is treated as retryable.
• No test verifies timeout abort behaviour.
• smoke.test.mjs has no scenario for a run that fails transiently and recovers.

Dependencies resolved: #109 (error taxonomy) ✅ and #111 (retry utility) ✅ are both merged. scripts/lib/retry.mjs, scripts/lib/error_taxonomy.mjs, and scripts/lib/anthropic_client.mjs (with retryWithBackoff) are all in place.

🧩 Scope

In:

• scripts/tests/groq_client.test.mjs: add cases for HTTP 500, 502, 503, 504 — verify each is retried up to maxAttempts then throws.
• scripts/tests/anthropic_client.test.mjs: add cases for 500/503 retry (exact call count), 401 immediate throw (1 call only), network reset retried.
• scripts/tests/llm_client.test.mjs:
  • Verify fallback to secondary provider does NOT occur when primary returns 401.
  • Verify fallback DOES occur when primary returns 503 and exhausts retries.
• scripts/tests/retry.test.mjs:
  • Backoff sequence correctness (~200, ~400, ~800 ms before jitter).
  • Timeout abort (AbortError) counted as a retryable attempt.
  • Retry-After header respected.
• scripts/tests/smoke.test.mjs: add one scenario — first LLM call returns 503, second succeeds — verify output is correct and attempt counter is 2.
• scripts/lib/anthropic_client.mjs: fix network TypeError currently marked as retryable: false — should be retryable.

Out:

• E2E tests requiring real network calls to Anthropic or Groq.
• Performance benchmarks.

🧪 Acceptance criteria

• When a 401 error occurs in callLLM, the secondary provider is never called and exactly one fetch call is made.
• When a network error (TypeError) occurs in callAnthropic, the retry count increases by 1 and the error is treated as retryable.
• When a timeout (AbortError) occurs, it is counted as a retryable attempt and the retry count increases by 1.
• When Retry-After header is present, the retry delay matches the header value (using fake timers — no real setTimeout).
• When all retry attempts are exhausted, the final error is thrown without additional fetch calls.
• A mocked 503 response on callAnthropic is retried and eventually returns the mocked success response (exact call count asserted).
• A mocked 503 response that persists across all attempts throws after exactly maxAttempts total calls.
• callLLM with a primary provider returning 503 and exhausting retries: secondary provider IS called.
• HTTP 500, 502, 503, 504 on callGroq are each retried up to maxAttempts then throw.
• Smoke: first LLM call returns 503, second succeeds — output is correct and attempt counter is 2.

⚙️ Constraints

• All tests must be deterministic — no flakiness from timing.
• No new test dependencies beyond what is already used in the suite.
• Tests must pass with node --test scripts/tests/*.test.mjs without any additional flags.

[github-actions]

✅ Issue Validation Report

Field	Value
Status	VALID — ready for development
Score	95/100 [██████████]

⚠️ Warnings (optional improvements)

• W2: Edge cases not covered — consider adding test for concurrent retry attempts or cascading failures
• W1: Missing technical context — specify which test utilities (e.g. mock fetch, fake timers) should be used for deterministic verification

💡 Suggested Acceptance Criteria

Copy the block below and paste it into the issue description, then save. Saving will re-trigger validation automatically.

## Acceptance Criteria

- [ ] When a 503 error occurs in `callGroq`, the retry count increases by 1 and the error is treated as retryable
- [ ] When a 500 error occurs in `callAnthropic`, the retry count increases by 1 and the error is treated as retryable
- [ ] When a 401 error occurs in `callLLM`, the secondary provider is never called and exactly one fetch call is made
- [ ] When a network error (`TypeError`) occurs in `callGroq`, the retry count increases by 1 and the error is treated as retryable
- [ ] When a timeout (`AbortError`) occurs in `callLLM`, it is counted as a retryable attempt and the retry count increases by 1

[koydas]

📊 État actuel (audit du 2026-06-02)

Les dépendances bloquantes sont levées : #109 ✅ et #111 ✅ sont fermées.

✅ Déjà en place

• scripts/lib/retry.mjs — retryWithBackoff implémenté avec backoff exponentiel + jitter
• scripts/lib/error_taxonomy.mjs — classification TRANSIENT / PERMANENT / UNKNOWN
• scripts/lib/anthropic_client.mjs — utilise retryWithBackoff avec les codes 429/5xx retryables, 401 non-retryable
• scripts/lib/llm_client.mjs — interrompt le fallback provider si errorType === 'PERMANENT'
• scripts/tests/retry.test.mjs — couverture de base : succès, épuisement, retryable: false, waitMs, cap, jitter, propagation de l'erreur exacte

❌ Reste à faire

scripts/tests/groq_client.test.mjs

• HTTP 500, 502, 503, 504 : vérifier que chaque code est retenté jusqu'à maxAttempts, puis throw

scripts/tests/anthropic_client.test.mjs

• 503 retenté : vérifier le nombre exact d'appels fetch (pas seulement le throw final)
• 401 immédiat : vérifier que fetch n'est appelé qu'une seule fois (pas de retry)
• Network reset (TypeError) : vérifier qu'il est compté comme tentative retryable

scripts/tests/llm_client.test.mjs

• Primary = 401 (PERMANENT) → fallback provider non appelé
• Primary = 503 exhaustant les retries → fallback provider bien appelé

scripts/tests/retry.test.mjs

• Séquence de backoff : les délais approx. [200, 400, 800] ms (avant jitter)
• Timeout abort : une AbortError est comptée comme tentative retryable
• Retry-After header : un waitMs de 2000 sur une 429 avec Retry-After: 2

scripts/tests/smoke.test.mjs

• Scénario de récupération transient : premier appel LLM retourne 503, deuxième réussit — vérifier que la sortie est correcte et que le compteur d'appels est 2

Note : anthropic_client.mjs marque actuellement les TypeError réseau comme retryable: false, ce qui contredit le critère d'acceptance. À corriger en même temps que les tests.

---

Generated by Claude Code