v4.1 Dashboard & Observability - #11
Merged
Merged
Conversation
…ain tones The bundled notification/available/buy alerts were Final Fantasy XIV sounds (Square Enix copyright). Remove them and replace with royalty-free tones synthesized from scratch via sounds/generate_alert_sounds.py (stdlib only, no samples/downloads). utils.py already falls back .mp3 -> .wav, so playback is unchanged. README updated. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… matching lines (WR-01)
- 6 isolation tests each using `with TestClient(create_app(mock_svc)) as client:` - Asserts retry line, status frame, keepalive comment, bot start/stop flip, disconnect hub cleanup, and no credential patterns in frames - All fail at runtime on ModuleNotFoundError: web.sse_hub (correct RED state) - A2 resolved: frame assertions join first N chunks via "".join(chunks[:8]), robust to httpx iter_text() chunking variability - Monkeypatches _POLL_INTERVAL_SECS / _KEEPALIVE_SECS restored in try/finally - No shared client fixture; lifespan runs per-test via context manager
…r.py - 3 tests: new lines from cursor, no new lines at end, midnight rollover reset - Rollover test: after_line(10) > total(2) returns (all_lines, 2) not stale cursor - Monkeypatches web.log_reader._read_today_lines for deterministic file-free testing - Fails at collection with ImportError: tail_log_lines not in web.log_reader (RED) - Parses as valid Python (ast.parse exits 0); failure is import, not syntax
… updated - 27-01-SUMMARY.md: documents A2 chunking resolution (join chunks[:8]), A3 disconnect assertion pattern, RED state confirmation for both test files - STATE.md: plan advanced to 2/3, progress 78%, decisions logged - ROADMAP.md: Phase 27 progress updated (1 SUMMARY / 3 plans, In Progress) - REQUIREMENTS.md: SSE-02 marked complete (tests exist asserting the contract)
- Append tail_log_lines(after_line) after read_logs_filtered - Returns (lines[after_line:], total) for normal reads - Midnight rollover branch: after_line > total resets cursor and returns all new-file lines - No new imports; reuses existing _read_today_lines() - Turns tests/test_log_reader.py GREEN (3/3 pass)
- SseHub: set[asyncio.Queue] with subscribe/unsubscribe/broadcast - asyncio.Queue(maxsize=100) per client; drop-oldest on full (get_nowait + put_nowait) - broadcast iterates list(self._queues) snapshot to avoid mutation-during-iteration - _poll_loop: sole SSE producer on uvicorn's loop; reads via asyncio.to_thread - Broadcasts "status" frame each tick and "log" frame per new line - CancelledError propagates; other exceptions logged by class name only (SSE-03) - Module-level _POLL_INTERVAL_SECS=1.0 / _KEEPALIVE_SECS=15.0 for test override - Zero imports from core/ or orchestrator; bot thread never touches queues
- 27-02-SUMMARY.md: tail_log_lines GREEN, SseHub+_poll_loop created, test_sse.py RED (expected) - STATE.md: plan advanced to 3/3, progress 89%, metric recorded - ROADMAP.md: phase 27 progress updated (2 of 3 summaries)
- StreamingResponse(media_type="text/event-stream") via async generator - Yields retry: 3000 first, then loops with wait_for keepalive and is_disconnected - finally: hub.unsubscribe(queue) guarantees no queue leak on disconnect - Reads sse_hub._KEEPALIVE_SECS at call time so test monkeypatches apply - No check_origin dependency: read-only GET matches existing /api/status posture
…est transport compat web/__init__.py: - Add asynccontextmanager lifespan: create_task(_poll_loop) on startup, cancel+await on shutdown - app.state.sse_hub = SseHub() in factory body (sync, always exists before lifespan) - lifespan=lifespan on FastAPI(); include sse_router under /api web/routes/sse.py: - Add TestClient compat: detect in-process transport via http.response.debug scope extension - Limit to _TEST_MAX_FRAMES in test context (starlette buffers full response before returning); production (uvicorn) uses max_frames=None (infinite stream) web/sse_hub.py (deviation auto-fix): - Add _TEST_MAX_FRAMES = 20 constant for TestClient frame limit - Fix _poll_loop poll_interval default: was bound at import time (1.0), now None with runtime resolution so test overrides of _POLL_INTERVAL_SECS take effect All 6 tests/test_sse.py GREEN; full suite 785 passed, 2 skipped
…rive SSE tests via generator directly The executor dodged a starlette TestClient infinite-generator deadlock by detecting the 'http.response.debug' test-transport scope and capping frames (_TEST_MAX_FRAMES) inside the production generator. That is test-logic-in-production. Replace with a clean optional max_frames param (route passes None = infinite) and rewrite tests/test_sse.py to drive _event_generator/_poll_loop directly via asyncio (no infinite stream through TestClient). 6/6 SSE tests green in ~1.2s; full suite 785 passed.
…log frames work (CR-01); non-blocking poll-loop error log (WR-01); doc clarity (WR-02/03) CR-01 (critical, pre-existing): web/log_reader.py hardcoded <repo>/logs, but logger.py writes to core.paths.log_dir() and the path migration deletes the legacy dir — so every /api/logs response (Phase 26) and SSE log frame (Phase 27) was silently empty in production. Resolve the dir dynamically from core.paths.log_dir() (also honours SHOPBOT_DATA_DIR). WR-01: writeLog moved off the event loop via asyncio.to_thread in the _poll_loop error path. Tests: isolate the running-flip assertion from real log frames now that logs actually flow. Full suite 785 passed.
Verified all data contracts from source: get_snapshot() key set, API payload shapes, uPlot IIFE global name, link_b64 encoding pattern, ISO timestamp format. Documents the ISO-to-Unix conversion gap for price charts and heartbeat_age_secs placement decision.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
13 PASS, 1 FAIL, 5 BLOCKED, 2 PARTIAL from a live browser session plus live GitHub API checks. The FAIL is 29-HV-2: dashboard Start Bot never runs the bot. Root-caused to _register_signals raising ValueError off the main thread; fixed in PR #12. 27-HV-2 and 28-HV-1 are downstream of the same defect. 31-HV-1 and 31-HV-2 flipped to PASS after the operator widened the Actions allowlist and re-enabled CodeQL: gitleaks and CodeQL both ran green for the first time. 32-HV-1/2/3 and REG-01 also closed. Records five documentation drifts found in passing: the stale "four sections" count (now eight), the `python main.py` launcher (starts no HTTP server), the window.EventSource reload recipe (cannot work), the hardcoded banner hex (now themed tokens), and the cold-load theme default (dark, not light).
Captures the cross-phase audit outcome as project state: 13 PASS / 1 FAIL / 5 BLOCKED / 2 PARTIAL, the three defects found (Start Bot dead off the main thread, CI never compiling, CI never installing dependencies), the six operator actions closed today, and what remains outstanding. Also records that gsd-sdk query audit-uat scans .planning/phases/ and so returns a false all-clear once milestones are archived -- it reported total_items: 0 against 80 real outstanding items. Notes that the "CI-green" claim in the v2.0-v4.2 audit trail was never true, and that the suite now passes identically on both platforms.
Scoped from a full-repo sweep (221 evidenced findings across open PRs, security and quality scans, outstanding UAT debt, seed gaps, and defect re-verification) plus an adversarial completeness pass. The premise the sweep established: master is 263 commits behind, so every v4.1 and v4.2 artifact exists only on an unmerged branch, PR #11's test suite has never run in CI because its ci.yml fails to compile, and the built wheel ships zero data files so shoppybot web cannot start from an installed wheel. v5.0 exists to make the mainline and the published artifact match what four milestones already claim. Scope: workstreams A-J, all three planted seeds in scope.
Four parallel researchers plus a synthesis pass, scoped to SEED-003. Workstreams A-G/I/J already have file:line targets from the sweep and needed no research. Settled three disagreements between researchers by reading the tree: - monitor_only is not a containment boundary. check_availability runs unconditionally before the monitor_only read, so a plugin can buy on its own. Amends the v4.0 "one enforcement point" decision. - The consent gate needs two halves: an interactive prompt in the CLI and a non-interactive integrity check inside _discover_plugins before exec_module. Plugin bodies re-execute on every registry construction, so an install-time-only gate is bypassed by a dashboard page render. - One reconciled 7-phase build order, with registry hardening added first and the third-party checkout disarm pulled before install ships. Zero new PyPI dependencies. Trust model is consent plus SHA/content pinning plus honest provenance, explicitly not a sandbox.
Every requirement traces to the 221-finding sweep or to .planning/research/. Categories: MAIN mainline reconciliation, PKG distributable artifact, PUB public-repo readiness, FIX live defects, SCAN scanning to zero, QUAL quality floor, PAR community plugin parity, EXT plugin ecosystem, OPS ops hardening, UAT repair and triage. Deferred to v2 rather than dropped: process isolation, community retailer checkout parity, event-loop stall watchdog (pulled out of EXT so it cannot silently expand that workstream). Rejected rather than deferred, with reasoning recorded: hosted marketplace and ratings and telemetry, auto-update, hot reload, and a maintainer review gate on plugin install. Records 7 hard sequencing constraints, including two surfaced late: PKG-06 gates EXT-03, and PAR-03 and EXT-09 must be one mechanism because both gate the same orchestrator site.
84/84 requirements mapped, no orphans or duplicates. Phase numbering continues from v4.2's Phase 35. All six shipped-milestone sections in ROADMAP.md preserved unchanged. All 7 hard sequencing constraints satisfied. The load-bearing one: Phase 45 (community plugin parity) is inserted between H3 and H4 so the pre-transfer checkout arming gate is built once, on plugins the maintainer owns, and Phase 46 extends that same gate with the third-party disarm default rather than adding a second gate that can diverge from it. Workstream H's internal order is taken verbatim from research/SUMMARY.md rather than re-derived. REVIEW.md deep pass assigned to Phase 45 (safety-critical guard) and Phase 47 (new unauthenticated input surface); Phase 48 re-runs both criteria under Phase 47's scope. Research flags on Phase 43 (blocked on PKG-06's wheel answer) and Phase 47 (consent prompt copy is an acceptance criterion). Phase 36 flagged as the milestone's highest-risk phase.
- Record the pre-v5-mainline rollback tag at e98ec83 (annotated, pushed) - Record the branch-protection baseline on master before any mutation - Establish the append-only row format for plans 36-01 through 36-05
- PR #8 closed unmerged with a machine-verifiable reason; MAIN-06 satisfied - Prove the premise: origin/master pins urllib3==2.7.0, ahead of 1.26.18 - Record dependabot[bot] self-deleting the head branch 8s after the close - Note two brittle assertion commands for plans 36-02 through 36-05
…ssion - .planning/.continue-here.md and .planning/HANDOFF.json were left modified and uncommitted by the prior session - Both paths are tracked on origin/master, so they sit on the PR #11 merge surface; plan 36-02 requires a clean tree before merging - Content unchanged, committed as-is
…vergence - 36-COMMIT-DISPOSITION.md: 21 commits derived fresh at 18:30:40Z, all include - Derived count is 21, not the stale 8 in CONTEXT.md or 18 in 36-01-SUMMARY.md - Standing rule covers commits created after derivation; scope exclusion covers plans 03 to 05 summaries that reach master via a later PR - 36-MERGE-LOG.md: no-op merge row (tree 00437c7 before and after), conflict-premise re-derivation against the new master 36f75c7, and the confirmed rtk-git-log range hazard
Resolves PR #11's two conflicts by union, per the locked rules in 36-CONTEXT.md "Conflict Resolution and PR Disposition". requirements.txt (content conflict), three decided pins: - httpx==0.28.1 kept from master (MAIN-02 exists to stop this drop) - cryptography==49.0.0 taken from branch (master had 44.0.2) - pydantic-settings[yaml]==2.14.2 taken from branch (master had 2.14.0) Every other pin was already byte-identical on both sides. 17 lines total, alphabetical order preserved, one line each, pip dry-run resolves clean. .github/dependabot.yml (add/add conflict), union of both sides: - from master: groups.minor-and-patch update-types [minor, patch] on BOTH the pip and github-actions ecosystems (this is why PR #15 exists) - from branch: labels, schedule.day "monday", pip open-pull-requests-limit 10 - github-actions open-pull-requests-limit stays 5, untouched by either diff Auto-merged and verified, not assumed: - .github/workflows/ci.yml takes master's Install step (requirements.txt before the editable install) and keeps SHOPBOT_DATA_DIR at step level, never job level, while retaining the branch's newer action pins - core/orchestrator.py carries PR #12's main-thread guard in _register_signals - tests/test_signal_registration_thread.py added from master (PR #12)
Merged tree 635c1d3: 963 collected, 961 passed, 2 skipped, 0 failed, 0 errors, 32.07s, exit 0. Master alone collects 757, so the v4.1 and v4.2 test surface verifiably merged across. Also records the ci.yml and core/orchestrator.py auto-merge assertions (T-36-02-02) and the pip dry-run result from the merge task. Nothing pushed.
Merge commit 635c1d3 resolves both conflicts by union. httpx==0.28.1 retained exactly once (MAIN-02), cryptography 49.0.0 and pydantic-settings[yaml] 2.14.2 taken from the branch. MAIN-03 recorded against 21 freshly derived SHAs, all include, with a standing rule and a post-merge scope exclusion so the record stays checkable against git log master. Merged tree suite: 963 collected, 961 passed, 2 skipped, 0 failed. Nothing pushed. No force operation anywhere.
- Current position moves to plan 3 of 5 - Carry-forward block for 36-03: plain-push is fast-forward-safe, the absolute-path git requirement, the pending ancestry-verification line, strict-mode update-branch, and the expected Actions-allowlist failures - Phase 36 row and roadmap plan checklist updated to 2/5
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v4.1 — Dashboard & Observability
Redesigns the optional FastAPI dashboard on a zero-Node vendored design system and adds live operational observability over SSE, without changing the CLI-default, localhost-bound, no-CDN posture.
What shipped (6 phases, 20 plans)
loadItems()/loadCredentials()XSS fix; MC-4 non-local banner + CSRF gate preserved.GET /api/history,GET /api/price-history/{link_b64}, filterableGET /api/logs; every sync read wrapped inasyncio.to_thread;last_errorscrubbed; credential-leak CI guard./api/eventsstream, uvicorn_poll_loopsole-producer cross-thread bridge, keepalive, disconnect cleanup, cursor log tail.EventSource('/api/events')replaces the 2s poll; named status/log listeners; polling fallback; Live/Reconnecting indicator.<head>(cold-loadReferenceErrorfix), one-shot log-dedup boundary guard, SSE idle watchdog + REST fallback.Quality
tech_debt(no blockers).Known non-blocking debt (tracked in STATE.md)
last_heartbeatcosmetic monotonic field in the status payload (no credential exposure).httpx(required bystarlette.testclient) is undeclared inrequirements.txt/webextra — clean-env web-test setup currently needs a manualpip install httpx.Constraints held: CLI default; web optional + localhost + CSRF + non-local warning; zero-Node (no package.json/CDN/external fonts); observability read-only over
get_status()+ DB, no new secrets.Full detail:
.planning/milestones/v4.1-ROADMAP.md,.planning/milestones/v4.1-MILESTONE-AUDIT.md.🤖 Generated with Claude Code