From cacefd6034ac28390085f57f147c8d9b9f5f098f Mon Sep 17 00:00:00 2001 From: Zoid Date: Thu, 25 Jun 2026 01:29:32 -0400 Subject: [PATCH 001/287] docs: start milestone v4.1 Dashboard & Observability --- .planning/PROJECT.md | 27 ++++++++++++++++++++------- .planning/STATE.md | 26 +++++++++++++------------- 2 files changed, 33 insertions(+), 20 deletions(-) diff --git a/.planning/PROJECT.md b/.planning/PROJECT.md index 09154d3..c278175 100644 --- a/.planning/PROJECT.md +++ b/.planning/PROJECT.md @@ -24,16 +24,29 @@ v4.0: 7 phases (18-24) / 29 plans, all complete. Full suite: 755 passed, 2 skipp **Key constraints (held):** secrets never in config.yml/logs/SQLite plaintext; full card number / CVV never persisted to disk or logs (retailer-saved payment + CVV-at-runtime only); GUI optional, CLI default; the credential-managing web UI binds to localhost by default. -## Next Milestone +## Current Milestone: v4.1 Dashboard & Observability -**No active milestone.** v4.0 Win-the-Drop shipped 2026-06-25. Start the next cycle with `/gsd:new-milestone` (questioning → research → requirements → roadmap). Phase numbering continues from 24. +**Goal:** Redesign the optional FastAPI web dashboard with a polished zero-dependency design system and surface rich live operational observability over SSE — without breaking the CLI-default, localhost-bound, no-Node posture. -Candidate directions from the v4.0 deferral list: +**Target features:** +- Polished vendored design system (tokens, components, light/dark) via the frontend-design skill — no Node/CDN/external fonts +- Live per-plugin health cards (status / heartbeat / last-check / degraded) from `get_status()` + `HealthRegistry` +- Run history + recent confirmed buys (order_id, confirmed_at — BUY-04 records) +- Price-history charts (per-item, from the `price_history` table) +- Better log viewer (filter by level/plugin, search, tail) +- SSE push for live status/log/health updates (replaces the 2s poll) + +**Constraints (held):** CLI default; web optional + localhost bind + CSRF + non-local warning; zero-Node (no package.json/CDN/external fonts — vendored CSS only); observability is read-only over `BotService.get_status()` + DB, no new secrets. + +**Research flags:** charts must be dependency-free (vendored tiny lib or hand-rolled SVG/canvas); price data is Amazon-only today (PRICE-02) so charts stay sparse for other plugins. + +## Future Candidate Directions (post-v4.1) + +**Active milestone:** v4.1 Dashboard & Observability (scoped 2026-06-25; phases continue from 24). Remaining candidates for later milestones: - **Public-release hardening** — git-history scrub/squash (SEED-001) + release-please version tagging (SEED-002); a dedicated release milestone. - **Checkout form-fill for the remaining 5 retailers** (v4.0 covers BestBuy + Amazon). - **Request/API-mode (hybrid) checkout** — faster than DOM but per-site reverse-engineering and an arms race. - **Outcome analytics** (success rate, time-to-checkout) built on the BUY-04 order records. -- **Richer health/observability on the web dashboard** (beyond the CLI/status payload).
Shipped: v4.0 Win-the-Drop (Acquisition Core + Reliability) — 2026-06-25 @@ -103,9 +116,9 @@ Candidate directions from the v4.0 deferral list: - ✓ Reliability: unified RetryPolicy (one backoff source) — v4.0 (Phase 21) - ✓ Server-safety: headless pygame import-crash guard + SIGTERM/SIGINT teardown bridge — v4.0 (Phases 24, 22) -### Active (next milestone) +### Active (v4.1 Dashboard & Observability) -_None yet — run `/gsd:new-milestone` to scope the next cycle. See "Next Milestone" above for candidate directions._ +_Requirements defined in `.planning/REQUIREMENTS.md` (mapped by the roadmap). Focus: dashboard redesign (vendored design system) + live observability — per-plugin health cards, run/buy history, price-history charts, better log viewer, SSE push._ ### Deferred @@ -160,4 +173,4 @@ This document evolves at phase transitions and milestone boundaries. 4. Update Context with current state --- -*Last updated: 2026-06-25 — after v4.0 Win-the-Drop milestone (shipped)* +*Last updated: 2026-06-25 — v4.1 Dashboard & Observability milestone started* diff --git a/.planning/STATE.md b/.planning/STATE.md index 05bd245..0cf36a5 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -1,16 +1,16 @@ --- gsd_state_version: 1.0 -milestone: v4.0 -milestone_name: Win-the-Drop -status: Awaiting next milestone -last_updated: "2026-06-25T01:32:12.122Z" -last_activity: 2026-06-25 — Milestone v4.0 completed and archived +milestone: v4.1 +milestone_name: Dashboard & Observability +status: planning +last_updated: "2026-06-25T05:28:34.396Z" +last_activity: 2026-06-25 progress: - total_phases: 7 - completed_phases: 7 - total_plans: 29 - completed_plans: 29 - percent: 100 + total_phases: 0 + completed_phases: 0 + total_plans: 0 + completed_plans: 0 + percent: 0 --- # ShopPyBot — State @@ -28,10 +28,10 @@ progress: ## Current Position -Phase: Milestone v4.0 complete +Phase: Not started (defining requirements) Plan: — -Status: Awaiting next milestone -Last activity: 2026-06-25 — Milestone v4.0 completed and archived +Status: Defining requirements +Last activity: 2026-06-25 — Milestone v4.1 started ## Phase Status From f2f892b9e2adc4066c993feaf8c6f60f76724f85 Mon Sep 17 00:00:00 2001 From: Zoid Date: Thu, 25 Jun 2026 01:53:50 -0400 Subject: [PATCH 002/287] docs: research v4.1 Dashboard & Observability (stack/features/arch/pitfalls + summary) --- .planning/research/ARCHITECTURE.md | 941 +++++++++++--------- .planning/research/FEATURES.md | 453 +++++----- .planning/research/PITFALLS.md | 986 +++++++++++---------- .planning/research/STACK.md | 1303 ++++++---------------------- .planning/research/SUMMARY.md | 328 +++---- 5 files changed, 1599 insertions(+), 2412 deletions(-) diff --git a/.planning/research/ARCHITECTURE.md b/.planning/research/ARCHITECTURE.md index 5247f03..75f7b33 100644 --- a/.planning/research/ARCHITECTURE.md +++ b/.planning/research/ARCHITECTURE.md @@ -1,533 +1,612 @@ # Architecture Research -**Domain:** Python shopping bot — asyncio/nodriver plugin framework, v4.0 acquisition + reliability integration -**Researched:** 2026-06-10 (v4.0 Win-the-Drop integration analysis; supersedes v3.0 research) -**Confidence:** HIGH (all integration points derived from direct source reads; no inference from training data) +**Domain:** FastAPI dashboard with SSE live-push + design-system layer + observability surfaces (v4.1 Dashboard & Observability) +**Researched:** 2026-06-25 +**Confidence:** HIGH (all integration points verified against actual source files and FastAPI 0.115.8 package API) ---- - -## v4.0 Scope +## Standard Architecture -Two feature clusters integrate into the shipped v3.0 architecture: +### System Overview -1. **Acquisition Core** — checkout profile/form-fill, order-confirmation capture, idempotent retry-on-cart, per-step/per-item time budget, central monitor-only safety gate -2. **Always-On Reliability** — per-coroutine supervision + backoff restart, browser-crash detection + relaunch, encrypted session/cookie persistence, DB read-path error isolation, per-item asyncio timeout, structured health/heartbeat surface - ---- +``` +Browser + | EventSource /api/events (SSE multiplex, replaces 2s poll) + | GET /api/history (confirmed buys read-only) + | GET /api/price-history/{b} (per-item price series read-only) + | GET /api/logs?level=... (enhanced log reader) + | (existing fetch CRUD unchanged) + v +FastAPI / Uvicorn (uvicorn event loop -- single thread) + | + |-- web/__init__.py create_app() [MODIFIED: add sse_router, lifespan, SseHub on app.state] + |-- web/routes/api.py [MODIFIED: add /history, /price-history] + |-- web/routes/sse.py [NEW: /api/events StreamingResponse] + |-- web/sse_hub.py [NEW: SseHub class + _poll_loop background task] + |-- web/log_reader.py [MODIFIED: add filter + tail-with-cursor] + |-- web/static/tokens.css [NEW: design tokens, light/dark vars] + |-- web/static/components.css [NEW: component library consuming tokens] + |-- web/static/dashboard.css [MODIFIED: layout only, @imports tokens+components] + |-- web/static/sparkline.js [NEW: vendored zero-dep MIT sparkline ~1KB] + |-- web/templates/dashboard.html [MODIFIED: health cards, run-history, price charts, + | log filter UI; EventSource client; theme-init script] + | + | asyncio.Queue per SSE client (owned by uvicorn event loop) + | _poll_loop calls asyncio.to_thread(svc.get_status) -- NO cross-loop touching + v +BotService (daemon thread, owns its own asyncio event loop) + | + |-- get_status() [in-memory read, non-blocking, safe to call from any thread] + |-- _health_registry.get_snapshot() [deep copy, private keys stripped, thread-safe] + v +SQLite (WAL mode, existing models.py) + |-- items table (read: name, order_id, confirmed_at, checkout_attempts) + |-- price_history table (read: price_cents, currency, scraped_at) +``` -## Existing Architecture (v3.0 Baseline) +### Component Responsibilities + +| Component | Status | Responsibility | +|-----------|--------|----------------| +| `web/routes/sse.py` | NEW | Single `/api/events` endpoint; per-client asyncio.Queue; SSE text/event-stream generator | +| `web/sse_hub.py` | NEW | SseHub (set of active queues, broadcast method); `_poll_loop` async background task that drives all events | +| `web/static/tokens.css` | NEW | CSS custom properties for color, spacing, radius, shadow, type scale; light/dark via `prefers-color-scheme` + `[data-theme]` override | +| `web/static/components.css` | NEW | Button, card, badge, status-dot, table, form, log-panel rules that only reference token vars | +| `web/static/dashboard.css` | MODIFIED | Layout-only (container, grid, section order); opens with `@import` of tokens and components | +| `web/static/sparkline.js` | NEW | Vendored `fnando/sparkline` MIT, ~1KB minified; no CDN, no npm | +| `web/templates/dashboard.html` | MODIFIED | New sections: health cards, run history, price charts, log viewer with filter; EventSource replaces setInterval; inline theme-init in head | +| `web/routes/api.py` | MODIFIED | Add `/api/history` and `/api/price-history/{link_b64}` read-only routes | +| `web/log_reader.py` | MODIFIED | Add `read_logs_filtered(n, level, plugin, search)` and `tail_log_lines(after_line) -> (list, int)` | +| `web/__init__.py` | MODIFIED | Include `sse_router`; attach `SseHub` to `app.state.sse_hub`; register lifespan for `_poll_loop` | +| `BotService` / `HealthRegistry` | UNCHANGED | `get_status()` and `get_snapshot()` remain the sole read surface; no new writes or APIs | +| `models.py` | UNCHANGED | No schema changes; new queries use existing `get_db_connection()` context manager | + +## Project Structure Changes ``` -BotService (core/service.py) - |-- background daemon thread owns its own asyncio event loop - |-- get_status() -> {"running": bool} <-- extend for health surface - +-- async_main (core/orchestrator.py) - |-- PluginRegistry (core/registry.py) - | |-- _discover_plugins(): importlib scan of plugins/shopbot_plugin_*.py - | |-- _all_plugins: eager list; no browser yet - | |-- _active_plugins: post-setup; one Browser per plugin instance - | +-- RetailerPlugin ABC (core/plugin_base.py) - | |-- setup() / teardown() - | |-- check_availability(url) -> bool [abstract] - | |-- auto_buy(url) -> bool [abstract] - | |-- get_price(url) -> int | None [concrete, returns None] - | |-- login() / detect_captcha() [no-op defaults] - | +-- plugins/shopbot_plugin_*.py [7 concrete plugins] - | - |-- write_queue (asyncio.Queue) - | +-- _write_queue_drain task -> models.py (SQLite WAL CRUD) - | - |-- run_plugin() per active plugin [long-lived poll coroutine] - | +-- _check_and_buy() -> _try_auto_buy() -> plugin.auto_buy() - | - +-- NotificationDispatcher (notifications/dispatcher.py) - +-- fan-out: SoundNotifier, DiscordNotifier, EmailNotifier, SmsNotifier - -core/config_schema.py -- Pydantic AppConfig; debug.test_mode; no checkout/profile section yet -core/credentials.py -- CredentialStore (keyring / encrypted-file / env-var); SECRET_KEYS list -core/stealth.py -- STEALTH_JS, apply_stealth, ProxyPool, setup_proxy_auth -models.py -- SQLite WAL: items + price_history tables; all writes via write_queue +web/ + __init__.py (MODIFIED: lifespan, sse_router, SseHub on app.state) + sse_hub.py (NEW) + log_reader.py (MODIFIED) + routes/ + api.py (MODIFIED: /history, /price-history) + sse.py (NEW) + credentials.py (unchanged) + config.py (unchanged) + pages.py (unchanged) + static/ + tokens.css (NEW) + components.css (NEW) + dashboard.css (MODIFIED: layout only, @imports) + sparkline.js (NEW: vendored) + templates/ + dashboard.html (MODIFIED) ``` -Key constraints that drive all integration decisions below: +## Architectural Patterns -- `RetailerPlugin.setup()` is the only place browser args are set. Any relaunch must reproduce the full setup sequence (stealth injection, proxy auth, login). -- The write-queue is the **sole write path** to SQLite. Any new write (order confirmation, session flush) must go through it or through a dedicated drain, never direct. -- `debug.test_mode` is the existing config flag for purchase suppression. Only `AmazonPlugin.auto_buy()` reads it; the other 6 plugins bypass it entirely. This is the critical safety hole v4.0 must close. -- `BotService.get_status()` currently returns only `{"running": bool}`. The health surface expands this dict. -- `CredentialStore.SECRET_KEYS` is the authoritative list; new secrets (checkout profile fields) must be added. +### Pattern 1: CSS Token/Component Split (3-File Design System) ---- +**What:** Split the single `dashboard.css` (182 lines) into three layers. `tokens.css` owns all custom property declarations scoped to `:root`. `components.css` owns component rules that ONLY reference those custom properties (never hardcoded hex values). `dashboard.css` is reduced to layout (container width, section ordering) and opens with `@import url("tokens.css"); @import url("components.css");`. -## v4.0 Component Map: New vs Modified - -### New Modules - -| Module | Placement | Responsibility | -|--------|-----------|----------------| -| `core/checkout_profile.py` | New module | `CheckoutProfile` dataclass; load from CredentialStore; no plaintext in config or DB | -| `core/confirmation.py` | New module | `detect_order_confirmation(tab, platform) -> str | None`; per-platform selector map; returns order_id or None | -| `core/supervisor.py` | New module | `supervise(coro, name, max_retries, backoff)` async wrapper; replaces bare task creation in orchestrator | -| `core/health.py` | New module | `HealthRegistry`; per-plugin heartbeat dict; queryable via `BotService.get_status()` | -| `core/session_store.py` | New module | Fernet-encrypted JSON cookie persistence; reuses `EncryptedFileBackend` pattern from `credentials.py` | -| `core/retry.py` | New module | `RetryPolicy` dataclass + `with_retry(coro, policy)` async helper; one unified backoff implementation | - -### Modified Modules - -| Module | What Changes | Why | -|--------|-------------|-----| -| `core/orchestrator.py` | Replace bare `tg.create_task` with `supervisor`; add per-item `asyncio.timeout`; wrap DB reads in try/except; thread heartbeat updates through `HealthRegistry` | Supervision, timeout, read isolation, heartbeat | -| `core/plugin_base.py` | Add `monitor_only` property (reads `AppConfig.debug.monitor_only`); add `place_order_guarded()` default that checks `monitor_only` before calling the plugin's actual place-order DOM click | Central safety gate — the ABC intercepts all plugins, not per-plugin code | -| `core/service.py` | Expand `get_status()` to pull from `HealthRegistry`; accept `monitor_only` param on `start()` | Health surface, monitor-only control | -| `core/config_schema.py` | Add `DebugConfig.monitor_only: bool = True`; add `CheckoutConfig` sub-model (per-step timeout, max cart retries, item budget); add `CheckoutConfig` to `AppConfig` | Config-driven gates and budgets | -| `core/credentials.py` | Add checkout profile keys to `SECRET_KEYS`: `CHECKOUT_FIRST_NAME`, `CHECKOUT_LAST_NAME`, `CHECKOUT_ADDRESS_LINE1`, `CHECKOUT_ADDRESS_LINE2`, `CHECKOUT_CITY`, `CHECKOUT_STATE`, `CHECKOUT_ZIP`, `CHECKOUT_COUNTRY`, `CHECKOUT_PHONE` (9 keys; no card numbers; CVV already threaded) | Secure profile storage | -| `models.py` | Add columns to `items` table: `order_id TEXT`, `confirmed_at TEXT`, `checkout_attempts INTEGER DEFAULT 0`; add new write-queue tag `confirmed` | Verified purchase tracking, retry idempotency | -| `plugins/shopbot_plugin_amazon.py` | Remove inline `test_mode` check; delegate to `place_order_guarded()` from ABC; add confirmation detection call after place-order | Centralize safety gate; verified purchase | -| `plugins/shopbot_plugin_bestbuy.py` | Same: delegate to `place_order_guarded()`; add confirmation detection | Same | -| 5 remaining plugins | Same: delegate to `place_order_guarded()` | Close the 6-of-7 safety hole | +**When to use:** This project. Keeps concerns separated without any build step; all three are vendored static assets served by the existing `StaticFiles` mount. No Node, no CDN, no build pipeline. ---- +**Trade-offs:** Three HTTP requests instead of one on first load (negligible for a localhost-only app). Each file stays under 200 lines. The existing `dashboard.css` content is partitioned, not discarded; no behavior changes in this step. -## Feature-to-Component Placement +**Light/dark without FOUC, no external fonts:** -### Acquisition Core +Use two-layer detection. The `:root` block in `tokens.css` defines light-mode token values. A `@media (prefers-color-scheme: dark)` block on `:root` overrides them to dark values. An `html[data-theme="dark"]` selector allows a JS toggle to override the media query (persisted to `localStorage`). The `:root` inside a `@media` query has lower specificity than `html[data-theme]`, so the explicit toggle always wins. -**Checkout profile / form-fill** +To prevent FOUC, put a synchronous inline ` + + + + +``` -**Order confirmation capture** +This script runs synchronously during HTML parse, before the browser issues any style-sheet requests. The `data-theme` attribute is present when the first style sheet is applied, so the correct token values are used from the first paint. No flash. -- Where: `core/confirmation.py` (new) -- `detect_order_confirmation(tab, platform: str) -> str | None` tries a platform-keyed selector map (e.g. Amazon: `#confirmedOrderId`, BestBuy: `.order-confirmation-number`) and returns an order ID string or None. -- Called inside each plugin's `auto_buy()` AFTER the place-order click. The orchestrator only enqueues the `("confirmed", link, order_id, ts)` write-queue tuple when this returns a non-None value. -- This is the gate: `purchased=1` is set only on confirmed orders. Returning True from `auto_buy()` without a confirmed order_id logs a warning but does NOT mark purchased. -- Must come before retry logic in the build order to avoid double-buy on retry. +No external fonts. The existing system font stack (`-apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, sans-serif`) stays in `tokens.css` as a `--font-family-base` token. Never `@import url(https://fonts.googleapis.com/...)` -- violates the zero-CDN constraint and breaks offline use. -**Idempotent retry-on-cart** +### Pattern 2: SSE Endpoint with Per-Client asyncio.Queue Bridge -- Where: `core/retry.py` (new) + orchestrator `_try_auto_buy` (modified) -- `RetryPolicy(max_attempts: int, backoff_base: float, jitter: float)` dataclass. -- `with_retry(coro, policy)` async helper: catches the specific "not in cart" / "cart expired" exception class (raised by the plugin), backs off, and re-calls. Does NOT retry on a confirmed order_id (idempotency: checks `items.order_id IS NOT NULL` via the write-queue drain before any retry). -- Max cart attempts is configurable via `CheckoutConfig.max_cart_retries` in config.yml (default: 3). +**What:** This is the highest-risk component. BotService runs on a daemon thread with its OWN asyncio event loop (`self._loop`). Uvicorn runs FastAPI on a SEPARATE asyncio event loop. These loops are independent; asyncio primitives are not shared between them. -**Per-step / per-item checkout time budget** +**The correct bridge: uvicorn-side poll task only.** -- Where: `core/config_schema.py` (new `CheckoutConfig` model) + orchestrator `_check_and_buy` (modified) -- `CheckoutConfig.item_timeout_secs: int = 120` wraps the entire `_check_and_buy` call with `asyncio.timeout(item_timeout_secs)`. -- `CheckoutConfig.step_timeout_secs: int = 15` is passed to each `tab.select(selector, timeout=step_timeout_secs)` call inside plugins. Plugins currently hardcode `timeout=10`; step_timeout is the config-driven replacement. -- Both settings live in `AppConfig.checkout` (new sub-model, alongside the existing `debug`, `proxy`, `captcha` sub-models). +A background async task running inside uvicorn's event loop polls `BotService.get_status()` using `asyncio.to_thread()`. The result is distributed to all active per-client `asyncio.Queue` objects via `queue.put_nowait()`. All queues, all SSE generators, and the poll task live exclusively in uvicorn's event loop. The bot daemon thread is NEVER aware of the SSE hub. -**Central monitor-only safety gate** +Key safety properties: +- `asyncio.Queue` is NOT thread-safe (Python docs confirmed). Never call `put_nowait` from the bot's daemon thread. +- `asyncio.to_thread()` offloads the sync `svc.get_status()` call to a thread-pool worker, returning the uvicorn event loop immediately. Available in Python 3.9+; confirmed available since the project requires Python 3.11+. +- `BotService.get_status()` is documented as "cheap and non-blocking: all reads are in-memory only." `HealthRegistry.get_snapshot()` returns a deep copy with private keys stripped. No lock needed from the caller. +- `self._loop` on BotService is `None` when the bot is not running. The uvicorn-side poll handles the `running=False` case naturally (get_status() is safe before `start()` per REL-07). -- Where: `core/plugin_base.py` (modified ABC) + `core/config_schema.py` (modified `DebugConfig`) -- Problem: `debug.test_mode` is only read by `AmazonPlugin.auto_buy()`. The other 6 plugins call their place-order selector click unconditionally. -- Solution: add `DebugConfig.monitor_only: bool = True` (default True for safety). Add `RetailerPlugin.place_order_guarded(tab, selector: str) -> bool` as a concrete method on the ABC. This method: - 1. Checks `self.config.debug.monitor_only` (not `test_mode`; the two are separate: `test_mode` is Amazon-legacy, `monitor_only` is the universal v4.0 gate). - 2. If `monitor_only=True`: logs "MONITOR-ONLY: skipping place-order click" and returns False. - 3. If `monitor_only=False`: performs the click and returns True. -- All 7 plugins replace their direct `await place_order.click()` call with `await self.place_order_guarded(tab, selector)`. -- `test_mode` in AmazonPlugin is deprecated in the same pass: Amazon reads `monitor_only` going forward; `test_mode` stays in config for backward compat but Amazon stops reading it for the purchase gate. -- `BotService.start(monitor_only: bool = True)` passes the value into `AppConfig` before calling `async_main`, so the CLI/web UI can override it without editing config.yml. +**SseHub module (`web/sse_hub.py`):** ---- +```python +import asyncio + +class SseHub: + def __init__(self): + self._queues: set[asyncio.Queue] = set() + + def subscribe(self, q: asyncio.Queue): + self._queues.add(q) + + def unsubscribe(self, q: asyncio.Queue): + self._queues.discard(q) + + def broadcast(self, payload: dict): + dead = set() + for q in self._queues: + try: + q.put_nowait(payload) + except asyncio.QueueFull: + dead.add(q) + self._queues -= dead +``` -### Always-On Reliability - -**Per-coroutine supervision + backoff restart** - -- Where: `core/supervisor.py` (new) + `core/orchestrator.py` (modified) -- `supervise(coro_factory, name, policy: RetryPolicy)` is an async wrapper that: - 1. Runs `coro_factory()` inside `asyncio.shield` to prevent TaskGroup cancellation on a single plugin crash. - 2. On exception: logs the error with class name and plugin name, waits `policy.backoff_base * 2^attempt + jitter` seconds, then calls `coro_factory()` again up to `policy.max_attempts` times. - 3. After exhausting retries: marks the plugin as DEAD in `HealthRegistry`, logs ERROR, and returns without raising (the TaskGroup continues running other plugins). -- The orchestrator replaces `tg.create_task(run_plugin(...))` with `tg.create_task(supervise(lambda: run_plugin(...), name, policy))`. -- This is the key constraint: `asyncio.TaskGroup` propagates the FIRST unhandled exception to all sibling tasks, tearing down the whole group. The supervisor absorbs exceptions before they reach the TaskGroup boundary. - -**Browser-crash detection + relaunch** - -- Where: `core/supervisor.py` (new) + `core/plugin_base.py` (modified) -- Browser crash manifests as `nodriver` raising on `tab.evaluate()` or `driver.get()` with a connection error. The supervisor catches this class of exception specifically. -- On crash detection: the supervisor calls `plugin.relaunch()` — a new concrete method on the ABC that: - 1. Calls `plugin.teardown()` (best-effort; ignores errors). - 2. Re-runs `registry.assign_proxy(plugin)` and `registry.assign_solver(plugin)` to refresh the proxy assignment. - 3. Calls `plugin.setup()` (re-creates the Browser, re-applies stealth, re-registers proxy auth). - 4. Calls `plugin.login()` to re-authenticate. - 5. Calls `session_store.restore(plugin)` if a session file exists (restore cookies before login to minimize re-auth friction). -- The relaunch sequence order is: teardown → assign_proxy → setup (creates browser, applies stealth) → setup_proxy_auth → restore_session → login. -- `plugin.relaunch()` is a concrete ABC method. Plugins override only if their relaunch needs custom steps (e.g. Amazon passkey re-dismissal). - -**Encrypted session/cookie persistence** - -- Where: `core/session_store.py` (new) -- Pattern mirrors `EncryptedFileBackend` in `credentials.py`: Fernet + scrypt KDF, atomic write via `tempfile.mkstemp + os.replace`. -- `SessionStore.save(plugin_name: str, cookies: list[dict]) -> None` serializes cookie dicts to JSON, encrypts, writes to `data/sessions/{plugin_name}.bin`. -- `SessionStore.restore(tab, plugin_name: str) -> bool` decrypts and injects cookies via `tab.send(cdp.network.set_cookies(...))`. Returns True if file existed, False otherwise. -- Called in the plugin's `setup()` after stealth injection (before first navigation). Also called after relaunch. -- The passphrase is the same `SHOPBOT_STORE_PASSPHRASE` env var already used by `EncryptedFileBackend` — no new secret. -- Sessions are **never** written to DB or logged. The `data/sessions/` directory is gitignored. -- Must come before relaunch in the build order: relaunch calls `restore_session`. - -**DB read-path error isolation** - -- Where: `core/orchestrator.py` (modified `_check_and_buy` and `run_plugin`) -- Currently `get_items_sync` is called via `run_in_executor` inside `run_plugin`. If it raises (corrupted DB, locked file), the exception bubbles to the TaskGroup and kills the whole run. -- Fix: wrap the `run_in_executor(None, get_items_sync)` call in try/except inside `run_plugin`. On failure: log ERROR with the exception class, skip this poll cycle, continue the `while True` loop. Do not re-raise. -- Same isolation applied to all other `run_in_executor` DB reads in `_check_and_buy` (`get_item_notification_state_sync`, `get_last_price_sync`, `get_item_price_config_sync`). -- Write-queue drain already has this isolation (the `except Exception` in `_write_queue_drain`). The read path does not. - -**Per-item asyncio timeout** - -- Where: `core/orchestrator.py` (modified `_check_and_buy`) + `core/config_schema.py` (modified `CheckoutConfig`) -- Wrap the `await _check_and_buy(...)` call inside `run_plugin` with `async with asyncio.timeout(cfg.checkout.item_timeout_secs)`. On `TimeoutError`: log WARNING with item name and elapsed seconds, continue loop. -- The existing `asyncio.wait_for(event.wait(), timeout=300)` in `_wait_user_action` is a different guard (manual intervention wait). Both coexist; the outer item timeout is the hard ceiling. - -**Structured health/heartbeat surface** - -- Where: `core/health.py` (new) + `core/service.py` (modified `get_status`) + `core/orchestrator.py` (modified) -- `HealthRegistry` is a thread-safe dict wrapper: `{plugin_name: HealthEntry}`. -- `HealthEntry` fields: `status: Literal["starting", "running", "crashed", "dead", "relaunching"]`, `last_heartbeat: float` (monotonic), `consecutive_errors: int`, `last_error: str | None`, `items_checked: int`, `orders_confirmed: int`. -- Orchestrator updates `HealthRegistry` at: poll cycle start (heartbeat timestamp), after each successful `check_availability` call (items_checked++), after confirmed order (orders_confirmed++), on exception in supervisor (consecutive_errors++, status="crashed"), after relaunch (status="relaunching"), after relaunch success (status="running"). -- `BotService.get_status()` returns: `{"running": bool, "plugins": {name: entry_dict}, "uptime_secs": float}`. -- Notifications dispatcher already fans out events for `detected` and `purchased` actions. Add `health_degraded` event type for when `consecutive_errors` exceeds a threshold (configurable via `CheckoutConfig.alert_on_errors: int = 5`). -- The health dict is queried by the FastAPI web UI (`GET /status`) and the existing notification dispatcher without new IPC — `BotService.get_status()` is thread-safe because `HealthRegistry` uses a `threading.Lock`. +**Background poll task (runs in uvicorn event loop via lifespan):** ---- +```python +async def _poll_loop(svc, hub: SseHub, interval: float = 1.0): + log_cursor = 0 + while True: + await asyncio.sleep(interval) + status = await asyncio.to_thread(svc.get_status) + hub.broadcast({"type": "status", "data": status}) + new_lines, log_cursor = await asyncio.to_thread(tail_log_lines, log_cursor) + if new_lines: + hub.broadcast({"type": "log", "data": new_lines}) +``` -## Data Flow: v4.0 Acquisition Path +**SSE endpoint (`web/routes/sse.py`):** +```python +@router.get("/events") +async def sse_events(request: Request): + hub = request.app.state.sse_hub + queue: asyncio.Queue = asyncio.Queue(maxsize=50) + hub.subscribe(queue) + async def generator(): + try: + while True: + if await request.is_disconnected(): + break + try: + payload = await asyncio.wait_for(queue.get(), timeout=15.0) + yield f"data: {json.dumps(payload)}\n\n" + except asyncio.TimeoutError: + yield ": keep-alive\n\n" + finally: + hub.unsubscribe(queue) + return StreamingResponse( + generator(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) ``` -run_plugin (per plugin coroutine, supervised) - | - +-- asyncio.timeout(item_timeout_secs) - | | - | +-- _check_and_buy(plugin, name, link, auto_buy, write_queue) - | | - | +-- plugin.check_availability(link) -> bool - | | [on crash: supervisor catches, relaunch, retry] - | | - | +-- [available=True, auto_buy=True] - | | - | +-- _try_auto_buy_with_retry(plugin, name, link, write_queue, policy) - | | - | +-- [attempt loop, max_cart_retries] - | | | - | | +-- checkout_profile.fill_shipping_form(tab) - | | +-- plugin.place_order_guarded(tab, selector) - | | | [if monitor_only=True: log, return False, no retry] - | | +-- confirmation.detect_order_confirmation(tab, platform) - | | | [returns order_id or None] - | | +-- [order_id is not None]: - | | write_queue.put(("confirmed", link, order_id, ts)) - | | -> models: purchased=1, order_id, confirmed_at - | | dispatcher.notify("purchased") - | | +-- [order_id is None, attempt < max_cart_retries]: - | | backoff sleep, retry from cart step - | | - | +-- [all attempts exhausted]: log WARNING, return False - | - +-- [TimeoutError]: log WARNING, continue loop -write_queue (asyncio.Queue, drained by _write_queue_drain) - +-- ("purchased", link) -- legacy, still valid - +-- ("confirmed", link, order_id, ts) -- new v4.0 tag - +-- ("set_available", link, ts) -- existing - +-- ("clear_available", link) -- existing -``` +**FastAPI version note:** FastAPI 0.115.8 (the pinned version in `pyproject.toml`) does NOT include `fastapi.sse` or `EventSourceResponse`. That was added in FastAPI 0.135.0. Use `StreamingResponse` directly as shown above. No new dependency needed. Do NOT add `sse-starlette` unless there is a specific reason; raw `StreamingResponse` with `text/event-stream` is the correct zero-dep approach here. ---- +**Client-side (browser):** + +Replace `setInterval(pollStatus, 2000)` and `setInterval(pollLogs, 2000)` with a single `EventSource`: -## Data Flow: v4.0 Reliability Path +```js +const es = new EventSource('/api/events'); +es.onmessage = (e) => { + const msg = JSON.parse(e.data); + if (msg.type === 'status') updateStatusUI(msg.data); + if (msg.type === 'log') appendLogLines(msg.data); +}; +es.onerror = () => { /* EventSource reconnects automatically */ }; +// Polling fallback only for environments without EventSource (extremely rare) +if (typeof EventSource === 'undefined') { + setInterval(pollStatus, 2000); + setInterval(pollLogs, 2000); +} ``` -async_main - | - +-- _staggered_setup (unchanged; adds session restore per plugin after setup) - | +-- plugin.setup() - | +-- session_store.restore(tab, plugin_name) [new step] - | +-- plugin.login() - | - +-- asyncio.TaskGroup - | - +-- _write_queue_drain (unchanged) - | - +-- supervise(lambda: run_plugin(plugin, ...), name, policy) [wraps each plugin] - | - +-- run_plugin [while True loop] - | +-- DB read isolation (try/except around all run_in_executor reads) - | +-- heartbeat update at top of each cycle - | - +-- [exception in run_plugin]: - supervisor catches - | - +-- [connection/crash exception]: plugin.relaunch() - | +-- plugin.teardown() - | +-- registry.assign_proxy(plugin) - | +-- plugin.setup() [new browser, stealth, proxy auth] - | +-- session_store.restore(tab, plugin_name) - | +-- plugin.login() - | +-- health.update(plugin_name, status="relaunching") - | - +-- [other exception]: backoff, retry run_plugin - | - +-- [retries exhausted]: health.update(status="dead"), return + +`EventSource` reconnects automatically on connection loss. No additional reconnection logic needed. + +**Lifespan wiring in `web/__init__.py`:** + +```python +from contextlib import asynccontextmanager +from web.sse_hub import SseHub, _poll_loop + +@asynccontextmanager +async def lifespan(app): + hub = SseHub() + app.state.sse_hub = hub + task = asyncio.create_task(_poll_loop(app.state.svc, hub)) + yield + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + +def create_app(svc, is_non_local=False): + app = FastAPI(..., lifespan=lifespan) + app.state.svc = svc + ... ``` ---- +### Pattern 3: Read-Only History API Endpoints -## Schema Changes (models.py) +**What:** Two new GET routes in `web/routes/api.py`. No schema changes; all columns exist from v4.0 (BUY-04: `order_id`, `confirmed_at`, `checkout_attempts` in `items`; v3.0: `price_history` table). + +**Run/buy history (`GET /api/history`):** + +Query (add a helper to `models.py` or inline via `get_db_connection()`): ```sql --- Idempotent ALTER TABLE additions (same pattern as v3.0 price columns) -ALTER TABLE items ADD COLUMN order_id TEXT; -ALTER TABLE items ADD COLUMN confirmed_at TEXT; -ALTER TABLE items ADD COLUMN checkout_attempts INTEGER NOT NULL DEFAULT 0; +SELECT name, link, order_id, confirmed_at, checkout_attempts +FROM items +WHERE purchased = 1 AND order_id IS NOT NULL +ORDER BY confirmed_at DESC +LIMIT 50 ``` -New write-queue tags handled by `_dispatch_write`: - -| Tag | Tuple shape | Model operation | -|-----|------------|-----------------| -| `confirmed` | `("confirmed", link, order_id, ts)` | `UPDATE items SET purchased=1, order_id=?, confirmed_at=? WHERE link=?`; increments `checkout_attempts` | -| existing `purchased` | `("purchased", link)` | unchanged; still valid for legacy test paths | +Response shape: +```json +{ + "confirmed_orders": [ + { + "name": "string", + "link": "string", + "order_id": "string", + "confirmed_at": "ISO8601", + "checkout_attempts": 1 + } + ] +} +``` -The `purchased` write-queue path is NOT removed. It stays as the write path when `auto_buy()` succeeds but confirmation detection fails (e.g. confirmation page loads too slowly). In that case a WARNING is logged: "purchase click succeeded but confirmation not detected — marking purchased without order_id". +Wrap the sync DB call in `asyncio.to_thread()` to avoid blocking uvicorn's event loop. ---- +**Price-history series (`GET /api/price-history/{link_b64}`):** -## Config Schema Changes (config_schema.py) +Decode `link_b64` using the same urlsafe-base64 scheme as the existing `/api/items/{link_b64}` DELETE endpoint. Call `get_price_history_sync(link, limit=100)` (already in `models.py`). Series comes back newest-first; convert `price_cents -> price_dollars` at the API boundary. -```python -class CheckoutConfig(BaseModel): - item_timeout_secs: int = 120 # outer asyncio.timeout per item - step_timeout_secs: int = 15 # tab.select timeout per DOM step - max_cart_retries: int = 3 # retry-on-cart attempts - backoff_base: float = 2.0 # seconds; doubles per attempt - backoff_jitter: float = 1.0 # random uniform [0, jitter] added - alert_on_errors: int = 5 # consecutive errors before health alert - -class DebugConfig(BaseModel): - logging_level: int = 5 - test_mode: bool = True # Amazon legacy; kept for compat - monitor_only: bool = True # v4.0 universal safety gate (default safe) - -# AppConfig gains: -checkout: CheckoutConfig = CheckoutConfig() +Response shape: +```json +{ + "item_link": "string", + "series": [ + {"price_cents": 4999, "price_dollars": 49.99, "currency": "USD", "scraped_at": "ISO8601"} + ] +} ``` -New `SECRET_KEYS` additions (9 keys; no card numbers; PCI constraint preserved): +Note: price data is Amazon-only today (PRICE-02, Phase 16). Other plugins will return an empty `series`. Surface gracefully: if `series` is empty, the template renders placeholder text "No price history available for this plugin." + +### Pattern 4: Enhanced Log Reader + +**What:** Two additive functions in `web/log_reader.py`. Existing `read_recent_logs(n)` is NOT changed; the API endpoint gains optional query params. + +**`read_logs_filtered(n, level, plugin, search) -> list[str]`:** + +Reads the full log file (same `_log_path()` logic), applies filters AND-combined, returns last `n` matching lines: +- `level`: string like `"ERROR"` -- match lines containing `[ERROR]` (or whatever the logger format emits) +- `plugin`: substring match on plugin name (e.g. `"amazon"`) +- `search`: case-insensitive substring match anywhere in the line +- All params are optional/nullable; when all are None it degrades to `read_recent_logs(n)` + +Wire into `GET /api/logs?level=ERROR&plugin=amazon&search=captcha&n=100` with query params on the existing endpoint. + +**`tail_log_lines(after_line: int) -> tuple[list[str], int]`:** + +Reads the current log file, returns `(lines[after_line:], total_line_count)`. Called by `_poll_loop` every second. When the log file rolls over at midnight (new filename), `after_line` will exceed the new file's line count; detect this (return value `new_cursor < after_line`) and reset the cursor to 0. ```python -"CHECKOUT_FIRST_NAME", "CHECKOUT_LAST_NAME", -"CHECKOUT_ADDRESS_LINE1", "CHECKOUT_ADDRESS_LINE2", -"CHECKOUT_CITY", "CHECKOUT_STATE", "CHECKOUT_ZIP", -"CHECKOUT_COUNTRY", "CHECKOUT_PHONE", +def tail_log_lines(after_line: int) -> tuple[list[str], int]: + path = _log_path() + if not path.exists(): + return [], after_line + lines = path.read_text(encoding="utf-8", errors="replace").splitlines() + total = len(lines) + if after_line > total: + after_line = 0 # log rolled + return lines[after_line:], total ``` ---- +### Pattern 5: Vendored Sparkline Chart -## Architectural Patterns +**What:** Download `sparkline.min.js` from github.com/fnando/sparkline (MIT license, ~1KB minified, zero dependencies). Vendor it to `web/static/sparkline.js`. Include via `` in `dashboard.html`. -### Pattern 1: ABC Concrete Method as Cross-Cutting Gate +**Usage:** -`RetailerPlugin.place_order_guarded()` is a concrete method on the ABC that all 7 plugins call instead of a direct `.click()`. This is the only design that closes the 6-of-7 safety hole without editing each plugin independently. The alternative (per-plugin `if self.config.debug.monitor_only: return False`) would require 7 edit points and would break again on every new community plugin. +```html + +``` -```python -# core/plugin_base.py (addition to RetailerPlugin) -async def place_order_guarded(self, tab, selector: str) -> bool: - """Click the place-order button unless monitor_only is active. - - All 7 plugins replace their direct await place_order.click() with this. - Returns True if click was performed, False if suppressed. - """ - monitor_only = getattr(getattr(self.config, "debug", None), "monitor_only", True) - if monitor_only: - writeLog( - f"[{self.__class__.__name__}] MONITOR-ONLY: skipping place-order click", - "INFO", - ) - return False - element = await tab.select(selector, timeout=15) - if not element: - return False - await element.click() - return True +```js +// series from /api/price-history, reversed to ascending time order +const values = series.map(s => s.price_dollars).reverse(); +if (values.length > 0) { + sparkline(svgEl, values); +} ``` -### Pattern 2: Supervisor-Wrapped Coroutine Factory +Render one sparkline per item in the items table. Empty series: show placeholder text, do not call `sparkline()`. -The orchestrator passes a factory (not the coroutine itself) to the supervisor so the supervisor can create a fresh coroutine on each restart attempt. Passing the coroutine directly would fail on the second attempt because a consumed coroutine cannot be re-awaited. +**Alternative (server-side SVG, zero JS):** Generate sparkline `` in Jinja2 using a custom template filter that normalizes the price series to SVG coordinates. Avoids even the vendored JS file. Downsides: Jinja2 math for coordinate normalization is verbose (min/max/normalize across a series); chart is static (no hover tooltip). Recommend vendored JS as the simpler path given the existing inline-JS pattern in `dashboard.html`. -```python -# core/orchestrator.py (modified tg.create_task call) -tg.create_task( - supervise( - coro_factory=lambda: run_plugin(plugin, write_queue, poll_interval, dispatcher), - name=f"poll-{plugin.__class__.__name__}", - policy=RetryPolicy(max_attempts=5, backoff_base=2.0, jitter=1.0), - health=health_registry, - ), - name=f"supervised-{plugin.__class__.__name__}", -) +## Data Flow + +### SSE Push Flow + +``` +BotService daemon thread (owns its own asyncio loop) + | + | get_status() -> {"running": bool, "uptime_secs": float, "plugins": {...}} + | [in-memory read, no blocking, thread-safe read of HealthRegistry deep copy] + | + v +uvicorn event loop + | + | _poll_loop() -- background task on uvicorn's loop + | await asyncio.to_thread(svc.get_status) -- offloads to thread-pool worker + | + v +SseHub.broadcast({"type": "status", "data": status_dict}) + | + | iterates set of asyncio.Queue objects (all in uvicorn loop) + | queue.put_nowait(payload) + | + v +SSE generator for each client + | await asyncio.wait_for(queue.get(), timeout=15) + | yields "data: {...}\n\n" + | + v +Browser EventSource + | msg.type === "status" -> updateStatusUI() + | msg.type === "log" -> appendLogLines() ``` -### Pattern 3: Confirmation-Before-Purchased Write +### Log Tail Flow + +``` +Log file (logs/YYYYMONTHDD.log) -- sync writes by bot via writeLog() + | + v +_poll_loop(): + await asyncio.to_thread(tail_log_lines, cursor) + -> (new_lines, new_cursor) + | + v +SseHub.broadcast({"type": "log", "data": new_lines}) [if new_lines] + | + v +Browser: appends lines to log 
; trims to last 500 lines to cap DOM size
+```
 
-The orchestrator must NOT enqueue `("purchased", link)` until `detect_order_confirmation()` returns. The existing `_try_auto_buy` returns True immediately after `place_order.click()` without waiting for confirmation. The v4.0 path changes this:
+### Price History Flow
 
 ```
-auto_buy(url) -> bool
-  [does DOM interactions up to and including place_order_guarded]
-  [does NOT call confirmation detection -- that stays in orchestrator]
-  returns True = "place-order click was performed and not suppressed"
-
-_try_auto_buy_with_retry (orchestrator):
-  success = await plugin.auto_buy(link)
-  if not success:
-      return
-  order_id = await confirmation.detect_order_confirmation(tab, plugin_name)
-  if order_id:
-      write_queue.put(("confirmed", link, order_id, ts))
-  else:
-      writeLog("WARNING: place-order succeeded but confirmation not detected", "WARNING")
-      write_queue.put(("purchased", link))   # fallback; no order_id persisted
+Browser -- on page load, per item in items list
+  |
+  | fetch GET /api/price-history/{link_b64}
+  v
+web/routes/api.py
+  | await asyncio.to_thread(get_price_history_sync, link, 100)
+  v
+SQLite price_history table (WAL, existing connection pattern)
+  |
+  v
+JSON {series: [{price_cents, price_dollars, currency, scraped_at}, ...]}
+  |
+  v
+Browser: sparkline(svgEl, series.map(s => s.price_dollars).reverse())
 ```
 
-This preserves backward compatibility: a plugin that cannot return the browser tab (e.g. opens a new window) still falls back to the legacy `purchased` write.
+### Theme Init Flow (prevents FOUC)
 
-### Pattern 4: Reuse EncryptedFileBackend for Session Store
+```
+Browser parses 
+  |
+  v
+Inline ` (via
+      a test item name in test mode) and confirm it renders as text in the log viewer.
+- [ ] **Non-local banner:** Start the server with `--host 0.0.0.0` and confirm the
+      banner is visible and styled in both light and dark mode.
+- [ ] **`MC-4` test:** Runs and passes after all template changes.
+- [ ] **Price chart empty state:** Add a Walmart item (no price history) and confirm an
+      explicit empty-state message appears instead of a blank chart.
+- [ ] **Secrets in health surface:** Trigger a plugin health-degraded event and confirm
+      `get_status()` response JSON contains no URLs, no `str(exc)` values, only class
+      names and timestamps.
+- [ ] **Event loop not blocked:** While SSE is streaming, confirm `/api/status` HTTP
+      endpoint responds in < 200ms via curl/network tab.
+- [ ] **FOUC:** Load dashboard on a slow machine or with CPU throttling in devtools and
+      confirm no white flash before background color appears.
 
 ---
 
@@ -598,12 +631,14 @@ must be built before checkout retry is added to any plugin.
 
 | Pitfall | Recovery Cost | Recovery Steps |
 |---------|---------------|----------------|
-| Double-buy detected | HIGH | Check retailer order history immediately; cancel second order via retailer cancel page (usually cancellable within 30 min); mark item `purchased=True` in DB manually via `shoppybot items` CLI |
-| False-positive purchased notification | MEDIUM | Set `purchased=False` in DB via CLI; re-enable monitoring; add confirmation selector to CI test to prevent recurrence |
-| CVV in log file | HIGH | Rotate payment card CVV with bank immediately; delete log file; audit for other credential exposure; add SC grep CI check before next run |
-| Browser crash loop (no backoff) | MEDIUM | Kill all Chrome processes (`pkill chrome` / Task Manager); restart bot; backoff implementation is the permanent fix |
-| Session cookie plaintext on disk | HIGH | Delete session file; revoke session via retailer "sign out all devices"; migrate to encrypted session store; add `.gitignore` entry |
-| DB corruption | HIGH | Stop bot; restore from last `data/shop_py_bot.db` backup (or delete and re-seed from config); add `PRAGMA integrity_check` to startup path |
+| SSE generator blocks uvicorn loop | HIGH — all HTTP frozen | Kill uvicorn; move blocking call to `asyncio.to_thread()`; restart |
+| XSS via innerHTML | MEDIUM — template change + test | Replace `innerHTML` with `createElement`/`textContent`; regression test |
+| Leaked SSE generators | LOW — restart uvicorn | Add `is_disconnected()` check; generators clean up on next server restart |
+| FOUC | LOW — CSS fix | Inline critical tokens in `