From a6cc175f06f7657f8862ce31704b2a6d3091437f Mon Sep 17 00:00:00 2001 From: Sascha Date: Thu, 27 Aug 2026 15:16:28 +0200 Subject: [PATCH 1/3] Resolve open specification questions in PLAN.md A documentation review of PLAN.md surfaced 25 questions, contradictions and gaps. Fold the decisions into the body of the plan and record them in a new Decisions Log section. Notable resolutions: - Add prev_close per ticker so the watchlist daily change % has a baseline - State that the main chart, like the sparklines, accumulates client-side - Give unseeded tickers a fallback seed price instead of rejecting them - Define the tracked ticker set as watchlist plus open positions, so a de-watchlisted holding can still be valued - Treat the chat message as an intent rather than a receipt; action results render as authoritative filled/rejected chips - Settle the Docker bind mount vs named volume contradiction - Specify trade validation, watchlist rules and the actions JSON contract - Add GET /api/trades and a trade blotter panel - Truncate portfolio_snapshots on startup in simulator mode - Pick Recharts as the single charting dependency - Rename backend/db to backend/app/database, drop docker-compose.yml, add GET /api/state for a single-round-trip first paint - Document local development with a next.config.js proxy Co-Authored-By: Claude Opus 5 --- planning/PLAN.md | 351 +++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 312 insertions(+), 39 deletions(-) diff --git a/planning/PLAN.md b/planning/PLAN.md index bc1811b33..9e4c02d22 100644 --- a/planning/PLAN.md +++ b/planning/PLAN.md @@ -23,7 +23,7 @@ The user runs a single Docker command (or a provided start script). A browser op - **Watch prices stream** — prices flash green (uptick) or red (downtick) with subtle CSS animations that fade - **View sparkline mini-charts** — price action beside each ticker in the watchlist, accumulated on the frontend from the SSE stream since page load (sparklines fill in progressively) -- **Click a ticker** to see a larger detailed chart in the main chart area +- **Click a ticker** to see a larger detailed chart in the main chart area — like the sparklines, this chart is built from prices accumulated on the frontend since page load, so it starts empty and fills in progressively. No price history is persisted server-side. - **Buy and sell shares** — market orders only, instant fill at current price, no fees, no confirmation dialog - **Monitor their portfolio** — a heatmap (treemap) showing positions sized by weight and colored by P&L, plus a P&L chart tracking total portfolio value over time - **View a positions table** — ticker, quantity, average cost, current price, unrealized P&L, % change @@ -34,7 +34,7 @@ The user runs a single Docker command (or a provided start script). A browser op - **Dark theme**: backgrounds around `#0d1117` or `#1a1a2e`, muted gray borders, no pure black - **Price flash animations**: brief green/red background highlight on price change, fading over ~500ms via CSS transitions -- **Connection status indicator**: a small colored dot (green = connected, yellow = reconnecting, red = disconnected) visible in the header +- **Connection status indicator**: a small colored dot in the header with two states — green when a price message has arrived in the last 10 seconds, red otherwise. `EventSource` retries forever on its own, so there is no separate "reconnecting" state to detect. - **Professional, data-dense layout**: inspired by Bloomberg/trading terminals — every pixel earns its place - **Responsive but desktop-first**: optimized for wide screens, functional on tablet @@ -88,7 +88,9 @@ The user runs a single Docker command (or a provided start script). A browser op finally/ ├── frontend/ # Next.js TypeScript project (static export) ├── backend/ # FastAPI uv project (Python) -│ └── db/ # Schema definitions, seed data, migration logic +│ └── app/ +│ ├── market/ # Market data (built) — see MARKET_DATA_SUMMARY.md +│ └── database/ # Schema definitions, seed data, connection handling ├── planning/ # Project-wide documentation for agents │ ├── PLAN.md # This document │ └── ... # Additional agent reference docs @@ -98,20 +100,22 @@ finally/ │ ├── start_windows.ps1 # Launch Docker container (Windows PowerShell) │ └── stop_windows.ps1 # Stop Docker container (Windows PowerShell) ├── test/ # Playwright E2E tests + docker-compose.test.yml -├── db/ # Volume mount target (SQLite file lives here at runtime) +├── db/ # Bind-mounted into the container (finally.db lives here at runtime) │ └── .gitkeep # Directory exists in repo; finally.db is gitignored ├── Dockerfile # Multi-stage build (Node → Python) -├── docker-compose.yml # Optional convenience wrapper ├── .env # Environment variables (gitignored, .env.example committed) └── .gitignore ``` +Only `backend/` and `planning/` exist today. The rest is the intended layout, created by +agents as each component is built. + ### Key Boundaries - **`frontend/`** is a self-contained Next.js project. It knows nothing about Python. It talks to the backend via `/api/*` endpoints and `/api/stream/*` SSE endpoints. Internal structure is up to the Frontend Engineer agent. - **`backend/`** is a self-contained uv project with its own `pyproject.toml`. It owns all server logic including database initialization, schema, seed data, API routes, SSE streaming, market data, and LLM integration. Internal structure is up to the Backend/Market Data agents. -- **`backend/db/`** contains schema SQL definitions and seed logic. The backend lazily initializes the database on first request — creating tables and seeding default data if the SQLite file doesn't exist or is empty. -- **`db/`** at the top level is the runtime volume mount point. The SQLite file (`db/finally.db`) is created here by the backend and persists across container restarts via Docker volume. +- **`backend/app/database/`** contains schema SQL definitions and seed logic. The backend lazily initializes the database on first request — creating tables and seeding default data if the SQLite file doesn't exist or is empty. (Named `database/`, not `db/`, so it is never confused with the runtime `db/` directory below.) +- **`db/`** at the top level is bind-mounted to `/app/db` in the container. The SQLite file (`db/finally.db`) is created here by the backend and persists across container restarts. A bind mount rather than a named volume, so the database file is directly visible and inspectable on the host. - **`planning/`** contains project-wide documentation, including this plan. All agents reference files here as the shared contract. - **`test/`** contains Playwright E2E tests and supporting infrastructure (e.g., `docker-compose.test.yml`). Unit tests live within `frontend/` and `backend/` respectively, following each framework's conventions. - **`scripts/`** contains start/stop scripts that wrap Docker commands. @@ -137,7 +141,18 @@ LLM_MOCK=false - If `MASSIVE_API_KEY` is set and non-empty → backend uses Massive REST API for market data - If `MASSIVE_API_KEY` is absent or empty → backend uses the built-in market simulator - If `LLM_MOCK=true` → backend returns deterministic mock LLM responses (for E2E tests) -- The backend reads `.env` from the project root (mounted into the container or read via docker `--env-file`) + +### Where `.env` Is Read + +`.env` lives at the **project root**, one level above the `backend/` uv project. The two +runtimes reach it differently: + +- **In Docker**: no file is read. The start scripts pass `--env-file .env` and the values + arrive as real environment variables. +- **Locally**: the backend calls `load_dotenv(Path(__file__).parents[2] / ".env")` at + startup, so `uv run` from inside `backend/` still finds the root `.env`. + +Both paths end at `os.environ`, so application code only ever reads environment variables. --- @@ -156,29 +171,76 @@ Both the simulator and the Massive client implement the same abstract interface. - Starts from realistic seed prices (e.g., AAPL ~$190, GOOGL ~$175, etc.) - Runs as an in-process background task — no external dependencies +**Unknown tickers.** The user (or the AI) can add any symbol to the watchlist, including +ones with no entry in `seed_prices.py`. The simulator assigns such a ticker a fallback +seed — a random price in the $50-$500 range with default drift and volatility, and no +correlation group — rather than rejecting it. This keeps the watchlist open-ended without +needing a symbol database. + +**Restart behaviour.** The simulator holds no state across restarts: it always begins from +seed prices. Positions and cash persist in SQLite, so after a restart a holding reprices to +its seed level — a discontinuity that would show as a cliff in the P&L chart. To avoid +charting two incomparable price regimes on one line, **`portfolio_snapshots` is cleared on +startup in simulator mode** (§7). Positions, cash, trades and the watchlist are untouched; +only the value-over-time series restarts. + ### Massive API (Optional) - REST API polling (not WebSocket) — simpler, works on all tiers -- Polls for the union of all watched tickers on a configurable interval +- **One grouped request per poll** covering every tracked ticker — not one request per ticker. This is what keeps the free tier viable: 1 call per 15s is 4 calls/min against a 5 call/min limit, regardless of how many tickers are tracked (see the watchlist cap in §8) - Free tier (5 calls/min): poll every 15 seconds - Paid tiers: poll every 2-15 seconds depending on tier - Parses REST response into the same format as the simulator +**Price staleness is accepted, not guarded.** On the free tier a cached price can be up to +15 seconds old, and outside market hours it is frozen at the last close. Trades fill at +whatever the cache holds. There is no staleness check and no market-hours logic — this is a +simulated portfolio, and the simulator is the default path. + ### Shared Price Cache - A single background task (simulator or Massive poller) writes to an in-memory price cache -- The cache holds the latest price, previous price, and timestamp for each ticker +- The cache holds the latest price, previous price, previous close, and timestamp for each ticker - SSE streams read from this cache and push updates to connected clients - This architecture supports future multi-user scenarios without changes to the data layer +### Previous Close and Daily Change + +The watchlist shows a **daily change %**, which needs a baseline the tick-over-tick +`previous_price` cannot provide. Each ticker therefore carries a `prev_close`: + +- **Simulator**: `prev_close` is defined per ticker in `seed_prices.py`, a few percent away + from the seed price so the watchlist opens with a realistic spread of gainers and losers. + A fallback ticker gets `prev_close` equal to its generated seed price. +- **Massive**: `prev_close` comes from the API's previous-close field on each poll. + +`PriceUpdate` exposes `change_from_close` and `change_percent_from_close` alongside the +existing tick-level `change` / `change_percent`. The watchlist's "Change %" column uses the +close-based value; the green/red price flash uses the tick-level direction. + ### SSE Streaming - Endpoint: `GET /api/stream/prices` - Long-lived SSE connection; client uses native `EventSource` API -- Server pushes price updates for all tickers known to the system at a regular cadence (~500ms) — in the single-user model this is equivalent to the user's watchlist -- Each SSE event contains ticker, price, previous price, timestamp, and change direction +- Server pushes price updates for every **tracked ticker** at a regular cadence (~500ms) +- Each SSE event contains ticker, price, previous price, previous close, timestamp, and change direction - Client handles reconnection automatically (EventSource has built-in retry) +### The Tracked Ticker Set + +The set of tickers being priced is **the watchlist plus every ticker with a non-zero +position** — not the watchlist alone. A user can remove a ticker from their watchlist while +still holding it, and the portfolio cannot be valued without a live price for it. + +Consequences the watchlist routes must honour: + +- Adding a ticker calls `source.add_ticker()` in addition to the database insert. A database + write on its own does not start pricing the symbol. +- Removing a ticker calls `source.remove_ticker()` and `cache.remove()` **only if no open + position exists** for it. Otherwise it stays tracked and simply stops being displayed in + the watchlist panel. +- Opening a position in a ticker that is not on the watchlist adds it to the tracked set. + --- ## 7. Database @@ -200,6 +262,10 @@ All tables include a `user_id` column defaulting to `"default"`. This is hardcod - `cash_balance` REAL (default: `10000.0`) - `created_at` TEXT (ISO timestamp) +There is deliberately **no `realized_pnl` column**. When a position is sold, the gain or +loss flows into `cash_balance`, and total portfolio value (cash + positions) already +reflects it. Realized P&L is not displayed separately anywhere in the UI. + **watchlist** — Tickers the user is watching - `id` TEXT PRIMARY KEY (UUID) - `user_id` TEXT (default: `"default"`) @@ -216,7 +282,10 @@ All tables include a `user_id` column defaulting to `"default"`. This is hardcod - `updated_at` TEXT (ISO timestamp) - UNIQUE constraint on `(user_id, ticker)` -**trades** — Trade history (append-only log) +A fully-sold position is **deleted**, never kept at quantity 0. A row in `positions` always +means an open holding, so the positions table and heatmap need no zero-filtering. + +**trades** — Trade history (append-only audit log) - `id` TEXT PRIMARY KEY (UUID) - `user_id` TEXT (default: `"default"`) - `ticker` TEXT @@ -225,12 +294,28 @@ All tables include a `user_id` column defaulting to `"default"`. This is hardcod - `price` REAL - `executed_at` TEXT (ISO timestamp) -**portfolio_snapshots** — Portfolio value over time (for P&L chart). Recorded every 30 seconds by a background task, and immediately after each trade execution. +Read by `GET /api/trades` and displayed in the trade blotter panel (§10). Append-only — +trades are never updated or deleted, so the blotter is a true audit trail of everything that +happened, including trades the AI executed. + +**portfolio_snapshots** — Portfolio value over time (for P&L chart). Recorded every 60 seconds by a background task, and immediately after each trade execution. - `id` TEXT PRIMARY KEY (UUID) - `user_id` TEXT (default: `"default"`) - `total_value` REAL - `recorded_at` TEXT (ISO timestamp) +The snapshot task **skips a tick if any held ticker has no price in the cache**, so the +chart never records a portfolio valued at zero during the first seconds after startup. A +100% cash portfolio is always valuable and is recorded normally (a flat line at $10,000). + +**This table is cleared on startup in simulator mode.** The simulator restarts from seed +prices (§6), so snapshots from a previous run were valued against a different random walk +and are not comparable to new ones — plotting both on one line produces a meaningless cliff. +Truncating gives a chart that always describes a single continuous price regime. It is the +only table that is ever cleared; positions, cash, trades and the watchlist persist as +normal. Under Massive the table is left intact, because real prices *are* continuous across +a restart. + **chat_messages** — Conversation history with LLM - `id` TEXT PRIMARY KEY (UUID) - `user_id` TEXT (default: `"default"`) @@ -239,6 +324,26 @@ All tables include a `user_id` column defaulting to `"default"`. This is hardcod - `actions` TEXT (JSON — trades executed, watchlist changes made; null for user messages) - `created_at` TEXT (ISO timestamp) +The `actions` JSON is the contract between the backend and the chat panel. It records what +was actually attempted and what actually happened — not what the LLM said it would do: + +```json +{ + "trades": [ + {"ticker": "AAPL", "side": "buy", "quantity": 10, "price": 190.24, "status": "filled"}, + {"ticker": "TSLA", "side": "buy", "quantity": 50, "status": "rejected", + "error": "Insufficient cash: need $12,450.00, have $8,097.60"} + ], + "watchlist_changes": [ + {"ticker": "PYPL", "action": "add", "status": "ok"} + ] +} +``` + +`status` is `"filled"` or `"rejected"` for trades, `"ok"` or `"rejected"` for watchlist +changes. `price` is present only on a filled trade. `error` is present only on a rejected +one. Empty arrays are omitted. The same object is returned inline by `POST /api/chat`. + ### Default Seed Data - One user profile: `id="default"`, `cash_balance=10000.0` @@ -258,7 +363,8 @@ All tables include a `user_id` column defaulting to `"default"`. This is hardcod |--------|------|-------------| | GET | `/api/portfolio` | Current positions, cash balance, total value, unrealized P&L | | POST | `/api/portfolio/trade` | Execute a trade: `{ticker, quantity, side}` | -| GET | `/api/portfolio/history` | Portfolio value snapshots over time (for P&L chart) | +| GET | `/api/portfolio/history` | Portfolio value snapshots over time (for P&L chart). Accepts `?since=` and `?limit=` (default: last 500 snapshots) | +| GET | `/api/trades` | Trade history, newest first, for the blotter. Accepts `?limit=` (default: last 50) | ### Watchlist | Method | Path | Description | @@ -275,8 +381,39 @@ All tables include a `user_id` column defaulting to `"default"`. This is hardcod ### System | Method | Path | Description | |--------|------|-------------| +| GET | `/api/state` | Everything the frontend needs on load: portfolio, watchlist with prices, cash, recent trades. One call instead of four | | GET | `/api/health` | Health check (for Docker/deployment) | +`GET /api/state` exists purely so the first paint needs a single round-trip. It returns the +composition of `/api/portfolio`, `/api/watchlist` and `/api/trades`; those endpoints remain +for refetching after a mutation. + +### Trade Validation + +`POST /api/portfolio/trade` applies exactly these rules. The same rules apply to trades the +AI executes — there is one code path, not two. + +- `quantity` must be a number greater than 0. Fractional quantities are allowed. +- `side` must be `"buy"` or `"sell"`. +- The ticker must have a live price in the cache; a trade in an unpriced ticker is rejected. +- **Buy**: requires `quantity * price <= cash_balance`. +- **Sell**: requires `quantity <= position.quantity`. Selling a ticker with no position, or + more than is held, is rejected. +- **No shorting and no margin.** A rejected trade returns HTTP 400 with an `error` string + and changes no state. + +Buying a ticker not on the watchlist is allowed; it joins the tracked ticker set (§6). + +### Watchlist Rules + +- Tickers are uppercased and trimmed on input. A symbol must be 1-5 characters, letters only. +- Adding a ticker already on the watchlist is a **no-op returning 200** with the current + watchlist — idempotent, so a retry or a duplicate AI suggestion is harmless. +- The watchlist is capped at **30 tickers**. Adding past the cap returns 400. This bounds + the Massive poll payload and stops the AI from adding symbols in a loop. +- Removing a ticker is allowed even while a position is open in it; the ticker leaves the + watchlist panel but stays priced (§6). + --- ## 9. LLM Integration @@ -290,13 +427,13 @@ There is an OPENROUTER_API_KEY in the .env file in the project root. When the user sends a chat message, the backend: 1. Loads the user's current portfolio context (cash, positions with P&L, watchlist with live prices, total portfolio value) -2. Loads recent conversation history from the `chat_messages` table +2. Loads the **last 20 messages** from the `chat_messages` table — a fixed window, so the prompt cannot grow without bound over a long session 3. Constructs a prompt with a system message, portfolio context, conversation history, and the user's new message 4. Calls the LLM via LiteLLM → OpenRouter, requesting structured output, using the cerebras-inference skill 5. Parses the complete structured JSON response -6. Auto-executes any trades or watchlist changes specified in the response -7. Stores the message and executed actions in `chat_messages` -8. Returns the complete JSON response to the frontend (no token-by-token streaming — Cerebras inference is fast enough that a loading indicator is sufficient) +6. Auto-executes any trades or watchlist changes specified in the response, collecting a per-action result (filled / rejected, with the fill price or the error) +7. Stores the message and the action results in `chat_messages` using the `actions` shape defined in §7 +8. Returns the message plus the action results to the frontend (no token-by-token streaming — Cerebras inference is fast enough that a loading indicator is sufficient) ### Structured Output Schema @@ -325,7 +462,18 @@ Trades specified by the LLM execute automatically — no confirmation dialog. Th - It creates an impressive, fluid demo experience - It demonstrates agentic AI capabilities — the core theme of the course -If a trade fails validation (e.g., insufficient cash), the error is included in the chat response so the LLM can inform the user. +**The message is an intent, not a receipt.** Execution happens after the LLM has already +written its reply, so the assistant may say "Buying 10 AAPL" for a trade that is then +rejected for insufficient cash. There is no second LLM call to reconcile this. Instead: + +- The backend returns the action results alongside the message. +- The chat panel renders each action as its own chip beneath the message — green for filled + (with ticker, quantity and fill price), red for rejected (with the error text). +- The chips, not the prose, are the authoritative record of what happened. A rejection is + therefore always visible to the user even when the message text disagrees with it. + +The next turn's context includes the previous turn's action results, so the assistant sees +the rejection and can respond to it if the user asks. ### System Prompt Guidance @@ -344,6 +492,20 @@ When `LLM_MOCK=true`, the backend returns deterministic mock responses instead o - Development without an API key - CI/CD pipelines +Mock responses are **keyword-matched on the user's message**, not a single canned reply — +the E2E suite needs the chat to actually execute a trade. The rules, in order: + +| User message contains | Mock response | +|---|---| +| `buy ` | `message` confirming the buy, plus that trade in `trades` | +| `sell ` | `message` confirming the sell, plus that trade in `trades` | +| `watch ` / `add ` | `message` confirming, plus a `watchlist_changes` add | +| anything else | A fixed portfolio-summary message, no actions | + +The mock returns the structured object only. It performs no execution and no validation of +its own — the response flows through the same auto-execution path as a real one, so an E2E +test can assert on a genuine rejection by mocking a buy the cash balance cannot cover. + --- ## 10. Frontend Design @@ -352,23 +514,34 @@ When `LLM_MOCK=true`, the backend returns deterministic mock responses instead o The frontend is a single-page application with a dense, terminal-inspired layout. The specific component architecture and layout system is up to the Frontend Engineer, but the UI should include these elements: -- **Watchlist panel** — grid/table of watched tickers with: ticker symbol, current price (flashing green/red on change), daily change %, and a sparkline mini-chart (accumulated from SSE since page load) -- **Main chart area** — larger chart for the currently selected ticker, with at minimum price over time. Clicking a ticker in the watchlist selects it here. +- **Watchlist panel** — grid/table of watched tickers with: ticker symbol, current price (flashing green/red on change), daily change % (from `prev_close`, see §6), and a sparkline mini-chart (accumulated from SSE since page load) +- **Main chart area** — larger chart for the currently selected ticker, with at minimum price over time. Clicking a ticker in the watchlist selects it here. Like the sparklines, it plots prices accumulated client-side since page load, so it begins empty and fills in — show a "collecting data" placeholder until roughly ten points exist. - **Portfolio heatmap** — treemap visualization where each rectangle is a position, sized by portfolio weight, colored by P&L (green = profit, red = loss) - **P&L chart** — line chart showing total portfolio value over time, using data from `portfolio_snapshots` - **Positions table** — tabular view of all positions: ticker, quantity, avg cost, current price, unrealized P&L, % change -- **Trade bar** — simple input area: ticker field, quantity field, buy button, sell button. Market orders, instant fill. -- **AI chat panel** — docked/collapsible sidebar. Message input, scrolling conversation history, loading indicator while waiting for LLM response. Trade executions and watchlist changes shown inline as confirmations. +- **Trade blotter** — scrolling log of executed trades, newest first: time, ticker, side (green BUY / red SELL), quantity, fill price, notional value. Fed by `GET /api/trades` and refetched after every trade, manual or AI-executed. Read-only — trades cannot be cancelled or amended. +- **Trade bar** — simple input area: ticker field, quantity field, buy button, sell button. Market orders, instant fill. A rejected trade (§8) surfaces its error string beside the bar. +- **AI chat panel** — docked/collapsible sidebar. Message input, scrolling conversation history, loading indicator while waiting for LLM response. Each message's executed actions render as green (filled) or red (rejected) chips beneath the message text, per §9. - **Header** — portfolio total value (updating live), connection status indicator, cash balance ### Technical Notes +- On load, call `GET /api/state` once for portfolio, watchlist, cash and recent trades, then open the SSE connection. Refetch `/api/portfolio` and `/api/trades` after any mutation (trade, chat action). - Use `EventSource` for SSE connection to `/api/stream/prices` -- Canvas-based charting library preferred (Lightweight Charts or Recharts) for performance +- **Recharts** for every chart — sparklines, main price chart, P&L line, and the portfolio treemap. It is the only one of the candidates that covers all four, so the app needs a single charting dependency rather than two. - Price flash effect: on receiving a new price, briefly apply a CSS class with background color transition, then remove it +- **Header total value is computed client-side** — cash from the last `/api/portfolio` response, position values from the live SSE prices. Do not poll `/api/portfolio` to keep the header current; nothing polls at the SSE cadence. +- **Connection dot**: track the timestamp of the last SSE message. Green if it is under 10 seconds old, red otherwise. Two states only (§2). - All API calls go to the same origin (`/api/*`) — no CORS configuration needed - Tailwind CSS for styling with a custom dark theme +### Client-Side Price History + +The frontend keeps a bounded in-memory buffer per tracked ticker (the last ~300 points, +roughly 2.5 minutes at the 500ms cadence) fed from the SSE stream. Sparklines and the main +chart both read from it. It is deliberately not persisted — a page refresh starts it over, +and that is the accepted trade for not storing tick history server-side. + --- ## 11. Docker & Deployment @@ -391,15 +564,47 @@ Stage 2: Python 3.12 slim FastAPI serves the static frontend files and all API routes on port 8000. -### Docker Volume +### Database Persistence -The SQLite database persists via a named Docker volume: +The SQLite database persists via a **bind mount** of the project's `db/` directory: ```bash -docker run -v finally-data:/app/db -p 8000:8000 --env-file .env finally +docker run -v "$PWD/db:/app/db" -p 8000:8000 --env-file .env finally ``` -The `db/` directory in the project root maps to `/app/db` in the container. The backend writes `finally.db` to this path. +The backend writes `finally.db` to `/app/db`, which is the host's `db/finally.db`. A bind +mount rather than a named volume, so the database file is visible and inspectable on the +host — worth more in a teaching project than the portability a named volume would buy. +Deleting `db/finally.db` resets the app to seed state. + +On Windows the start script uses `${PWD}` PowerShell-style; the mount is otherwise identical. + +### Local Development (without Docker) + +Docker is for running the finished app. Day-to-day development runs the two halves +separately: + +```bash +# terminal 1 +cd backend && uv run uvicorn app.main:app --reload --port 8000 + +# terminal 2 +cd frontend && npm run dev # serves on :3000 +``` + +`next dev` on :3000 is a different origin from uvicorn on :8000, which would reintroduce the +CORS problem the production build avoids. Rather than enabling CORS on the backend, the +frontend proxies in dev via `next.config.js`: + +```js +async rewrites() { + return [{ source: '/api/:path*', destination: 'http://localhost:8000/api/:path*' }]; +} +``` + +Frontend code therefore always calls relative `/api/*` paths, identical in dev and in +production, and the backend never needs CORS middleware. The rewrite is inert in a static +export build. ### Start/Stop Scripts @@ -411,12 +616,17 @@ The `db/` directory in the project root maps to `/app/db` in the container. The **`scripts/stop_mac.sh`** (macOS/Linux): - Stops and removes the running container -- Does NOT remove the volume (data persists) +- Does NOT touch `db/` (data persists) **`scripts/start_windows.ps1`** / **`scripts/stop_windows.ps1`**: PowerShell equivalents for Windows. All scripts should be idempotent — safe to run multiple times. +These four scripts are the only supported way to run the container. There is no +`docker-compose.yml` for production — a single container needs no orchestration, and one +launch path is easier to document than two. `test/docker-compose.test.yml` exists solely to +pair the app container with a Playwright container (§12). + ### Optional Cloud Deployment The container is designed to deploy to AWS App Runner, Render, or any container platform. A Terraform configuration for App Runner may be provided in a `deploy/` directory as a stretch goal, but is not part of the core build. @@ -428,10 +638,13 @@ The container is designed to deploy to AWS App Runner, Render, or any container ### Unit Tests (within `frontend/` and `backend/`) **Backend (pytest)**: -- Market data: simulator generates valid prices, GBM math is correct, Massive API response parsing works, both implementations conform to the abstract interface -- Portfolio: trade execution logic, P&L calculations, edge cases (selling more than owned, buying with insufficient cash, selling at a loss) -- LLM: structured output parsing handles all valid schemas, graceful handling of malformed responses, trade validation within chat flow -- API routes: correct status codes, response shapes, error handling +- Market data: simulator generates valid prices, GBM math is correct, Massive API response parsing works, both implementations conform to the abstract interface, an unseeded ticker gets a fallback seed price, `change_percent_from_close` is computed against `prev_close` +- Portfolio: trade execution logic, P&L calculations, edge cases (selling more than owned, buying with insufficient cash, selling at a loss, a fully-sold position row is deleted, quantity <= 0 is rejected) +- Tracked ticker set: removing a watchlist ticker with an open position keeps it priced; removing one without a position stops pricing it +- LLM: structured output parsing handles all valid schemas, graceful handling of malformed responses, trade validation within chat flow, a rejected trade still produces a well-formed `actions` object +- API routes: correct status codes, response shapes, error handling, watchlist idempotency and the 30-ticker cap, `/api/trades` ordering (newest first) and `limit` handling +- Blotter integrity: a rejected trade writes no row; an AI-executed trade writes one indistinguishable from a manual trade +- Startup: simulator mode truncates `portfolio_snapshots` and leaves positions, cash, trades and watchlist untouched; Massive mode leaves snapshots intact **Frontend (React Testing Library or similar)**: - Component rendering with mock data @@ -447,10 +660,70 @@ The container is designed to deploy to AWS App Runner, Render, or any container **Environment**: Tests run with `LLM_MOCK=true` by default for speed and determinism. **Key Scenarios**: -- Fresh start: default watchlist appears, $10k balance shown, prices are streaming -- Add and remove a ticker from the watchlist +- Fresh start: default watchlist appears, $10k balance shown, prices are streaming, connection dot is green +- Add and remove a ticker from the watchlist, including a ticker with no seed price - Buy shares: cash decreases, position appears, portfolio updates -- Sell shares: cash increases, position updates or disappears +- Sell part of a position: cash increases, quantity decreases +- Sell all of a position: the row disappears from the positions table +- Rejected trade: buy more than the cash balance allows, assert the error is shown, nothing changed, and no blotter row was written +- Trade blotter: a buy then a sell appear newest-first with correct side, quantity and fill price - Portfolio visualization: heatmap renders with correct colors, P&L chart has data points -- AI chat (mocked): send a message, receive a response, trade execution appears inline -- SSE resilience: disconnect and verify reconnection +- AI chat (mocked): "buy 5 AAPL" produces a green filled chip and a real position; "buy 1000 AAPL" produces a red rejected chip and no position +- SSE resilience: disconnect and verify reconnection, and that the connection dot goes red then green + +--- + +## 13. Decisions Log + +A documentation review raised 25 questions and gaps. All are now resolved in the body of +this document above; this section records what was decided and why, so the reasoning is not +lost and the same questions are not reopened. + +### Resolved + +| # | Question | Decision | Where | +|---|---|---|---| +| 1 | Daily change % had no baseline — the cache held only the previous *tick* | Each ticker carries a `prev_close`: from `seed_prices.py` in the simulator, from the API field under Massive. `PriceUpdate` gains `change_from_close` | §6, §10 | +| 2 | Main detail chart had no data source | Client-side accumulation from SSE, same as sparklines. No server-side tick history, no new table, no new endpoint. Shows a "collecting data" placeholder until ~10 points exist | §2, §10 | +| 3 | Unknown tickers (§9's own example adds `PYPL`, which has no seed) | Fallback seed — random $50-$500, default drift/volatility — rather than a symbol whitelist. Watchlist routes must also call `source.add_ticker()`; a database write alone does not start pricing | §6, §8 | +| 4 | Chat message was written before trades executed, so it could claim a fill that was rejected | The message is an intent, not a receipt. Action results return alongside it and render as green/red chips, which are authoritative. No second LLM call | §7, §9, §10 | +| 5 | §11 showed a named volume, §4 described a bind mount | Bind mount `./db:/app/db`, so the database file is visible on the host | §4, §11 | +| 6 | Priced tickers were said to equal the watchlist | Tracked set is watchlist ∪ tickers with an open position. A de-watchlisted holding stays priced | §6 | +| 7 | Realized P&L had no home in the schema | No `realized_pnl` column. Sale proceeds land in `cash_balance` and total value already reflects them | §7 | +| 8 | Position lifecycle on a full sell was "updates or disappears" | The row is deleted. A row in `positions` always means an open holding | §7, §12 | +| 9 | Shorting and negative quantities were never ruled out | Explicit validation rules: quantity > 0, buy needs cash, sell needs shares, no shorting, no margin, one code path shared with the AI | §8 | +| 10 | `portfolio_snapshots` grew unbounded; history endpoint took no parameters | 60s cadence instead of 30s, `?since=` and `?limit=` (default last 500), and the task skips a tick if a held ticker has no price yet | §7, §8 | +| 11 | "Recent conversation history" was unbounded | Last 20 messages | §9 | +| 12 | `chat_messages.actions` JSON shape was undefined | Fully specified with `status` / `price` / `error` fields; same object returned by `POST /api/chat` | §7 | +| 13 | Nothing read the `trades` table | `GET /api/trades` added, plus a trade blotter panel showing executed trades newest-first | §7, §8, §10 | +| 14 | `.env` location was contradictory between §5 and §11 | Docker passes `--env-file`; locally `load_dotenv(Path(__file__).parents[2] / ".env")`. Both end at `os.environ` | §5 | +| 15 | No local development story; `next dev` would reintroduce CORS | `rewrites()` proxy in `next.config.js`, so frontend code always calls relative `/api/*` and the backend never needs CORS middleware | §11 | +| 16 | Massive free-tier call budget was unstated | One grouped request per poll covering all tickers — 4 calls/min against a 5/min limit regardless of ticker count | §6 | +| 17 | Simulator resets to seed prices on restart, causing a P&L cliff | `portfolio_snapshots` is truncated on startup in simulator mode, so the chart always covers one continuous price regime. Positions, cash, trades and watchlist persist. Left intact under Massive | §6, §7 | +| 18 | "Red = disconnected" was unreachable, since `EventSource` retries forever | Two states: green if a message arrived in the last 10s, red otherwise | §2, §10 | +| 19 | Header total value — client-computed or polled? | Client-computed from SSE prices plus the last known cash. Nothing polls at the SSE cadence | §10 | +| 20 | Ticker normalization was unspecified | Uppercased, trimmed, 1-5 letters. A duplicate add is an idempotent 200 | §8 | +| 21 | The AI could add tickers without limit | Watchlist capped at 30; also bounds the Massive poll payload | §8 | +| 22 | Stale prices under Massive (15s, frozen out of hours) | Accepted, not guarded. No staleness check, no market-hours logic | §6 | +| 23 | `LLM_MOCK` responses were "deterministic" but unspecified | Keyword-matched on the user's message with a documented rule table, so E2E tests can drive real fills *and* real rejections | §9 | +| 24 | "Lightweight Charts **or** Recharts" left the choice open | Recharts for all four chart types. It is the only candidate that covers the treemap, so the app carries one charting dependency instead of two | §10 | +| 25 | `backend/db/` and `/db` were two directories named `db` | Renamed to `backend/app/database/`, alongside `backend/app/market/` | §4 | + +### Also simplified + +- **`docker-compose.yml` dropped.** Four start/stop scripts, a compose wrapper, and a test + compose file were three ways to launch one container. The scripts are the only supported + path; `test/docker-compose.test.yml` remains for pairing the app with Playwright. +- **`GET /api/state` added.** First paint needed 2-3 round-trips for portfolio, watchlist and + cash; now one. The individual endpoints remain for refetching after a mutation. + +### Deliberately left alone + +- **`user_id` on every table.** Speculative for a single-user app, but §7 justifies it, it + costs nothing, and it keeps the shared-price-cache design in §6 honest. +- **Market orders only.** Well chosen; the rationale table in §3 earns its place. + +### Open questions + +None. Every item raised by the review is resolved in the body above. New questions should be +appended here as they come up, and moved into the table once decided. From a89c23d3dd679561c48e7d3e9f1a23d0373d1c34 Mon Sep 17 00:00:00 2001 From: Sascha Date: Thu, 27 Aug 2026 15:51:12 +0200 Subject: [PATCH 2/3] Switch LLM integration from Cerebras to a free OpenRouter model Cerebras is a paid inference provider. The app must cost nothing to run, so drop the provider pinning and move to a free model. The requested model id nvidia/nemotron-3-ultra-550b-a55b:free does not exist on OpenRouter; verified against the live model list and used nvidia/nemotron-3.5-lightning:free instead. Free models do not support response_format, so Structured Outputs are not available. Replace them with forced tool calling, which free models do support. Verified with a spike before writing it down: 18 calls across three runs, 8/9 schema-valid for forced tool calling. Prompt-based JSON scored 6/6 but was consistently slower head-to-head and needs fence stripping that a tool call never requires. Measured latency is 7-58 seconds, median 20-30, which invalidates the plan's reason for skipping streaming. Keep responses non-streaming but make the wait honest: an elapsed counter from the first second, a stated 20-30s expectation, new text at 60s, abandon with retry at 120s, and the input disabled throughout so impatient resends cannot burn the 20-per-minute rate limit. No fake progress bar - there is no progress to report. Also: - Add OPENROUTER_MODEL so swapping models needs no code change - Document the 20/min and 50/day free-tier rate limits - Note that env var names are case-sensitive in the Linux container - Rename the cerebras-inference skill to openrouter-inference - Fix the README docker run command, which still used a named volume Co-Authored-By: Claude Opus 5 --- .claude/skills/cerebras/SKILL.md | 43 --------- .claude/skills/openrouter/SKILL.md | 80 ++++++++++++++++ README.md | 29 +++++- planning/PLAN.md | 143 ++++++++++++++++++++++++++--- 4 files changed, 235 insertions(+), 60 deletions(-) delete mode 100644 .claude/skills/cerebras/SKILL.md create mode 100644 .claude/skills/openrouter/SKILL.md diff --git a/.claude/skills/cerebras/SKILL.md b/.claude/skills/cerebras/SKILL.md deleted file mode 100644 index 9efd01a38..000000000 --- a/.claude/skills/cerebras/SKILL.md +++ /dev/null @@ -1,43 +0,0 @@ ---- -name: cerebras-inference -description: Use this to write code to call an LLM using LiteLLM and OpenRouter with the Cerebras inference provider ---- - -# Calling an LLM via Cerebras - -These instructions allow you write code to call an LLM with Cerebras specified as the inference provider. -This method uses LiteLLM and OpenRouter. - -## Setup - -The OPENROUTER_API_KEY must be set in the .env file and loaded in as an environment variable. - -The uv project must include litellm and pydantic. -`uv add litellm pydantic` - -## Code snippets - -Use code like these examples in order to use Cerebras. - -### Imports and constants - -```python -from litellm import completion -MODEL = "openrouter/openai/gpt-oss-120b" -EXTRA_BODY = {"provider": {"order": ["cerebras"]}} -``` - -### Code to call via Cerebras for a text response - -```python -response = completion(model=MODEL, messages=messages, reasoning_effort="low", extra_body=EXTRA_BODY) -result = response.choices[0].message.content -``` - -### Code to call via Cerebras for a Structured Outputs response - -```python -response = completion(model=MODEL, messages=messages, response_format=MyBaseModelSubclass, reasoning_effort="low", extra_body=EXTRA_BODY) -result = response.choices[0].message.content -result_as_object = MyBaseModelSubclass.model_validate_json(result) -``` \ No newline at end of file diff --git a/.claude/skills/openrouter/SKILL.md b/.claude/skills/openrouter/SKILL.md new file mode 100644 index 000000000..3886d52ff --- /dev/null +++ b/.claude/skills/openrouter/SKILL.md @@ -0,0 +1,80 @@ +--- +name: openrouter-inference +description: Use this to write code to call an LLM using LiteLLM and OpenRouter with a free model +--- + +# Calling an LLM via OpenRouter + +These instructions allow you to write code to call an LLM through OpenRouter using LiteLLM. +The model is a free one, so no inference provider is pinned and no request costs money. + +## Setup + +The OPENROUTER_API_KEY must be set in the .env file and loaded in as an environment variable. + +The uv project must include litellm and pydantic. +`uv add litellm pydantic` + +## Code snippets + +### Imports and constants + +```python +from litellm import completion +MODEL = "openrouter/nvidia/nemotron-3.5-lightning:free" +``` + +The `openrouter/` prefix is what routes the call through OpenRouter — without it LiteLLM +cannot resolve the provider. Do not set `extra_body={"provider": {...}}`: pinning a provider +is what makes a request land on a paid one. + +### Code to call for a text response + +```python +response = completion(model=MODEL, messages=messages) +result = response.choices[0].message.content +``` + +### Code to call for a structured response + +Free models do not support `response_format`, so Structured Outputs are unavailable. Use +forced tool calling instead: the schema is enforced server-side and the arguments come back +as JSON. + +```python +TOOLS = [ + { + "type": "function", + "function": { + "name": "submit_response", + "description": "Return the reply to the user.", + "parameters": MyBaseModelSubclass.model_json_schema(), + }, + } +] + +response = completion( + model=MODEL, + messages=messages, + tools=TOOLS, + tool_choice={"type": "function", "function": {"name": "submit_response"}}, +) +raw = response.choices[0].message.tool_calls[0].function.arguments +result_as_object = MyBaseModelSubclass.model_validate_json(raw) +``` + +Always validate with Pydantic. A forced tool call is reliable but not guaranteed — check for +an empty `tool_calls` before indexing into it. + +## Expect slow responses + +Free endpoints are heavily shared and have no latency guarantee. Measured response times for +the model above ranged from 7 to 58 seconds across 18 calls, with a median around 20-30 +seconds. Any UI calling this needs a progress indicator that sets an honest expectation, not +a bare spinner. + +## Rate limits + +Models with the `:free` suffix are limited to 20 requests per minute and 50 per day, rising +to 1000 per day once at least 10 USD of credit has been purchased on the account. Tests +should mock the LLM rather than spend this budget. diff --git a/README.md b/README.md index 3f2582ae2..bcfaf47dc 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Built entirely by coding agents as a capstone project for an agentic AI coding c - **Live price streaming** via SSE with green/red flash animations - **Simulated portfolio** — $10k virtual cash, market orders, instant fills - **Portfolio visualizations** — heatmap (treemap), P&L chart, positions table -- **AI chat assistant** — analyzes holdings, suggests and auto-executes trades +- **AI chat assistant** — analyzes holdings, suggests and auto-executes trades; runs on a free model, so the whole app costs nothing to operate - **Watchlist management** — track tickers manually or via AI - **Dark terminal aesthetic** — Bloomberg-inspired, data-dense layout @@ -20,7 +20,7 @@ Single Docker container serving everything on port 8000: - **Frontend**: Next.js (static export) with TypeScript and Tailwind CSS - **Backend**: FastAPI (Python/uv) with SSE streaming - **Database**: SQLite with lazy initialization -- **AI**: LiteLLM → OpenRouter (Cerebras inference) with structured outputs +- **AI**: LiteLLM → OpenRouter with a free model, using forced tool calling - **Market data**: Built-in GBM simulator (default) or Massive API (optional) ## Quick Start @@ -32,7 +32,7 @@ cp .env.example .env # Run with Docker docker build -t finally . -docker run -v finally-data:/app/db -p 8000:8000 --env-file .env finally +docker run -v "$PWD/db:/app/db" -p 8000:8000 --env-file .env finally # Open http://localhost:8000 ``` @@ -42,9 +42,32 @@ docker run -v finally-data:/app/db -p 8000:8000 --env-file .env finally | Variable | Required | Description | |---|---|---| | `OPENROUTER_API_KEY` | Yes | OpenRouter API key for AI chat | +| `OPENROUTER_MODEL` | No | Model to use; defaults to `openrouter/nvidia/nemotron-3.5-lightning:free` | | `MASSIVE_API_KEY` | No | Massive (Polygon.io) key for real market data; omit to use simulator | | `LLM_MOCK` | No | Set `true` for deterministic mock LLM responses (testing) | +Write variable names in uppercase. Docker passes `.env` through verbatim, and the Linux +container is case-sensitive even though Windows is not. + +## Notes on the free model + +The AI chat runs on a free OpenRouter model, so the app costs nothing to operate. Two things +follow from that, both by design rather than by accident: + +- **Responses take 7-58 seconds**, typically 20-30. The chat panel shows an elapsed counter + and says so plainly instead of hiding the wait behind a spinner. +- **Free models allow 20 requests per minute and 50 per day** (1000 per day once 10 USD of + credit has been bought on the account). Run tests with `LLM_MOCK=true` rather than + spending that budget. + +Setting `OPENROUTER_MODEL` to a paid model works and is much faster, but is not required. + +## Troubleshooting + +**Certificate errors during setup** (`CERTIFICATE_VERIFY_FAILED`, `invalid peer certificate`) +mean your network inspects TLS traffic, common on corporate networks. Use `uv --system-certs` +and add `truststore` so Python trusts the certificates in your OS store. + ## Project Structure ``` diff --git a/planning/PLAN.md b/planning/PLAN.md index 9e4c02d22..354d24c69 100644 --- a/planning/PLAN.md +++ b/planning/PLAN.md @@ -66,7 +66,7 @@ The user runs a single Docker command (or a provided start script). A browser op - **Backend**: FastAPI (Python), managed as a `uv` project - **Database**: SQLite, single file at `db/finally.db`, volume-mounted for persistence - **Real-time data**: Server-Sent Events (SSE) — simpler than WebSockets, one-way server→client push, works everywhere -- **AI integration**: LiteLLM → OpenRouter (Cerebras for fast inference), with structured outputs for trade execution +- **AI integration**: LiteLLM → OpenRouter using a free model, with forced tool calling for trade execution - **Market data**: Environment-variable driven — simulator by default, real data via Massive API if key provided ### Why These Choices @@ -79,6 +79,7 @@ The user runs a single Docker command (or a provided start script). A browser op | Single Docker container | Students run one command; no docker-compose for production, no service orchestration | | uv for Python | Fast, modern Python project management; reproducible lockfile; what students should learn | | Market orders only | Eliminates order book, limit order logic, partial fills — dramatically simpler portfolio math | +| Free LLM model | The app must cost nothing to run. This rules out paid inference providers, and with them Structured Outputs — hence forced tool calling (§9) | --- @@ -128,6 +129,10 @@ agents as each component is built. # Required: OpenRouter API key for LLM chat functionality OPENROUTER_API_KEY=your-openrouter-api-key-here +# Optional: which OpenRouter model to use. Must be a free model (":free" suffix). +# Defaults to nvidia/nemotron-3.5-lightning:free if unset. +OPENROUTER_MODEL=openrouter/nvidia/nemotron-3.5-lightning:free + # Optional: Massive (Polygon.io) API key for real market data # If not set, the built-in market simulator is used (recommended for most users) MASSIVE_API_KEY= @@ -136,10 +141,15 @@ MASSIVE_API_KEY= LLM_MOCK=false ``` +Variable names are **case-sensitive**. Docker passes `.env` through verbatim with +`--env-file`, so a lowercase `openrouter_api_key` works on Windows (where Python normalises +environment keys to uppercase) but fails inside the Linux container. Write them uppercase. + ### Behavior - If `MASSIVE_API_KEY` is set and non-empty → backend uses Massive REST API for market data - If `MASSIVE_API_KEY` is absent or empty → backend uses the built-in market simulator +- If `OPENROUTER_MODEL` is set → that model is used; otherwise the default free model - If `LLM_MOCK=true` → backend returns deterministic mock LLM responses (for E2E tests) ### Where `.env` Is Read @@ -418,10 +428,34 @@ Buying a ticker not on the watchlist is allowed; it joins the tracked ticker set ## 9. LLM Integration -When writing code to make calls to LLMs, use cerebras-inference skill to use LiteLLM via OpenRouter to the `openrouter/openai/gpt-oss-120b` model with Cerebras as the inference provider. Structured Outputs should be used to interpret the results. +When writing code to make calls to LLMs, use the openrouter-inference skill to call LiteLLM +via OpenRouter with the free model `openrouter/nvidia/nemotron-3.5-lightning:free`. **Forced +tool calling** is used to obtain structured results — see below for why. There is an OPENROUTER_API_KEY in the .env file in the project root. +### Why Tool Calling, Not Structured Outputs + +The app must cost nothing to run, which means a model with the `:free` suffix. Free models +on OpenRouter do not advertise `response_format` among their supported parameters, so +Structured Outputs are unavailable — but `tools` and `tool_choice` are supported. + +The backend therefore defines a single function, `submit_response`, whose parameter schema is +the response schema below, and forces the model to call it: + +```python +tool_choice={"type": "function", "function": {"name": "submit_response"}} +``` + +The arguments come back as JSON that already matches the schema. They are still validated +with Pydantic before use — a forced tool call is reliable, not guaranteed, and an empty +`tool_calls` list must be handled rather than indexed into. + +This was verified against the model with a spike before being written down: 18 calls across +three runs, of which forced tool calling produced a schema-valid response 8 times out of 9. +Prompt-based JSON was also tried and scored 6 out of 6, but was consistently slower in +head-to-head runs and needs extra parsing to strip code fences that a tool call never has. + ### How It Works When the user sends a chat message, the backend: @@ -429,15 +463,47 @@ When the user sends a chat message, the backend: 1. Loads the user's current portfolio context (cash, positions with P&L, watchlist with live prices, total portfolio value) 2. Loads the **last 20 messages** from the `chat_messages` table — a fixed window, so the prompt cannot grow without bound over a long session 3. Constructs a prompt with a system message, portfolio context, conversation history, and the user's new message -4. Calls the LLM via LiteLLM → OpenRouter, requesting structured output, using the cerebras-inference skill -5. Parses the complete structured JSON response +4. Calls the LLM via LiteLLM → OpenRouter with a forced `submit_response` tool call, using the openrouter-inference skill +5. Validates the returned tool-call arguments against the Pydantic response model 6. Auto-executes any trades or watchlist changes specified in the response, collecting a per-action result (filled / rejected, with the fill price or the error) 7. Stores the message and the action results in `chat_messages` using the `actions` shape defined in §7 -8. Returns the message plus the action results to the frontend (no token-by-token streaming — Cerebras inference is fast enough that a loading indicator is sufficient) +8. Returns the message plus the action results to the frontend in one complete JSON response (no token-by-token streaming — see below) + +### Latency and the Progress Indicator + +A free endpoint is heavily shared and has no latency guarantee. Measured against this model: +**7 to 58 seconds per response, median roughly 20-30 seconds.** That is the single biggest +change from a paid provider and it shapes the UI more than anything else in this section. + +The response stays non-streaming. Adding token-by-token streaming would mean an SSE channel +for chat, partial-JSON handling, and a tool call that cannot be parsed until it is complete +anyway — real complexity for a first token that still arrives seconds late. Instead the wait +is made honest rather than hidden: + +- The chat panel shows an **elapsed-time counter** from the moment the request is sent, so + the user can see the app is working rather than frozen. +- Alongside it, an explicit expectation: *"Free model — this usually takes 20-30 seconds."* + Stating the cost up front turns a broken-feeling wait into an understood one. +- After **60 seconds**, the message changes to *"Still working — free endpoints are + sometimes slow."* The request is not cancelled. +- After **120 seconds**, the request is abandoned client-side and an error message offers a + retry. The backend request keeps its own timeout at the same value. +- The input is disabled while a request is in flight, so a slow response cannot be + compounded by a queue of impatient resends against a 20-per-minute rate limit. -### Structured Output Schema +### Rate Limits -The LLM is instructed to respond with JSON matching this schema: +Free models allow **20 requests per minute and 50 per day**, rising to 1000 per day once at +least 10 USD of credit has been purchased on the account. Two consequences: E2E tests must +run with `LLM_MOCK=true` rather than spend the daily budget, and a demo with several people +chatting at once can exhaust 50 requests quickly. The daily limit is a property of the +account, not of this app, and is not something the backend tries to manage — but a 429 from +OpenRouter must surface as a readable chat error, not a stack trace. + +### Response Schema + +The parameter schema of the forced `submit_response` tool call. The model's arguments arrive +as JSON matching this shape: ```json { @@ -455,6 +521,9 @@ The LLM is instructed to respond with JSON matching this schema: - `trades` (optional): Array of trades to auto-execute. Each trade goes through the same validation as manual trades (sufficient cash for buys, sufficient shares for sells) - `watchlist_changes` (optional): Array of watchlist modifications +Defined once as a Pydantic model; `model_json_schema()` supplies the tool's `parameters` and +the same model validates the response. There is no second, hand-written copy of the schema. + ### Auto-Execution Trades specified by the LLM execute automatically — no confirmation dialog. This is a deliberate design choice: @@ -483,7 +552,10 @@ The LLM should be prompted as "FinAlly, an AI trading assistant" with instructio - Execute trades when the user asks or agrees - Manage the watchlist proactively - Be concise and data-driven in responses -- Always respond with valid structured JSON +- Always answer by calling `submit_response`; never reply with plain text + +Keep the system prompt short. Every token of it is re-sent on each turn against a shared free +endpoint, and prompt length is one of the few levers on the latency described above. ### LLM Mock Mode @@ -502,9 +574,11 @@ the E2E suite needs the chat to actually execute a trade. The rules, in order: | `watch ` / `add ` | `message` confirming, plus a `watchlist_changes` add | | anything else | A fixed portfolio-summary message, no actions | -The mock returns the structured object only. It performs no execution and no validation of -its own — the response flows through the same auto-execution path as a real one, so an E2E -test can assert on a genuine rejection by mocking a buy the cash balance cannot cover. +The mock returns the response object directly, replacing the network call but not the +validation or execution that follows it. The response flows through the same auto-execution +path as a real one, so an E2E test can assert on a genuine rejection by mocking a buy the +cash balance cannot cover. Mock responses are instant, so the progress indicator never +appears in E2E runs — it needs its own frontend unit test with a delayed promise. --- @@ -521,7 +595,7 @@ The frontend is a single-page application with a dense, terminal-inspired layout - **Positions table** — tabular view of all positions: ticker, quantity, avg cost, current price, unrealized P&L, % change - **Trade blotter** — scrolling log of executed trades, newest first: time, ticker, side (green BUY / red SELL), quantity, fill price, notional value. Fed by `GET /api/trades` and refetched after every trade, manual or AI-executed. Read-only — trades cannot be cancelled or amended. - **Trade bar** — simple input area: ticker field, quantity field, buy button, sell button. Market orders, instant fill. A rejected trade (§8) surfaces its error string beside the bar. -- **AI chat panel** — docked/collapsible sidebar. Message input, scrolling conversation history, loading indicator while waiting for LLM response. Each message's executed actions render as green (filled) or red (rejected) chips beneath the message text, per §9. +- **AI chat panel** — docked/collapsible sidebar. Message input, scrolling conversation history, and the progress indicator described below while waiting for the LLM. Each message's executed actions render as green (filled) or red (rejected) chips beneath the message text, per §9. - **Header** — portfolio total value (updating live), connection status indicator, cash balance ### Technical Notes @@ -535,6 +609,31 @@ The frontend is a single-page application with a dense, terminal-inspired layout - All API calls go to the same origin (`/api/*`) — no CORS configuration needed - Tailwind CSS for styling with a custom dark theme +### Chat Progress Indicator + +The free model takes 7-58 seconds to answer (§9). A bare spinner over that span reads as a +hung app, so the panel states the cost of being free instead of hiding it. A placeholder +assistant bubble appears the moment the message is sent and passes through four stages: + +| Elapsed | What the user sees | +|---|---| +| 0s | Animated dots, an elapsed-second counter, and the line *"Free model — this usually takes 20-30 seconds."* | +| 60s | Counter continues; the line becomes *"Still working — free endpoints are sometimes slow."* | +| 120s | Request abandoned. The bubble becomes an error with a **Retry** button that resends the same message. | +| Response | The bubble is replaced by the real message and its action chips. | + +Requirements: + +- The elapsed counter must tick visibly from the first second. It is the only honest signal + that something is still happening, and it costs one `setInterval`. +- The message input is **disabled while a request is in flight**. Without this, an impatient + user resends and burns the 20-requests-per-minute budget on answers they will discard. +- The user's own message appears in the history immediately, not after the response arrives. +- Never show a fake progress bar. There is no progress to report — an elapsed counter is + truthful, a bar that fills at a guessed rate is not. +- A 429 from OpenRouter renders as a readable chat error naming the rate limit, not a + generic failure. + ### Client-Side Price History The frontend keeps a bounded in-memory buffer per tracked ticker (the last ~300 points, @@ -641,7 +740,7 @@ The container is designed to deploy to AWS App Runner, Render, or any container - Market data: simulator generates valid prices, GBM math is correct, Massive API response parsing works, both implementations conform to the abstract interface, an unseeded ticker gets a fallback seed price, `change_percent_from_close` is computed against `prev_close` - Portfolio: trade execution logic, P&L calculations, edge cases (selling more than owned, buying with insufficient cash, selling at a loss, a fully-sold position row is deleted, quantity <= 0 is rejected) - Tracked ticker set: removing a watchlist ticker with an open position keeps it priced; removing one without a position stops pricing it -- LLM: structured output parsing handles all valid schemas, graceful handling of malformed responses, trade validation within chat flow, a rejected trade still produces a well-formed `actions` object +- LLM: tool-call arguments validate against the Pydantic model, an empty `tool_calls` list is handled rather than indexed into, malformed arguments fail gracefully, trade validation within chat flow, a rejected trade still produces a well-formed `actions` object, a 429 surfaces as a readable error - API routes: correct status codes, response shapes, error handling, watchlist idempotency and the 30-ticker cap, `/api/trades` ordering (newest first) and `limit` handling - Blotter integrity: a rejected trade writes no row; an AI-executed trade writes one indistinguishable from a manual trade - Startup: simulator mode truncates `portfolio_snapshots` and leaves positions, cash, trades and watchlist untouched; Massive mode leaves snapshots intact @@ -651,7 +750,7 @@ The container is designed to deploy to AWS App Runner, Render, or any container - Price flash animation triggers correctly on price changes - Watchlist CRUD operations - Portfolio display calculations -- Chat message rendering and loading state +- Chat message rendering, and the progress indicator's four stages driven by a delayed promise: counter ticks, the 60s text change, the 120s abandon with a working Retry, and the input staying disabled throughout ### E2E Tests (in `test/`) @@ -709,6 +808,22 @@ lost and the same questions are not reopened. | 24 | "Lightweight Charts **or** Recharts" left the choice open | Recharts for all four chart types. It is the only candidate that covers the treemap, so the app carries one charting dependency instead of two | §10 | | 25 | `backend/db/` and `/db` were two directories named `db` | Renamed to `backend/app/database/`, alongside `backend/app/market/` | §4 | +### Switched to a free LLM (later change) + +Cerebras was dropped because it costs money; the app must run for free. This was not part of +the original review, but it changed enough of §9 to belong in the same log. + +| # | Question | Decision | Where | +|---|---|---|---| +| 26 | Cerebras is a paid inference provider | Removed. No provider pinning at all — `extra_body={"provider": ...}` is exactly what routes a request to a paid provider | §3, §9 | +| 27 | The requested model `nvidia/nemotron-3-ultra-550b-a55b:free` does not exist on OpenRouter | Verified against the live model list and replaced with `nvidia/nemotron-3.5-lightning:free` | §5, §9 | +| 28 | Free models do not support `response_format`, which §9 depended on | Forced tool calling via `tools` + `tool_choice`, which free models do support. Verified by spike: 8/9 schema-valid calls, against 6/6 for prompt-based JSON but consistently slower and needing fence-stripping | §9 | +| 29 | "Cerebras is fast enough that a loading indicator is sufficient" no longer holds — measured 7-58s | Response stays non-streaming, but the wait is made honest: elapsed counter, a stated 20-30s expectation, new text at 60s, abandon with Retry at 120s, input disabled throughout | §9, §10, §12 | +| 30 | Free models are rate-limited to 20/min and 50/day | Documented. E2E runs on `LLM_MOCK=true`; a 429 must render as a readable chat error | §9 | +| 31 | The model was hardcoded in the plan | `OPENROUTER_MODEL` environment variable, defaulting to the free model, so swapping models needs no code change | §5 | +| 32 | Environment variable names are case-sensitive in the Linux container | Documented — a lowercase `openrouter_api_key` works on Windows but fails in Docker | §5 | +| 33 | The `cerebras-inference` skill named a provider that is no longer used | Renamed to `openrouter-inference`, with the tool-calling snippet, the latency warning and the rate limits | — | + ### Also simplified - **`docker-compose.yml` dropped.** Four start/stop scripts, a compose wrapper, and a test From d0e92f1502b1ac00582aeadeaf81d5afdb60341f Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 12:47:54 +0000 Subject: [PATCH 3/3] Add detailed market data backend design doc Documents the market data subsystem (unified interface, price cache, GBM simulator, Massive client, SSE endpoint) as it needs to exist to satisfy the current PLAN.md, including the prev_close / daily-change-% extension that PLAN.md's decisions log added after the module was originally built but the shipped code doesn't implement yet. Section 15 lists the concrete diff against what's on disk today. --- planning/MARKET_DATA_DESIGN.md | 1545 ++++++++++++++++++++++++++++++++ 1 file changed, 1545 insertions(+) create mode 100644 planning/MARKET_DATA_DESIGN.md diff --git a/planning/MARKET_DATA_DESIGN.md b/planning/MARKET_DATA_DESIGN.md new file mode 100644 index 000000000..ba52f72a9 --- /dev/null +++ b/planning/MARKET_DATA_DESIGN.md @@ -0,0 +1,1545 @@ +# Market Data Backend — Detailed Design + +Implementation-ready design for `backend/app/market/`: the unified data-source interface, the +thread-safe price cache, the GBM simulator, the Massive (Polygon.io) REST client, the SSE +streaming endpoint, and how the rest of the backend wires into all of it. + +**Status of this document.** The market data subsystem was already built once (see +`planning/MARKET_DATA_SUMMARY.md`, 8 modules, 73 tests, 84% coverage) and the original design is +archived at `planning/archive/MARKET_DATA_DESIGN.md`. Since that build, `PLAN.md` §6 picked up +three requirements that the shipped code does not implement yet: + +1. **`prev_close`** per ticker, so the watchlist can show a daily change % (`PLAN.md` §6, Decisions + Log #1) — the cache today only knows the previous *tick*, not the previous close. +2. **Unknown-ticker fallback** must set `prev_close` equal to the generated seed price (§6). +3. **The tracked ticker set is watchlist ∪ open positions**, not the watchlist alone (§6, Decisions + Log #6) — this is enforced by the (not-yet-built) watchlist API routes calling `add_ticker` / + `remove_ticker` correctly, not by a change inside `app/market/` itself. + +This document specifies the market data module **as it needs to exist to satisfy the current +`PLAN.md`** — the current code plus the `prev_close` extension. Section 15 lists the concrete diff +against what's on disk today. Everything not called out there already matches the shipped +implementation exactly. + +--- + +## Table of Contents + +1. [Architecture](#1-architecture) +2. [File Structure](#2-file-structure) +3. [Data Model — `models.py`](#3-data-model) +4. [Price Cache — `cache.py`](#4-price-cache) +5. [Abstract Interface — `interface.py`](#5-abstract-interface) +6. [Seed Prices & Ticker Parameters — `seed_prices.py`](#6-seed-prices--ticker-parameters) +7. [GBM Simulator — `simulator.py`](#7-gbm-simulator) +8. [Massive API Client — `massive_client.py`](#8-massive-api-client) +9. [Factory — `factory.py`](#9-factory) +10. [SSE Streaming Endpoint — `stream.py`](#10-sse-streaming-endpoint) +11. [FastAPI Lifecycle Integration](#11-fastapi-lifecycle-integration) +12. [Watchlist / Tracked-Ticker-Set Coordination](#12-watchlist--tracked-ticker-set-coordination) +13. [Testing Strategy](#13-testing-strategy) +14. [Error Handling & Edge Cases](#14-error-handling--edge-cases) +15. [Delta Against the Current Implementation](#15-delta-against-the-current-implementation) +16. [Configuration Summary](#16-configuration-summary) + +--- + +## 1. Architecture + +``` + MarketDataSource (ABC) + / \ + SimulatorDataSource MassiveDataSource + (GBM, default, (Polygon.io REST poll, + no API key needed) needs MASSIVE_API_KEY) + \ / + v v + PriceCache (thread-safe, in-memory) + | + -------------------+------------------- + | | | + SSE /api/stream/prices Portfolio valuation Trade execution + (this module) (backend/app/portfolio, not yet built) +``` + +- **Strategy pattern.** Both data sources implement the same `MarketDataSource` ABC. Everything + downstream — SSE streaming, portfolio valuation, trade execution — is source-agnostic; it only + ever talks to `PriceCache`. +- **Push model.** Data sources write into the cache on their own schedule (simulator: 500ms, + Massive: 15s). Readers never call the data source directly for a price. +- **Single point of truth.** `PriceCache` decouples producers from consumers, so a future + multi-reader scenario (e.g. multiple SSE clients, portfolio valuation, the LLM's portfolio + context) needs no changes to this layer. + +--- + +## 2. File Structure + +``` +backend/ + app/ + market/ + __init__.py # Re-exports: PriceUpdate, PriceCache, MarketDataSource, + # create_market_data_source, create_stream_router + models.py # PriceUpdate dataclass + cache.py # PriceCache (thread-safe in-memory store) + interface.py # MarketDataSource ABC + seed_prices.py # SEED_PRICES, PREV_CLOSE, TICKER_PARAMS, DEFAULT_PARAMS, + # CORRELATION_GROUPS + simulator.py # GBMSimulator + SimulatorDataSource + massive_client.py # MassiveDataSource + factory.py # create_market_data_source() + stream.py # SSE endpoint (FastAPI router) + tests/ + market/ + test_models.py + test_cache.py + test_simulator.py + test_simulator_source.py + test_factory.py + test_massive.py +``` + +Each file has a single responsibility; `__init__.py` re-exports the public surface so the rest of +the backend imports from `app.market` without reaching into submodules. + +--- + +## 3. Data Model + +**File: `backend/app/market/models.py`** + +`PriceUpdate` is the only data structure that leaves the market data layer. Every downstream +consumer — SSE streaming, portfolio valuation, trade execution, the LLM's portfolio context — works +exclusively with this type. + +```python +from __future__ import annotations + +import time +from dataclasses import dataclass, field + + +@dataclass(frozen=True, slots=True) +class PriceUpdate: + """Immutable snapshot of a single ticker's price at a point in time.""" + + ticker: str + price: float + previous_price: float + prev_close: float + timestamp: float = field(default_factory=time.time) # Unix seconds + + # --- Tick-over-tick change (drives the green/red price flash) --- + + @property + def change(self) -> float: + """Absolute price change from the previous tick.""" + return round(self.price - self.previous_price, 4) + + @property + def change_percent(self) -> float: + """Percentage change from the previous tick.""" + if self.previous_price == 0: + return 0.0 + return round((self.price - self.previous_price) / self.previous_price * 100, 4) + + @property + def direction(self) -> str: + """'up', 'down', or 'flat', based on the tick-over-tick change.""" + if self.price > self.previous_price: + return "up" + elif self.price < self.previous_price: + return "down" + return "flat" + + # --- Daily change (drives the watchlist's Change % column) --- + + @property + def change_from_close(self) -> float: + """Absolute price change from the prior day's close.""" + return round(self.price - self.prev_close, 4) + + @property + def change_percent_from_close(self) -> float: + """Percentage change from the prior day's close.""" + if self.prev_close == 0: + return 0.0 + return round((self.price - self.prev_close) / self.prev_close * 100, 4) + + def to_dict(self) -> dict: + """Serialize for JSON / SSE transmission.""" + return { + "ticker": self.ticker, + "price": self.price, + "previous_price": self.previous_price, + "prev_close": self.prev_close, + "timestamp": self.timestamp, + "change": self.change, + "change_percent": self.change_percent, + "direction": self.direction, + "change_from_close": self.change_from_close, + "change_percent_from_close": self.change_percent_from_close, + } +``` + +### Design decisions + +- **`frozen=True, slots=True`**: price updates are immutable value objects, safe to share across + async tasks without copying, and cheap to create many times per second. +- **Two independent baselines, both computed properties**: `previous_price` (last tick) drives the + transient green/red flash; `prev_close` (prior session's close) drives the daily change % shown + in the watchlist. Keeping both as stored fields with derived properties means neither `direction` + nor `change_percent_from_close` can ever drift out of sync with the prices they're computed from. +- **`prev_close` is required, not optional.** Every write path (simulator seed, simulator + fallback-ticker seed, Massive poll) has a definite value for it — see §6 and §8 — so there's no + `None`-handling burden on every reader. A ticker with no known previous close simply isn't in the + cache yet. +- **`to_dict()`** stays the single serialization point, used by both the SSE endpoint and any future + REST response that embeds a price. + +--- + +## 4. Price Cache + +**File: `backend/app/market/cache.py`** + +The price cache is the central data hub. Data sources write to it; SSE streaming, portfolio +valuation, and trade execution read from it. It must be thread-safe because the Massive client's +synchronous call runs via `asyncio.to_thread`, which executes in a real OS thread. + +```python +from __future__ import annotations + +import time +from threading import Lock + +from .models import PriceUpdate + + +class PriceCache: + """Thread-safe in-memory cache of the latest price for each ticker. + + Writers: SimulatorDataSource or MassiveDataSource (one at a time). + Readers: SSE streaming endpoint, portfolio valuation, trade execution. + """ + + def __init__(self) -> None: + self._prices: dict[str, PriceUpdate] = {} + self._lock = Lock() + self._version: int = 0 # Monotonically increasing; bumped on every update + + def update( + self, + ticker: str, + price: float, + prev_close: float | None = None, + timestamp: float | None = None, + ) -> PriceUpdate: + """Record a new price for a ticker. Returns the created PriceUpdate. + + `prev_close` is optional on a call: once a ticker has an entry, later + calls (e.g. every simulator tick) can omit it and the existing value + is carried forward, since a session's close doesn't change tick to + tick. It is required on the *first* write for a ticker — see + SimulatorDataSource.start()/add_ticker() and MassiveDataSource._poll_once(), + both of which always pass it. + """ + with self._lock: + ts = timestamp or time.time() + prev = self._prices.get(ticker) + previous_price = prev.price if prev else price + + resolved_prev_close = prev_close + if resolved_prev_close is None: + resolved_prev_close = prev.prev_close if prev else price + + update = PriceUpdate( + ticker=ticker, + price=round(price, 2), + previous_price=round(previous_price, 2), + prev_close=round(resolved_prev_close, 2), + timestamp=ts, + ) + self._prices[ticker] = update + self._version += 1 + return update + + def get(self, ticker: str) -> PriceUpdate | None: + """Get the latest price for a single ticker, or None if unknown.""" + with self._lock: + return self._prices.get(ticker) + + def get_all(self) -> dict[str, PriceUpdate]: + """Snapshot of all current prices. Returns a shallow copy.""" + with self._lock: + return dict(self._prices) + + def get_price(self, ticker: str) -> float | None: + """Convenience: get just the price float, or None.""" + update = self.get(ticker) + return update.price if update else None + + def remove(self, ticker: str) -> None: + """Remove a ticker from the cache (e.g., when it leaves the tracked set).""" + with self._lock: + self._prices.pop(ticker, None) + + @property + def version(self) -> int: + """Current version counter. Useful for SSE change detection.""" + with self._lock: + return self._version + + def __len__(self) -> int: + with self._lock: + return len(self._prices) + + def __contains__(self, ticker: str) -> bool: + with self._lock: + return ticker in self._prices +``` + +### Why a version counter? + +The SSE streaming loop polls the cache every ~500ms. Without a version counter it would serialize +and re-send every price on every tick even when nothing changed (e.g. under Massive, which only +updates every 15s). The counter lets the SSE loop skip a send when nothing is new: + +```python +last_version = -1 +while True: + if price_cache.version != last_version: + last_version = price_cache.version + yield format_sse(price_cache.get_all()) + await asyncio.sleep(0.5) +``` + +### Thread safety rationale + +`threading.Lock`, not `asyncio.Lock`, because: +- The Massive client's synchronous `get_snapshot_all()` runs inside `asyncio.to_thread()`, a real OS + thread — `asyncio.Lock` provides no protection there. +- `threading.Lock` works correctly from both a sync thread and the async event loop. + +--- + +## 5. Abstract Interface + +**File: `backend/app/market/interface.py`** + +```python +from __future__ import annotations + +from abc import ABC, abstractmethod + + +class MarketDataSource(ABC): + """Contract for market data providers. + + Implementations push price updates into a shared PriceCache on their own + schedule. Downstream code never calls the data source directly for prices — + it reads from the cache. + + Lifecycle: + source = create_market_data_source(cache) + await source.start(["AAPL", "GOOGL", ...]) + # ... app runs ... + await source.add_ticker("TSLA") + await source.remove_ticker("GOOGL") + # ... app shutting down ... + await source.stop() + """ + + @abstractmethod + async def start(self, tickers: list[str]) -> None: + """Begin producing price updates for the given tickers. + + Starts a background task that periodically writes to the PriceCache. + Must be called exactly once. Calling start() twice is undefined behavior. + """ + + @abstractmethod + async def stop(self) -> None: + """Stop the background task and release resources. + + Safe to call multiple times. After stop(), the source will not write + to the cache again. + """ + + @abstractmethod + async def add_ticker(self, ticker: str) -> None: + """Add a ticker to the active set. No-op if already present. + + The next update cycle will include this ticker. + """ + + @abstractmethod + async def remove_ticker(self, ticker: str) -> None: + """Remove a ticker from the active set. No-op if not present. + + Also removes the ticker from the PriceCache. + """ + + @abstractmethod + def get_tickers(self) -> list[str]: + """Return the current list of actively tracked tickers.""" +``` + +This interface is unchanged from the shipped implementation, and deliberately stays unaware of +*why* a ticker is tracked. §12 explains why "tracked = watchlist ∪ open positions" is enforced one +layer up, by the callers of `add_ticker` / `remove_ticker`, not inside this module. + +### Why the source writes to the cache instead of returning prices + +This push model decouples timing. The simulator ticks at 500ms, Massive polls at 15s, but SSE +always reads from the cache at its own 500ms cadence — it doesn't need to know which data source is +active or what its update interval is. + +--- + +## 6. Seed Prices & Ticker Parameters + +**File: `backend/app/market/seed_prices.py`** + +Constants only — no logic, no imports beyond stdlib. Adds `PREV_CLOSE`, a baseline a few percent +away from `SEED_PRICES` so the watchlist opens with a realistic mix of gainers and losers (`PLAN.md` +§6) rather than every ticker showing 0.00% on first paint. + +```python +"""Seed prices and per-ticker parameters for the market simulator.""" + +# Realistic starting prices for the default watchlist (as of project creation) +SEED_PRICES: dict[str, float] = { + "AAPL": 190.00, + "GOOGL": 175.00, + "MSFT": 420.00, + "AMZN": 185.00, + "TSLA": 250.00, + "NVDA": 800.00, + "META": 500.00, + "JPM": 195.00, + "V": 280.00, + "NFLX": 600.00, +} + +# Prior session's close for each ticker — the baseline for daily change %. +# Deliberately a few percent off SEED_PRICES so the watchlist opens with a +# realistic spread of gainers and losers instead of a flat 0.00% row. +PREV_CLOSE: dict[str, float] = { + "AAPL": 187.50, # +1.33% today + "GOOGL": 177.20, # -1.24% today + "MSFT": 415.00, # +1.20% today + "AMZN": 188.40, # -1.82% today + "TSLA": 241.30, # +3.60% today + "NVDA": 812.00, # -1.48% today + "META": 493.00, # +1.42% today + "JPM": 193.10, # +0.98% today + "V": 282.50, # -0.88% today + "NFLX": 589.00, # +1.87% today +} + +# Per-ticker GBM parameters +# sigma: annualized volatility (higher = more price movement) +# mu: annualized drift / expected return +TICKER_PARAMS: dict[str, dict[str, float]] = { + "AAPL": {"sigma": 0.22, "mu": 0.05}, + "GOOGL": {"sigma": 0.25, "mu": 0.05}, + "MSFT": {"sigma": 0.20, "mu": 0.05}, + "AMZN": {"sigma": 0.28, "mu": 0.05}, + "TSLA": {"sigma": 0.50, "mu": 0.03}, # High volatility + "NVDA": {"sigma": 0.40, "mu": 0.08}, # High volatility, strong drift + "META": {"sigma": 0.30, "mu": 0.05}, + "JPM": {"sigma": 0.18, "mu": 0.04}, # Low volatility (bank) + "V": {"sigma": 0.17, "mu": 0.04}, # Low volatility (payments) + "NFLX": {"sigma": 0.35, "mu": 0.05}, +} + +# Default parameters for tickers not in the list above (dynamically added) +DEFAULT_PARAMS: dict[str, float] = {"sigma": 0.25, "mu": 0.05} + +# Correlation groups for the simulator's Cholesky decomposition +# Tickers in the same group have higher intra-group correlation +CORRELATION_GROUPS: dict[str, set[str]] = { + "tech": {"AAPL", "GOOGL", "MSFT", "AMZN", "META", "NVDA", "NFLX"}, + "finance": {"JPM", "V"}, +} + +# Correlation coefficients +INTRA_TECH_CORR = 0.6 # Tech stocks move together +INTRA_FINANCE_CORR = 0.5 # Finance stocks move together +CROSS_GROUP_CORR = 0.3 # Between sectors / unknown tickers +TSLA_CORR = 0.3 # TSLA does its own thing +``` + +`PREV_CLOSE` has no entry for a ticker the user adds dynamically — that's intentional. A fallback +ticker's `prev_close` is set equal to its generated seed price (§7), per `PLAN.md` §6: *"A fallback +ticker gets `prev_close` equal to its generated seed price."* This yields exactly a 0.00% daily +change on first paint for a ticker with no real history, which is the honest answer, not a +guess. + +--- + +## 7. GBM Simulator + +**File: `backend/app/market/simulator.py`** + +Two classes: `GBMSimulator` (pure math engine, advances prices one step at a time) and +`SimulatorDataSource` (the `MarketDataSource` implementation wrapping it in an async loop and +writing to the `PriceCache`). + +### 7.1 GBMSimulator — the math engine + +Unchanged from the shipped implementation — `prev_close` is a cache-layer concern (a fixed daily +baseline), not something the per-tick random walk needs to know about. + +```python +from __future__ import annotations + +import asyncio +import logging +import math +import random + +import numpy as np + +from .cache import PriceCache +from .interface import MarketDataSource +from .seed_prices import ( + CORRELATION_GROUPS, + CROSS_GROUP_CORR, + DEFAULT_PARAMS, + INTRA_FINANCE_CORR, + INTRA_TECH_CORR, + PREV_CLOSE, + SEED_PRICES, + TICKER_PARAMS, + TSLA_CORR, +) + +logger = logging.getLogger(__name__) + + +class GBMSimulator: + """Geometric Brownian Motion simulator for correlated stock prices. + + Math: + S(t+dt) = S(t) * exp((mu - sigma^2/2) * dt + sigma * sqrt(dt) * Z) + + Where: + S(t) = current price + mu = annualized drift (expected return) + sigma = annualized volatility + dt = time step as fraction of a trading year + Z = correlated standard normal random variable + + The tiny dt (~8.5e-8 for 500ms ticks over 252 trading days * 6.5h/day) + produces sub-cent moves per tick that accumulate naturally over time. + """ + + TRADING_SECONDS_PER_YEAR = 252 * 6.5 * 3600 # 5,896,800 + DEFAULT_DT = 0.5 / TRADING_SECONDS_PER_YEAR # ~8.48e-8 + + def __init__( + self, + tickers: list[str], + dt: float = DEFAULT_DT, + event_probability: float = 0.001, + ) -> None: + self._dt = dt + self._event_prob = event_probability + + self._tickers: list[str] = [] + self._prices: dict[str, float] = {} + self._prev_close: dict[str, float] = {} + self._params: dict[str, dict[str, float]] = {} + + self._cholesky: np.ndarray | None = None + + for ticker in tickers: + self._add_ticker_internal(ticker) + self._rebuild_cholesky() + + # --- Public API --- + + def step(self) -> dict[str, float]: + """Advance all tickers by one time step. Returns {ticker: new_price}. + + This is the hot path — called every 500ms. Keep it fast. + """ + n = len(self._tickers) + if n == 0: + return {} + + z_independent = np.random.standard_normal(n) + z_correlated = self._cholesky @ z_independent if self._cholesky is not None else z_independent + + result: dict[str, float] = {} + for i, ticker in enumerate(self._tickers): + params = self._params[ticker] + mu = params["mu"] + sigma = params["sigma"] + + drift = (mu - 0.5 * sigma**2) * self._dt + diffusion = sigma * math.sqrt(self._dt) * z_correlated[i] + self._prices[ticker] *= math.exp(drift + diffusion) + + # Random event: ~0.1% chance per tick per ticker + if random.random() < self._event_prob: + shock_magnitude = random.uniform(0.02, 0.05) + shock_sign = random.choice([-1, 1]) + self._prices[ticker] *= 1 + shock_magnitude * shock_sign + logger.debug( + "Random event on %s: %.1f%% %s", + ticker, + shock_magnitude * 100, + "up" if shock_sign > 0 else "down", + ) + + result[ticker] = round(self._prices[ticker], 2) + + return result + + def add_ticker(self, ticker: str) -> None: + """Add a ticker to the simulation. Rebuilds the correlation matrix.""" + if ticker in self._prices: + return + self._add_ticker_internal(ticker) + self._rebuild_cholesky() + + def remove_ticker(self, ticker: str) -> None: + """Remove a ticker from the simulation. Rebuilds the correlation matrix.""" + if ticker not in self._prices: + return + self._tickers.remove(ticker) + del self._prices[ticker] + del self._prev_close[ticker] + del self._params[ticker] + self._rebuild_cholesky() + + def get_price(self, ticker: str) -> float | None: + """Current price for a ticker, or None if not tracked.""" + return self._prices.get(ticker) + + def get_prev_close(self, ticker: str) -> float | None: + """Prior session's close for a ticker, or None if not tracked.""" + return self._prev_close.get(ticker) + + def get_tickers(self) -> list[str]: + """Return the list of currently tracked tickers.""" + return list(self._tickers) + + # --- Internals --- + + def _add_ticker_internal(self, ticker: str) -> None: + """Add a ticker without rebuilding Cholesky (for batch initialization).""" + if ticker in self._prices: + return + self._tickers.append(ticker) + seed = SEED_PRICES.get(ticker) + if seed is None: + # Unknown ticker: fallback seed in $50-$500, and per PLAN.md §6 its + # prev_close equals that same seed (0.00% change on first paint — + # there is no real history to invent a baseline from). + seed = random.uniform(50.0, 500.0) + self._prev_close[ticker] = seed + else: + self._prev_close[ticker] = PREV_CLOSE.get(ticker, seed) + self._prices[ticker] = seed + self._params[ticker] = TICKER_PARAMS.get(ticker, dict(DEFAULT_PARAMS)) + + def _rebuild_cholesky(self) -> None: + """Rebuild the Cholesky decomposition of the ticker correlation matrix. + + Called whenever tickers are added or removed. O(n^2) but n < 50. + """ + n = len(self._tickers) + if n <= 1: + self._cholesky = None + return + + corr = np.eye(n) + for i in range(n): + for j in range(i + 1, n): + rho = self._pairwise_correlation(self._tickers[i], self._tickers[j]) + corr[i, j] = rho + corr[j, i] = rho + + self._cholesky = np.linalg.cholesky(corr) + + @staticmethod + def _pairwise_correlation(t1: str, t2: str) -> float: + """Determine correlation between two tickers based on sector grouping. + + Correlation structure: + - Same tech sector: 0.6 + - Same finance sector: 0.5 + - TSLA with anything: 0.3 (it does its own thing) + - Cross-sector: 0.3 + - Unknown tickers: 0.3 + """ + tech = CORRELATION_GROUPS["tech"] + finance = CORRELATION_GROUPS["finance"] + + if t1 == "TSLA" or t2 == "TSLA": + return TSLA_CORR + if t1 in tech and t2 in tech: + return INTRA_TECH_CORR + if t1 in finance and t2 in finance: + return INTRA_FINANCE_CORR + return CROSS_GROUP_CORR +``` + +**What changed from the shipped `simulator.py`:** the `_prev_close: dict[str, float]` dict, its +population in `_add_ticker_internal` (with the unknown-ticker fallback rule), its cleanup in +`remove_ticker`, and the new `get_prev_close()` accessor. `step()` itself is untouched — a daily +close doesn't move tick to tick. + +### 7.2 SimulatorDataSource — async wrapper + +```python +class SimulatorDataSource(MarketDataSource): + """MarketDataSource backed by the GBM simulator. + + Runs a background asyncio task that calls GBMSimulator.step() every + `update_interval` seconds and writes results to the PriceCache. + """ + + def __init__( + self, + price_cache: PriceCache, + update_interval: float = 0.5, + event_probability: float = 0.001, + ) -> None: + self._cache = price_cache + self._interval = update_interval + self._event_prob = event_probability + self._sim: GBMSimulator | None = None + self._task: asyncio.Task | None = None + + async def start(self, tickers: list[str]) -> None: + self._sim = GBMSimulator(tickers=tickers, event_probability=self._event_prob) + # Seed the cache with initial price + prev_close so SSE has data + # (and a correct daily change %) immediately. + for ticker in tickers: + price = self._sim.get_price(ticker) + prev_close = self._sim.get_prev_close(ticker) + if price is not None: + self._cache.update(ticker=ticker, price=price, prev_close=prev_close) + self._task = asyncio.create_task(self._run_loop(), name="simulator-loop") + logger.info("Simulator started with %d tickers", len(tickers)) + + async def stop(self) -> None: + if self._task and not self._task.done(): + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + self._task = None + logger.info("Simulator stopped") + + async def add_ticker(self, ticker: str) -> None: + if self._sim: + self._sim.add_ticker(ticker) + price = self._sim.get_price(ticker) + prev_close = self._sim.get_prev_close(ticker) + if price is not None: + self._cache.update(ticker=ticker, price=price, prev_close=prev_close) + logger.info("Simulator: added ticker %s", ticker) + + async def remove_ticker(self, ticker: str) -> None: + if self._sim: + self._sim.remove_ticker(ticker) + self._cache.remove(ticker) + logger.info("Simulator: removed ticker %s", ticker) + + def get_tickers(self) -> list[str]: + return self._sim.get_tickers() if self._sim else [] + + async def _run_loop(self) -> None: + """Core loop: step the simulation, write to cache, sleep.""" + while True: + try: + if self._sim: + prices = self._sim.step() + for ticker, price in prices.items(): + # prev_close omitted: it's fixed for the session, so + # PriceCache.update() carries the existing value forward. + self._cache.update(ticker=ticker, price=price) + except Exception: + logger.exception("Simulator step failed") + await asyncio.sleep(self._interval) +``` + +**What changed from the shipped `SimulatorDataSource`:** `start()` and `add_ticker()` now also pull +`get_prev_close()` and pass it into `cache.update()`. `_run_loop()` is unchanged — see the +`PriceCache.update()` fallback in §4 that carries `prev_close` forward on ticks that don't supply +one. + +### Key behaviors + +- **Immediate seeding**: the cache is populated with seed price *and* `prev_close` before the loop + begins, so the SSE endpoint's very first tick already has a correct price and a correct daily + change %. +- **Graceful cancellation**: `stop()` cancels the task and awaits it, catching `CancelledError` — + clean shutdown during FastAPI lifespan teardown. +- **Exception resilience**: the loop catches exceptions per-step so one bad tick doesn't kill the + feed. + +--- + +## 8. Massive API Client + +**File: `backend/app/market/massive_client.py`** + +Polls the Massive (Polygon.io-compatible) REST snapshot endpoint on a configurable interval. The +synchronous client runs in `asyncio.to_thread()` to avoid blocking the event loop. + +```python +from __future__ import annotations + +import asyncio +import logging + +from massive import RESTClient +from massive.rest.models import SnapshotMarketType + +from .cache import PriceCache +from .interface import MarketDataSource + +logger = logging.getLogger(__name__) + + +class MassiveDataSource(MarketDataSource): + """MarketDataSource backed by the Massive (Polygon.io) REST API. + + Polls GET /v2/snapshot/locale/us/markets/stocks/tickers for all watched + tickers in a single API call, then writes results to the PriceCache. + + Rate limits: + - Free tier: 5 req/min → poll every 15s (default) + - Paid tiers: higher limits → poll every 2-15s + """ + + def __init__( + self, + api_key: str, + price_cache: PriceCache, + poll_interval: float = 15.0, + ) -> None: + self._api_key = api_key + self._cache = price_cache + self._interval = poll_interval + self._tickers: list[str] = [] + self._task: asyncio.Task | None = None + self._client: RESTClient | None = None + + async def start(self, tickers: list[str]) -> None: + self._client = RESTClient(api_key=self._api_key) + self._tickers = list(tickers) + + await self._poll_once() # immediate first poll: cache has data right away + + self._task = asyncio.create_task(self._poll_loop(), name="massive-poller") + logger.info( + "Massive poller started: %d tickers, %.1fs interval", len(tickers), self._interval + ) + + async def stop(self) -> None: + if self._task and not self._task.done(): + self._task.cancel() + try: + await self._task + except asyncio.CancelledError: + pass + self._task = None + self._client = None + logger.info("Massive poller stopped") + + async def add_ticker(self, ticker: str) -> None: + ticker = ticker.upper().strip() + if ticker not in self._tickers: + self._tickers.append(ticker) + logger.info("Massive: added ticker %s (will appear on next poll)", ticker) + + async def remove_ticker(self, ticker: str) -> None: + ticker = ticker.upper().strip() + self._tickers = [t for t in self._tickers if t != ticker] + self._cache.remove(ticker) + logger.info("Massive: removed ticker %s", ticker) + + def get_tickers(self) -> list[str]: + return list(self._tickers) + + # --- Internal --- + + async def _poll_loop(self) -> None: + """Poll on interval. First poll already happened in start().""" + while True: + await asyncio.sleep(self._interval) + await self._poll_once() + + async def _poll_once(self) -> None: + """Execute one poll cycle: fetch snapshots, update cache.""" + if not self._tickers or not self._client: + return + + try: + snapshots = await asyncio.to_thread(self._fetch_snapshots) + processed = 0 + for snap in snapshots: + try: + price = snap.last_trade.price + timestamp = snap.last_trade.timestamp / 1000.0 # ms → s + + # Previous session's close, for the daily change %. + # `prev_day` is the Polygon/Massive snapshot's OHLC block + # for the prior trading session; if it's ever absent (a + # brand-new listing, an odd API response) fall back to the + # tick price itself so change_percent_from_close reads + # 0.00% instead of crashing the whole poll. + prev_close = getattr(getattr(snap, "prev_day", None), "close", None) + if prev_close is None: + prev_close = price + + self._cache.update( + ticker=snap.ticker, + price=price, + prev_close=prev_close, + timestamp=timestamp, + ) + processed += 1 + except (AttributeError, TypeError) as e: + logger.warning( + "Skipping snapshot for %s: %s", getattr(snap, "ticker", "???"), e + ) + logger.debug("Massive poll: updated %d/%d tickers", processed, len(self._tickers)) + + except Exception as e: + logger.error("Massive poll failed: %s", e) + # Don't re-raise — the loop retries on the next interval. + # Common failures: 401 (bad key), 429 (rate limit), network errors. + + def _fetch_snapshots(self) -> list: + """Synchronous call to the Massive REST API. Runs in a thread.""" + return self._client.get_snapshot_all( + market_type=SnapshotMarketType.STOCKS, + tickers=self._tickers, + ) +``` + +**What changed from the shipped `massive_client.py`:** `_poll_once()` now reads `snap.prev_day.close` +(guarded with `getattr` in case a response is missing that block) and passes it as `prev_close` into +`cache.update()`. Everything else — the lazy-vs-eager import question was already resolved in the +shipped code by making `massive` a core dependency (see §15.4) — is unchanged. + +> **Note on the field name.** `massive` (the Polygon.io-compatible client this project depends on, +> pinned `>=1.0.0`, `2.2.0` resolved in `uv.lock`) was not installed in the environment this document +> was written in, so `snap.prev_day.close` is inferred from Polygon.io's public snapshot schema +> (`TickerSnapshot.prev_day` is the prior session's OHLC bar) rather than confirmed against the +> installed package. Confirm the exact attribute name against `massive`'s actual `TickerSnapshot` +> model before merging — the `getattr(..., None)` guard means a wrong name degrades to "0.00% daily +> change," not a crash, but it should still be corrected. + +### Error handling philosophy + +| Error | Behavior | +|-------|----------| +| **401 Unauthorized** | Logged as error. Poller keeps running (user might fix `.env` and restart). | +| **429 Rate Limited** | Logged as error. Next poll retries after `poll_interval` seconds. | +| **Network timeout** | Logged as error. Retries automatically on next cycle. | +| **Malformed snapshot** | Individual ticker skipped with a warning; others still processed. | +| **Missing `prev_day`** | `prev_close` falls back to the tick price (0.00% daily change), not a crash. | +| **All tickers fail** | Cache retains last-known prices. SSE keeps streaming stale data (better than none). | + +--- + +## 9. Factory + +**File: `backend/app/market/factory.py`** + +Unchanged from the shipped implementation. + +```python +from __future__ import annotations + +import logging +import os + +from .cache import PriceCache +from .interface import MarketDataSource +from .massive_client import MassiveDataSource +from .simulator import SimulatorDataSource + +logger = logging.getLogger(__name__) + + +def create_market_data_source(price_cache: PriceCache) -> MarketDataSource: + """Create the appropriate market data source based on environment variables. + + - MASSIVE_API_KEY set and non-empty → MassiveDataSource (real market data) + - Otherwise → SimulatorDataSource (GBM simulation) + + Returns an unstarted source. Caller must await source.start(tickers). + """ + api_key = os.environ.get("MASSIVE_API_KEY", "").strip() + + if api_key: + logger.info("Market data source: Massive API (real data)") + return MassiveDataSource(api_key=api_key, price_cache=price_cache) + else: + logger.info("Market data source: GBM Simulator") + return SimulatorDataSource(price_cache=price_cache) +``` + +### Usage at app startup + +```python +price_cache = PriceCache() +source = create_market_data_source(price_cache) +await source.start(initial_tickers) # e.g. the watchlist loaded from SQLite +``` + +--- + +## 10. SSE Streaming Endpoint + +**File: `backend/app/market/stream.py`** + +Unchanged from the shipped implementation — it already serializes whatever `PriceUpdate.to_dict()` +returns, so adding `prev_close` / `change_from_close` / `change_percent_from_close` to the model in +§3 flows through automatically with no edits here. + +```python +from __future__ import annotations + +import asyncio +import json +import logging +from collections.abc import AsyncGenerator + +from fastapi import APIRouter, Request +from fastapi.responses import StreamingResponse + +from .cache import PriceCache + +logger = logging.getLogger(__name__) + +router = APIRouter(prefix="/api/stream", tags=["streaming"]) + + +def create_stream_router(price_cache: PriceCache) -> APIRouter: + """Create the SSE streaming router with a reference to the price cache. + + This factory pattern lets us inject the PriceCache without globals. + """ + + @router.get("/prices") + async def stream_prices(request: Request) -> StreamingResponse: + """SSE endpoint for live price updates. + + Streams all tracked ticker prices every ~500ms. The client connects + with EventSource and receives events in the format: + + data: {"AAPL": {"ticker": "AAPL", "price": 190.50, ...}, ...} + + Includes a retry directive so the browser auto-reconnects on + disconnection (EventSource built-in behavior). + """ + return StreamingResponse( + _generate_events(price_cache, request), + media_type="text/event-stream", + headers={ + "Cache-Control": "no-cache", + "Connection": "keep-alive", + "X-Accel-Buffering": "no", # Disable nginx buffering if proxied + }, + ) + + return router + + +async def _generate_events( + price_cache: PriceCache, + request: Request, + interval: float = 0.5, +) -> AsyncGenerator[str, None]: + """Async generator that yields SSE-formatted price events. + + Sends all prices every `interval` seconds. Stops when the client + disconnects (detected via request.is_disconnected()). + """ + yield "retry: 1000\n\n" + + last_version = -1 + client_ip = request.client.host if request.client else "unknown" + logger.info("SSE client connected: %s", client_ip) + + try: + while True: + if await request.is_disconnected(): + logger.info("SSE client disconnected: %s", client_ip) + break + + current_version = price_cache.version + if current_version != last_version: + last_version = current_version + prices = price_cache.get_all() + + if prices: + data = {ticker: update.to_dict() for ticker, update in prices.items()} + payload = json.dumps(data) + yield f"data: {payload}\n\n" + + await asyncio.sleep(interval) + except asyncio.CancelledError: + logger.info("SSE stream cancelled for: %s", client_ip) +``` + +### SSE wire format + +``` +data: {"AAPL":{"ticker":"AAPL","price":190.50,"previous_price":190.42,"prev_close":187.50,"timestamp":1707580800.5,"change":0.08,"change_percent":0.042,"direction":"up","change_from_close":3.00,"change_percent_from_close":1.6}, "GOOGL":{...}} + +``` + +Client-side: + +```javascript +const eventSource = new EventSource('/api/stream/prices'); +eventSource.onmessage = (event) => { + const prices = JSON.parse(event.data); + // prices["AAPL"].change_percent_from_close → watchlist "Change %" column + // prices["AAPL"].direction → green/red price flash +}; +``` + +### Why poll-and-push instead of event-driven? + +The SSE endpoint polls the cache on a fixed interval rather than being notified by the data source. +Simpler, and it produces regularly-spaced updates, which matters because the frontend accumulates +them into sparklines and the main chart client-side (`PLAN.md` §10) — irregular spacing there would +look wrong. + +--- + +## 11. FastAPI Lifecycle Integration + +The market data system starts and stops with the FastAPI app via the `lifespan` context manager. + +**In `backend/app/main.py`** (not yet built — this is the shape it needs): + +```python +from contextlib import asynccontextmanager + +from fastapi import FastAPI + +from app.market import PriceCache, create_market_data_source, create_stream_router +from app.market.interface import MarketDataSource + + +@asynccontextmanager +async def lifespan(app: FastAPI): + """Manage startup and shutdown of background services.""" + + # --- STARTUP --- + + price_cache = PriceCache() + app.state.price_cache = price_cache + + source = create_market_data_source(price_cache) + app.state.market_source = source + + # Tracked set = watchlist ∪ open positions (PLAN.md §6) — computed once + # here from the database, which the (not-yet-built) watchlist/portfolio + # modules own. See §12 for how it stays correct after startup. + initial_tickers = await load_tracked_tickers() # reads watchlist + positions from SQLite + await source.start(initial_tickers) + + app.include_router(create_stream_router(price_cache)) + + yield # App is running + + # --- SHUTDOWN --- + await source.stop() + + +app = FastAPI(title="FinAlly", lifespan=lifespan) + + +def get_price_cache() -> PriceCache: + return app.state.price_cache + + +def get_market_source() -> MarketDataSource: + return app.state.market_source +``` + +### Accessing market data from other routes + +```python +from fastapi import APIRouter, Depends, HTTPException + +router = APIRouter(prefix="/api") + + +@router.post("/portfolio/trade") +async def execute_trade( + trade: TradeRequest, + price_cache: PriceCache = Depends(get_price_cache), +): + current_price = price_cache.get_price(trade.ticker) + if current_price is None: + raise HTTPException(400, f"No price available for {trade.ticker}") + # ... validate against cash/position and execute at current_price (PLAN.md §8) ... + + +@router.post("/watchlist") +async def add_to_watchlist( + payload: WatchlistAdd, + source: MarketDataSource = Depends(get_market_source), +): + # ... insert into the watchlist table ... + await source.add_ticker(payload.ticker) # a DB write alone does not start pricing it + # ... + + +@router.delete("/watchlist/{ticker}") +async def remove_from_watchlist( + ticker: str, + source: MarketDataSource = Depends(get_market_source), +): + # ... delete from the watchlist table ... + # Only stop tracking if there's no open position — see §12. + # ... +``` + +--- + +## 12. Watchlist / Tracked-Ticker-Set Coordination + +`PLAN.md` §6 defines the tracked set as **the watchlist plus every ticker with a non-zero +position** — not the watchlist alone, because a user can de-watchlist a ticker they still hold, and +the portfolio can't be valued without a live price for it. This logic belongs to the watchlist and +portfolio routes (not yet built), not to `app/market/` — the market module only ever does what it's +told via `add_ticker` / `remove_ticker`. This section documents the contract those routes must +satisfy. + +### Flow: adding a ticker (manual or via LLM chat) + +``` +POST /api/watchlist {ticker: "PYPL"} + → INSERT INTO watchlist (SQLite) + → await source.add_ticker("PYPL") + Simulator: adds to GBMSimulator (fallback seed + prev_close, §7), rebuilds + Cholesky, seeds cache immediately + Massive: appends to the poll ticker list, appears on the next poll (≤15s) + → 200 {ticker, price if already cached} +``` + +A database write alone does **not** start pricing the symbol — `source.add_ticker()` must be +called in the same request, or the ticker sits in the watchlist table with no price and every +SSE frame and trade attempt for it fails. + +### Flow: removing a ticker + +``` +DELETE /api/watchlist/PYPL + → DELETE FROM watchlist (SQLite) + → position = await db.get_position("PYPL") + → if position is None: + await source.remove_ticker("PYPL") # stops pricing, drops from cache + else: + # ticker stays tracked and priced; it just stops appearing in the + # watchlist panel. Portfolio valuation for the open position still + # needs a live price. + → 200 {} +``` + +```python +@router.delete("/watchlist/{ticker}") +async def remove_from_watchlist( + ticker: str, + source: MarketDataSource = Depends(get_market_source), +): + await db.delete_watchlist_entry(ticker) + + position = await db.get_position(ticker) + if position is None: + await source.remove_ticker(ticker) + + return {"status": "ok"} +``` + +### Flow: opening a position in a ticker that isn't on the watchlist + +Buying a ticker not currently tracked (§8's trade validation requires a cached price first, so in +practice the trade route or the chat tool-call handler must call `add_ticker` before or as part of +executing the buy): + +```python +@router.post("/portfolio/trade") +async def execute_trade( + trade: TradeRequest, + price_cache: PriceCache = Depends(get_price_cache), + source: MarketDataSource = Depends(get_market_source), +): + if trade.ticker not in price_cache: + await source.add_ticker(trade.ticker) + # Simulator seeds synchronously — price is available immediately after. + # Massive does not; the trade route should recheck the cache and + # reject with a clear "price not yet available, try again shortly" + # rather than blocking on a poll (see §14.2). + current_price = price_cache.get_price(trade.ticker) + if current_price is None: + raise HTTPException(400, f"Price not yet available for {trade.ticker}") + # ... proceed with trade validation and execution ... +``` + +The ticker then joins the tracked set with no separate watchlist entry required. + +--- + +## 13. Testing Strategy + +The shipped suite (73 tests, 84% coverage — see `MARKET_DATA_SUMMARY.md`) already covers +`models.py`, `cache.py`, `interface.py`, `seed_prices.py`, `simulator.py`, `factory.py`, and +`massive_client.py` (mocked) at or near 100%. Extending it for `prev_close` means adding cases to +the existing files rather than new ones. + +### 13.1 `test_models.py` — additions + +```python +def test_change_from_close_positive(): + update = PriceUpdate(ticker="AAPL", price=192.00, previous_price=191.50, prev_close=190.00) + assert update.change_from_close == 2.00 + assert round(update.change_percent_from_close, 2) == 1.05 + +def test_change_from_close_zero_prev_close_is_safe(): + update = PriceUpdate(ticker="ZZZZ", price=100.0, previous_price=100.0, prev_close=0.0) + assert update.change_percent_from_close == 0.0 # no division by zero + +def test_to_dict_includes_close_baseline_fields(): + update = PriceUpdate(ticker="AAPL", price=192.00, previous_price=191.50, prev_close=190.00) + d = update.to_dict() + assert d["prev_close"] == 190.00 + assert "change_from_close" in d + assert "change_percent_from_close" in d +``` + +### 13.2 `test_cache.py` — additions + +```python +def test_update_requires_prev_close_on_first_write(): + cache = PriceCache() + update = cache.update("AAPL", 190.00, prev_close=187.50) + assert update.prev_close == 187.50 + +def test_prev_close_carries_forward_when_omitted(): + cache = PriceCache() + cache.update("AAPL", 190.00, prev_close=187.50) + update = cache.update("AAPL", 191.00) # simulator tick, no prev_close passed + assert update.prev_close == 187.50 + +def test_missing_prev_close_on_first_write_falls_back_to_price(): + cache = PriceCache() + update = cache.update("ZZZZ", 120.00) # no prev_close supplied at all + assert update.prev_close == 120.00 +``` + +### 13.3 `test_simulator.py` — additions + +```python +def test_known_ticker_prev_close_from_table(): + sim = GBMSimulator(tickers=["AAPL"]) + assert sim.get_prev_close("AAPL") == PREV_CLOSE["AAPL"] + +def test_unknown_ticker_prev_close_equals_seed(): + sim = GBMSimulator(tickers=["ZZZZ"]) + assert sim.get_prev_close("ZZZZ") == sim.get_price("ZZZZ") + +def test_prev_close_removed_with_ticker(): + sim = GBMSimulator(tickers=["AAPL"]) + sim.remove_ticker("AAPL") + assert sim.get_prev_close("AAPL") is None +``` + +### 13.4 `test_simulator_source.py` — additions + +```python +async def test_start_seeds_cache_with_prev_close(): + cache = PriceCache() + source = SimulatorDataSource(price_cache=cache, update_interval=0.1) + await source.start(["AAPL"]) + assert cache.get("AAPL").prev_close == PREV_CLOSE["AAPL"] + await source.stop() + +async def test_prev_close_stable_across_ticks(): + cache = PriceCache() + source = SimulatorDataSource(price_cache=cache, update_interval=0.05) + await source.start(["AAPL"]) + initial_prev_close = cache.get("AAPL").prev_close + await asyncio.sleep(0.3) + assert cache.get("AAPL").prev_close == initial_prev_close # never drifts + await source.stop() +``` + +### 13.5 `test_massive.py` — additions + +```python +def _make_snapshot(ticker, price, timestamp_ms, prev_close=None): + snap = MagicMock() + snap.ticker = ticker + snap.last_trade.price = price + snap.last_trade.timestamp = timestamp_ms + if prev_close is not None: + snap.prev_day.close = prev_close + else: + snap.prev_day = None + return snap + +async def test_poll_captures_prev_close(): + cache = PriceCache() + source = MassiveDataSource(api_key="test-key", price_cache=cache, poll_interval=60.0) + snap = _make_snapshot("AAPL", 190.50, 1707580800000, prev_close=187.50) + with patch.object(source, "_fetch_snapshots", return_value=[snap]): + await source._poll_once() + assert cache.get("AAPL").prev_close == 187.50 + +async def test_missing_prev_day_falls_back_to_price(): + cache = PriceCache() + source = MassiveDataSource(api_key="test-key", price_cache=cache, poll_interval=60.0) + snap = _make_snapshot("AAPL", 190.50, 1707580800000, prev_close=None) + with patch.object(source, "_fetch_snapshots", return_value=[snap]): + await source._poll_once() + assert cache.get("AAPL").prev_close == 190.50 +``` + +### 13.6 What stays out of scope here + +Per `PLAN.md` §12, watchlist idempotency, the 30-ticker cap, and "removing a watchlist ticker with +an open position keeps it priced" are backend API-route tests (against the not-yet-built watchlist +endpoints), not market-module tests — `app/market/` only needs to prove `add_ticker`/`remove_ticker` +behave correctly in isolation, which the existing suite already does. + +--- + +## 14. Error Handling & Edge Cases + +### 14.1 Startup: empty watchlist + +`start([])` — both sources handle it gracefully: the simulator produces no prices, the Massive +poller skips its API call, and the SSE endpoint sends nothing until a ticker is added. + +### 14.2 Price cache miss during a trade + +```python +price = price_cache.get_price(ticker) +if price is None: + raise HTTPException( + status_code=400, + detail=f"Price not yet available for {ticker}. Please wait a moment and try again.", + ) +``` + +The simulator avoids this in practice by seeding synchronously in `add_ticker()`. Massive has an +inherent gap (up to `poll_interval` seconds) between adding a ticker and its first price — the 400 +with a clear message is the correct response, not a blocking wait. + +### 14.3 Massive API key invalid + +First poll fails with 401; the poller logs and keeps retrying. SSE keeps streaming (connected, just +empty for the affected tickers). Fix is to correct `MASSIVE_API_KEY` and restart. + +### 14.4 Massive snapshot missing `prev_day` + +Handled by the `getattr(..., None)` fallback in §8 — `prev_close` degrades to the tick price +(0.00% daily change shown) rather than raising and dropping the whole ticker from that poll cycle. + +### 14.5 Thread safety under load + +`PriceCache` uses a plain mutex; the critical section is a dict lookup and assignment. Negligible +contention at the project's scale (≤30 tickers, one writer). A `ReadWriteLock` would only matter at +a scale this project doesn't target. + +### 14.6 Simulator precision + +Prices are `round()`ed to 2 decimals in `GBMSimulator.step()`; the exponential formulation is +numerically stable and always positive, so GBM can never produce a negative or zero price. + +--- + +## 15. Delta Against the Current Implementation + +Everything in §3–§10 that isn't called out below is byte-for-byte what's already on disk in +`backend/app/market/`. To bring the shipped code up to this design: + +| # | File | Change | +|---|------|--------| +| 1 | `models.py` | Add `prev_close: float` field to `PriceUpdate`; add `change_from_close` / `change_percent_from_close` properties; include both in `to_dict()`. | +| 2 | `cache.py` | Add `prev_close: float \| None = None` parameter to `update()`; carry the existing value forward when omitted, or fall back to `price` on a ticker's first write with none supplied. | +| 3 | `seed_prices.py` | Add the `PREV_CLOSE: dict[str, float]` table (§6). | +| 4 | `simulator.py` (`GBMSimulator`) | Add `_prev_close` dict; populate it in `_add_ticker_internal` (known ticker → `PREV_CLOSE` table; unknown → equal to the generated seed); delete the entry in `remove_ticker`; add `get_prev_close()`. | +| 5 | `simulator.py` (`SimulatorDataSource`) | `start()` and `add_ticker()` pass `prev_close=self._sim.get_prev_close(ticker)` into `cache.update()`. | +| 6 | `massive_client.py` | `_poll_once()` reads `snap.prev_day.close` (guarded, falling back to the tick price) and passes it as `prev_close`. **Verify the exact attribute name against the installed `massive` package** — see the note in §8. | +| 7 | `tests/market/*` | Add the cases in §13. | + +`interface.py`, `factory.py`, and `stream.py` need **no changes** — the interface doesn't mention +prices at all, the factory only picks a class, and the SSE endpoint just serializes whatever +`to_dict()` returns. + +Also worth folding in from the existing review (`planning/archive/MARKET_DATA_REVIEW.md`), since +they're touching some of the same files: + +- §3.5 there: `SimulatorDataSource.get_tickers()` in the archived design reached into + `self._sim._tickers` (private). The current shipped code already fixed this with a public + `GBMSimulator.get_tickers()` (confirmed in `simulator.py:140-142` on disk) — §7.1 above reflects + the fixed version; no further action needed. +- §3.4 there (the `version` property not being read under the lock) is fixed in §4 above — it now + acquires `self._lock`. + +--- + +## 16. Configuration Summary + +| Parameter | Location | Default | Description | +|-----------|----------|---------|-------------| +| `MASSIVE_API_KEY` | Environment variable | `""` (empty) | If set, use Massive API; otherwise use the simulator. | +| `update_interval` | `SimulatorDataSource.__init__` | `0.5` (seconds) | Time between simulator ticks. | +| `poll_interval` | `MassiveDataSource.__init__` | `15.0` (seconds) | Time between Massive API polls (free tier: 5 req/min). | +| `event_probability` | `GBMSimulator.__init__` | `0.001` | Chance of a random shock event per ticker per tick. | +| `dt` | `GBMSimulator.__init__` | `~8.5e-8` | GBM time step (fraction of a trading year). | +| SSE push interval | `_generate_events()` | `0.5` (seconds) | Time between SSE pushes to the client. | +| SSE retry directive | `_generate_events()` | `1000` (ms) | Browser `EventSource` reconnection delay. | +| `PREV_CLOSE` spread | `seed_prices.py` | ±1-4% of seed | Static per-ticker baseline for the daily change % on first paint. | + +### Package `__init__.py` + +Unchanged from the shipped implementation. + +```python +"""Market data subsystem for FinAlly. + +Public API: + PriceUpdate - Immutable price snapshot dataclass + PriceCache - Thread-safe in-memory price store + MarketDataSource - Abstract interface for data providers + create_market_data_source - Factory that selects simulator or Massive + create_stream_router - FastAPI router factory for SSE endpoint +""" + +from .cache import PriceCache +from .factory import create_market_data_source +from .interface import MarketDataSource +from .models import PriceUpdate +from .stream import create_stream_router + +__all__ = [ + "PriceUpdate", + "PriceCache", + "MarketDataSource", + "create_market_data_source", + "create_stream_router", +] +```