docs: add comprehensive market data design reference - #2
Conversation
Writes planning/MARKET_DATA_DESIGN.md as an implementation-ready reference covering the unified MarketDataSource interface, PriceCache, GBM simulator, Massive API client, SSE streaming, and FastAPI lifecycle wiring. Code snippets are verified against the current backend/app/market/ implementation, correcting drift present in the older planning/archive/MARKET_DATA_DESIGN.md (lazy-import framing for the massive client, GBMSimulator.get_tickers() visibility, stream.py's generator return type). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RyKr4TS8gMZPNU8roFiYUT
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| router = APIRouter(prefix="/api/stream", tags=["streaming"]) | ||
|
|
||
|
|
||
| def create_stream_router(price_cache: PriceCache) -> APIRouter: | ||
| """Create the SSE streaming router with a reference to the price cache. | ||
|
|
||
| This factory pattern lets us inject the PriceCache without globals. |
There was a problem hiding this comment.
router is declared at module scope here, but the docstring on line 1093 claims the factory pattern "lets us inject the PriceCache without globals" — that's the opposite of what the code does. create_stream_router() decorates and returns this same shared global router rather than constructing a new APIRouter() internally.
Since the only call site is inside lifespan (line 1227), calling it more than once (e.g. multiple app/test instances in the same process) would register a second /prices route on the same global router, and Starlette would only ever match the first-registered one — silently serving from a stale/wrong PriceCache.
Fix: construct APIRouter(...) inside create_stream_router() instead of at module level, so the docstring's claim is actually true.
| @router.post("/portfolio/trade") | ||
| async def execute_trade(trade: TradeRequest, price_cache: PriceCache = Depends(get_price_cache)): | ||
| current_price = price_cache.get_price(trade.ticker) | ||
| if current_price is None: | ||
| raise HTTPException(404, f"No price available for {trade.ticker}") |
There was a problem hiding this comment.
This raises HTTPException(404, ...) when there's no cached price for the trade ticker, but §13.2 describes the exact same condition (cache miss during a trade) and explicitly argues 400 is "the correct response for that window," using the identical price_cache.get_price(ticker) is None check. An implementer following this section would ship a different status code than the one §13.2 specifies as correct — these two need to agree on one status code.
| async def add_ticker(self, ticker: str) -> None: | ||
| ticker = ticker.upper().strip() | ||
| if ticker not in self._tickers: | ||
| self._tickers.append(ticker) | ||
| logger.info("Massive: added ticker %s (will appear on next poll)", ticker) | ||
|
|
||
| async def remove_ticker(self, ticker: str) -> None: | ||
| ticker = ticker.upper().strip() |
There was a problem hiding this comment.
MassiveDataSource.add_ticker/remove_ticker normalize the ticker via .upper().strip(), but the simulator's equivalent methods (lines 734, 743) pass the ticker straight through without normalization. This contradicts the doc's own claim that downstream code "never knows or cares" which source is active — a lowercase ticker like "tsla" would work correctly against Massive but would miss SEED_PRICES["TSLA"] under the simulator, creating a distinct phantom ticker with a random seed price. Since no upstream layer (route handlers) normalizes either, this asymmetry should be resolved — ideally by normalizing once at the interface boundary rather than duplicating it into the simulator.
|
|
||
| @pytest.mark.asyncio | ||
| async def test_sse_stream_emits_prices(): | ||
| transport = httpx.ASGITransport(app=app) | ||
| async with httpx.AsyncClient(transport=transport, base_url="http://test") as client: | ||
| async with client.stream("GET", "/api/stream/prices") as response: | ||
| assert response.status_code == 200 |
There was a problem hiding this comment.
This recommended test uses httpx.ASGITransport without running the ASGI lifespan protocol, but the SSE route is only registered inside lifespan (line 1227), not at module scope. ASGITransport doesn't run lifespan by default, so /api/stream/prices would never be registered and this request would return 404, failing the status_code == 200 assertion on line 1446 — the opposite of what this section is recommending it for (closing the stream.py coverage gap). It needs asgi_lifespan.LifespanManager (or an equivalent lifespan-aware test client) to work as intended.
| |--------|-------|-----------------| | ||
| | `test_models.py` | 11 | `PriceUpdate` properties (`change`, `direction`, `change_percent`), `to_dict()` | | ||
| | `test_cache.py` | 13 | `PriceCache` update/get/get_all/remove, version increments, first-update-is-flat | | ||
| | `test_simulator.py` | 17 | `GBMSimulator` math: positive prices, drift over many steps, add/remove ticker rebuilds Cholesky, unknown-ticker random seed | |
There was a problem hiding this comment.
This row says test_simulator.py has 17 tests, but the total stated two lines above is "73 tests" — the table as written sums to 71 (11+13+17+10+7+13), not 73. The actual count in backend/tests/market/test_simulator.py is 19, which makes the table sum to 73 and matches the stated total — so this row's count is the one that's wrong.
| | `test_simulator.py` | 17 | `GBMSimulator` math: positive prices, drift over many steps, add/remove ticker rebuilds Cholesky, unknown-ticker random seed | | |
| | `test_simulator.py` | 19 | `GBMSimulator` math: positive prices, drift over many steps, add/remove ticker rebuilds Cholesky, unknown-ticker random seed | |
| Rate limits: **free tier → 5 req/min → poll every 15s** (default); | ||
| **paid tiers → poll every 2–5s**. See `planning/MASSIVE_API.md` for the full |
There was a problem hiding this comment.
This states paid tiers poll "every 2–5s" (repeated at line 872), but planning/PLAN.md explicitly states "Paid tiers: poll every 2-15 seconds depending on tier." The design doc narrows this to a 2-5s upper bound in both places, understating PLAN.md's documented max interval.
2 participants
Summary
Add a detailed, implementation-ready design document for the market data subsystem that serves as both an onboarding guide and a specification reference. This document captures the current state of the fully-built and tested market data layer, superseding an earlier archived design that had drifted from the actual implementation.
Changes
planning/MARKET_DATA_DESIGN.md(1,557 lines)PriceUpdatedata model, thread-safePriceCache, abstractMarketDataSourceinterface, GBM simulator with correlated price movements, Massive API client, factory pattern for source selection, SSE streaming endpoint, and FastAPI lifecycle integrationbackend/app/market/MASSIVE_API_KEYis set)Notable Details
This document is the implementation-ready reference that a fresh implementation could be rebuilt from, and doubles as the primary onboarding material for the market data subsystem.
https://claude.ai/code/session_01RyKr4TS8gMZPNU8roFiYUT