diff --git a/README.md b/README.md index 3be6857..c12e74b 100644 --- a/README.md +++ b/README.md @@ -143,38 +143,31 @@ valid token can run this repeatedly (a real Bedrock call and Discord post each t [docs/07-budget-protection.md](docs/07-budget-protection.md) before relying on this in a deploy you leave running unattended. -## ⚠️ Concurrency Limitations & Scaling Up - -The **SQLite-rehydrated-by-S3** pattern operates under a strict **Single-Writer / Low-Concurrency** constraint. Because Amazon S3 does not support partial file locking, standard POSIX filesystem locks (`WAL` mode, `IMMEDIATE` transactions) are completely blind to concurrent AWS Lambda execution containers. - -### The Split-Brain Risk -If two Lambda functions invoke concurrently and attempt to mutate state: -1. **Lambda A** and **Lambda B** both download the same original database file from S3. -2. Both modify their local copy in `/tmp`. -3. Whichever Lambda finishes last will execute `PutObject` and overwrite the other's changes completely. This results in **silent data loss** (lost updates) and state divergence. - ---- - -### How to Scale Beyond Single-Writer Concurrency - -If your agent outgrows a single-writer schedule and requires concurrent read/write access, choose one of the following paths depending on your infrastructure preferences: - -#### 1. EFS Mount: The Zero-Server Alternative (Single-Writer Only With Care) -If you want to keep using SQLite without managing a traditional database server, attach an **Amazon EFS (Elastic File System)** to your Lambda function. -* **How it works:** AWS mounts an EFS network drive directly to `/mnt/storage` inside your Lambda container. -* **The Benefit:** A single Lambda can `hydrate-from-EFS` instead of S3, eliminating the per-invocation S3 download. -* **Trade-off:** Requires moving your Lambda function into a VPC, which introduces minimal network configuration overhead. **EFS is not a true multi-writer substitute for a relational database.** EFS exposes NFSv4 advisory locking only; SQLite's `WAL` mode requires POSIX shared memory that no network filesystem provides, and concurrent writers across multiple Lambda hosts risk 'database is locked' errors and (in failure cases) corruption. The hydrating-Lambda pattern (one writer at a time, S3 as the truth) is the only SQLite-on-Lambda shape the tutorial guarantees. If you genuinely need concurrent writers, skip EFS and pick a client/server database. - -#### 2. Litestream / Litefs: The Replication Stream -[Litestream](https://litestream.io) runs a background sidecar process alongside SQLite that continuously streams WAL (Write-Ahead Log) frames to an S3 bucket every second. -* **How it works:** Instead of pulling/pushing a giant database file, it replicates granular changes. -* **The Benefit:** Drastically reduces S3 network I/O, protects against data loss down to the second, and scales read concurrency beautifully. -* **Trade-off:** Best suited for long-running containers (ECS Fargate) rather than short-lived, ephemeral Lambda functions. - -#### 3. Shift to an Architectural Serverless DB -When cross-agent transactional consistency becomes a core app requirement, migrate the SQLite relational schema and vector lookups into dedicated cloud-native databases: -* **Relational Data:** Migrate to **Amazon Aurora Serverless v2 (PostgreSQL/MySQL)** or **DynamoDB**. -* **Vector Engine:** If using `sqlite-vec` for RAG, migrate those embeddings into **Amazon OpenSearch Serverless**, **pgvector** (on Aurora), or **Pinecone**. +## ⚠️ Concurrency + +S3 has no partial file locking, so SQLite's own locking (`WAL` mode, `IMMEDIATE` +transactions) is blind to a second Lambda container holding its own copy in `/tmp`. What +keeps this safe is the conditional write: every publish carries one. When the snapshot +already exists, the put carries `If-Match: `; on the very +first write — no object yet — it carries `If-None-Match: "*"` instead, a conditional +create that fails if the key already exists. Either way, a writer whose base version has +moved gets a `412` instead of silently clobbering the winner. (Two related failures — +`404 NoSuchKey` and `409 Conditional Request Conflict` — are translated to the same +abort condition by `src/store/s3.ts`; the previous snapshot stays authoritative in all +three cases.) That is optimistic concurrency control applied to a whole database file — +the same pattern behind +[S3 conditional writes](https://simonwillison.net/2024/Nov/26/s3-conditional-writes/) and +[distributed SQLite on S3](https://dev.to/chris_king_bcff3b9663e84a/why-i-built-a-distributed-sqlite-on-s3-and-why-you-might-care-3h9h). + +This tutorial treats a conditional-write failure as an abort rather than rebasing and +retrying: the tick's database work is discarded, but the Bedrock call and Discord post +it already made are not. On the fixed schedule with `reservedConcurrentExecutions: 1` +that never fires — it becomes reachable as soon as a second write path (a manual +trigger, say) can race the loop. + +[docs/10-concurrency.md](docs/10-concurrency.md) covers the full topology, how to add +rebase-and-retry, the SQS single-writer queue for high contention, and why EFS is not the +multi-writer escape hatch it looks like. ## What's here @@ -189,6 +182,7 @@ When cross-agent transactional consistency becomes a core app requirement, migra | [docs/07-budget-protection.md](docs/07-budget-protection.md) | Setting up an AWS Budget alert, and what could actually drive cost up | | [docs/08-rag-vector-search.md](docs/08-rag-vector-search.md) | SQLite as a vector database too: sqlite-vec + Titan embeddings | | [docs/09-lesson-script.md](docs/09-lesson-script.md) | A 10-lesson script for teaching the RAG extension (frame, check-in questions, expected reasoning) | +| [docs/10-concurrency.md](docs/10-concurrency.md) | Optimistic S3 rehydration, 412 handling, rebase-and-retry, the single-writer queue | | [docs/bedrock-model-comparison.md](docs/bedrock-model-comparison.md) | Why `zai.glm-4.7-flash` is the default, and alternatives | ## Cost diff --git a/docs/10-concurrency.md b/docs/10-concurrency.md new file mode 100644 index 0000000..542be5f --- /dev/null +++ b/docs/10-concurrency.md @@ -0,0 +1,157 @@ +# Concurrency: optimistic S3 rehydration + +SQLite-rehydrated-from-S3 is not a database server, and it does not pretend to be one. +S3 has no partial file locking, so SQLite's own concurrency machinery — `WAL` mode, +`BEGIN IMMEDIATE`, POSIX advisory locks — is entirely blind to a second Lambda execution +container holding its own copy of the same file in `/tmp`. Whatever safety this pattern +has comes from one place: the conditional write back to S3. + +That's enough to be safe, and it is a well-worn pattern rather than a local invention. +It is *optimistic concurrency control* (OCC) — the same compare-and-swap discipline +relational databases use for row versions — applied to a whole database file, and it +became practical for S3 the day AWS shipped conditional writes. See Simon Willison's +[write-up of S3 conditional writes](https://simonwillison.net/2024/Nov/26/s3-conditional-writes/) +and Chris King's +["Why I Built a Distributed SQLite on S3"](https://dev.to/chris_king_bcff3b9663e84a/why-i-built-a-distributed-sqlite-on-s3-and-why-you-might-care-3h9h) +for the wider context. + +## The topology: master and sub-copies + +S3 holds the master. Each Lambda's `/tmp` holds a short-lived, disposable sub-copy. The +ETag is the version token that ties a sub-copy back to the master revision it came from. + +```text + ┌────────────────────────────────────────────────────────┐ + │ Amazon S3 (MASTER) │ + │ [ agent.db ] ETag: "xyz123" │ + └───────────────────────────┬────────────────────────────┘ + │ + ┌────────────────┴────────────────┐ + ▼ (download sub-copy) ▼ (download sub-copy) +┌───────────────────────┐ ┌───────────────────────┐ +│ Lambda Worker A │ │ Lambda Worker B │ +│ Local: /tmp/agent.db │ │ Local: /tmp/agent.db │ +│ Base ETag: "xyz123" │ │ Base ETag: "xyz123" │ +└───────────┬───────────┘ └───────────┬───────────┘ + │ (mutates state) │ (mutates state) + ▼ ▼ + [ wins & uploads ] [ loses & aborts ] + PutObject with PutObject with + If-Match: "xyz123" If-Match: "xyz123" + (accepted; ETag advances) (rejected: 412 Precondition Failed) +``` + +### The lifecycle + +1. **Hydrate.** The tick downloads the snapshot from S3 and keeps its ETag. +2. **Work locally.** All reads and writes happen against the sub-copy in `/tmp`. +3. **Compare-and-swap.** The upload is a `PutObject` carrying `If-Match: `. + On a bootstrap write — no object yet — it carries `If-None-Match: "*"` instead. +4. **Resolve the collision.** If nobody else wrote, the ETag still matches, S3 commits, + and the master advances. If a concurrent writer got there first, the master ETag has + already moved and S3 answers `412 Precondition Failed`. + +The important property is what *doesn't* happen: the loser never overwrites the winner. +There is no silent lost update. The failure is loud and it is on the losing side. + +S3 returns three distinct errors when a conditional write cannot land, and this tutorial +treats them as one and the same abort condition (`src/store/s3.ts`): + +- `412 Precondition Failed` — ETag mismatch on a conditional update. +- `404 NoSuchKey` — the conditional update targets a missing object (the snapshot was + deleted out from under us between hydrate and publish). +- `409 Conditional Request Conflict` — a concurrent operation raced the write (for + example a delete arriving mid-put). + +All three mean "the precondition did not hold, and the previous snapshot stays +authoritative." The store translates each one to the same `PreconditionFailedError`, and +the run row is marked `outcome='error'`. + +## What this repo actually does with a 412 + +The tutorial implements steps 1–4 (`src/store/s3.ts`, `src/agent/fetch.ts`) but stops +short of the last move in the classic pattern — **rebase and retry**. A +`PreconditionFailedError` is treated as an abort, not a retryable tick-level failure: the +run row is marked `outcome='error'` in the local copy, the exception propagates, and the +tick is abandoned. The next scheduled tick starts clean from whatever the master now is. + +Two consequences worth being explicit about: + +- **The losing tick's database work is discarded.** That is the intended trade — losing a + tick's rows is cheap, and a snapshot-wide "merge" would mean reconciling arbitrary + SQL side effects. +- **Side effects already performed are not discarded.** By the time the publish is + attempted, the tick has already called Bedrock and already posted to Discord. A 412 + therefore means: message delivered, database state rolled back. On the fixed 5-minute + schedule with `reservedConcurrentExecutions: 1`, this doesn't happen — it becomes + reachable once a manual trigger can race the scheduled loop. + +If you add write paths, serialize them through one coordinator, or use a durable outbox +with idempotent consumers. Moving side effects before or after a successful publish only +changes which failure loses work; it does not make the operations atomic. Reserved +concurrency only serializes invocations of *one* function; it does nothing about two +different functions writing the same key. + +## Adding rebase-and-retry + +The natural next step, and the one the pattern normally includes: catch the 412, +re-download the now-current master, replay the tick's *inputs* against the fresh +sub-copy, and publish again with the new ETag. Bound the attempts and back off between +them. + +This works well when the tick's work is a pure function of freshly fetched data — which +is close to true here — and badly when replaying means re-running expensive or externally +visible steps. That is the real reason to hoist the Bedrock call and the Discord post out +of the retryable region before adding retries. + +## High contention: the single-writer queue + +Retry loops degrade under load. With enough concurrent writers, every attempt invalidates +someone else's ETag and the fleet spends its time thrashing instead of committing — +progress goes down as concurrency goes up. Past a handful of writers, stop contending and +serialize instead. + +```text +┌─────────────────┐ +│ Lambda Agent 1 │ ──┐ +└─────────────────┘ │ (write requests) +┌─────────────────┐ ▼ ┌───────────────────┐ ┌───────────────────────┐ +│ Lambda Agent 2 │ ────────> │ Amazon SQS Queue │ ─────> │ SINGLE-WRITER LAMBDA │ +└─────────────────┘ ▲ └───────────────────┘ │ reserved concurrency 1│ +┌─────────────────┐ │ └───────────┬───────────┘ +│ Lambda Agent 3 │ ──┘ │ (exclusive) +└─────────────────┘ ▼ + ┌───────────────────────┐ + │ Amazon S3 (master) │ + └───────────────────────┘ +``` + +1. **Make the agents read-only.** They hydrate sub-copies and query them; they never + `PutObject`. +2. **Send writes as messages.** A new fact, a log row, an embedding — serialize the + intent and push it to SQS instead of pushing bytes to S3. +3. **Pin one coordinator.** A separate Lambda with maximum concurrency set to 1 is the + only thing that touches the master. +4. **Batch.** The coordinator drains a batch, downloads the snapshot once, applies every + transaction in one pass, and uploads once. + +Collisions become structurally impossible rather than merely detected, and the S3 write +rate drops to one per batch instead of one per agent. The cost is latency — writes are +now asynchronous — plus the usual SQS concerns: ordering is only per-message-group with a +FIFO queue, and at-least-once delivery means the coordinator's apply step must be +idempotent. + +## When to stop doing this at all + +If you need concurrent writers with transactional consistency across agents, this pattern +is the wrong shape and no amount of tuning fixes it. See +[05-from-tutorial-to-prod.md](05-from-tutorial-to-prod.md) for the exits: Litestream for +long-running containers, Aurora Serverless v2 or DynamoDB for relational state, and +OpenSearch Serverless or pgvector for the embeddings. + +One trap worth naming, because it looks like an easy win: **EFS is not a multi-writer +substitute.** Mounting EFS at `/mnt/storage` removes the per-invocation download, but EFS +offers NFSv4 advisory locking only, and SQLite's `WAL` mode needs POSIX shared memory +that no network filesystem provides. Concurrent writers across Lambda hosts risk +`database is locked` and, in failure cases, corruption. EFS changes where the file lives; +it does not change how many writers SQLite can safely have. diff --git a/docs/superpowers/specs/2026-08-09-smoke-status-iam-design.md b/docs/superpowers/specs/2026-08-09-smoke-status-iam-design.md index ec77fe5..62e47af 100644 --- a/docs/superpowers/specs/2026-08-09-smoke-status-iam-design.md +++ b/docs/superpowers/specs/2026-08-09-smoke-status-iam-design.md @@ -1,7 +1,7 @@ # Loop Mode Smoke Test: IAM-Protected Status Probe — Design **Date:** 2026-08-09 -**Status:** Implemented (spec aligned with shipped code, 2026-08-09) +**Status:** Implemented and live-verified (spec aligned with shipped code, 2026-08-09; live post-deploy verification recorded in §9.1) **Scope:** Make `scripts/smoke.sh` a read-only, IAM-authenticated status probe now that EventBridge owns all scheduled `fetch` writes. Align the Function URL configuration, smoke-test behavior, and public documentation so the tutorial genuinely teaches and verifies SigV4 access without creating duplicate Discord posts or unnecessary Bedrock calls. --- @@ -223,6 +223,19 @@ The implementation is complete when all of the following pass: - [x] The shell harness asserts in every scenario that `aws lambda invoke` and the `fetch` payload are never sent. - [x] The CDK synth suite asserts `AuthType: AWS_IAM` on the `AWS::Lambda::Url` resource, both URL invocation permissions, and the unchanged EventBridge state. - [x] `npm run typecheck`, `npm run build`, and `cdk synth` complete cleanly. -- [x] `npm run smoke` runs to completion immediately after `npm run deploy` (no fetch, `200` empty state, exit 0) and while a loop tick is active (no fetch, signed `200` after any `429` retries, exit 0). *(Pre-deploy verification: the shell harness in `tests/smoke.test.ts` exercises every branch with stubbed `aws`/`curl`/`sleep` and asserts the read-only invariant — see §5.1.)* -- [ ] `npm run smoke` runs to completion immediately after `npm run deploy` (no fetch, `200` empty state, exit 0) and while a loop tick is active (no fetch, signed `200` after any `429` retries, exit 0). *(Live post-deploy check is an operator action; pre-deploy verifications all pass.)* +- [x] `npm run smoke` runs to completion immediately after `npm run deploy` (no fetch, `200` empty state, exit 0) and while a loop tick is active (no fetch, signed `200` after any `429` retries, exit 0). *(Harness verification: the shell harness in `tests/smoke.test.ts` exercises every branch with stubbed `aws`/`curl`/`sleep` and asserts the read-only invariant — see §5.1. Live verification against the deployed stack covered the populated-response and `429`-retry paths only; the `snapshotVersion: null` empty-state branch remains harness-only — see §9.1.)* + +### 9.1 Live verification record (2026-08-09) + +Verified against the deployed stack after `npm run deploy` landed the `AWS_IAM` URL change. + +- **Regression gate proved itself against a real misconfiguration.** Run before the deploy, against the then-current stack whose URL was still `AuthType: NONE`, the unsigned probe returned `200` and the script exited non-zero with the §4 "URL misconfigured back to `NONE`" message. This is the §4 row firing against a genuinely public URL, not a stub. +- **Post-deploy, loop stopped.** Unsigned → `403` (body `{"Message":"Forbidden"}`, rejected at the AWS layer before reaching the handler). Signed → `200` on attempt 1. Exit 0. +- **Post-deploy, loop running.** 127 consecutive `scripts/smoke.sh` runs over 8 minutes, spanning three live `rate(5 minutes)` ticks: 127/127 unsigned probes returned `403`; all signed probes ended `200`; zero non-zero exits. +- **`429` retry path caught live.** Run 127 at `16:57:14Z` logged `Attempt 1: status 429` then `Attempt 2: status 200`, correlating with the fetch tick at `16:57:17Z` (`Duration: 2311.45 ms`). The signed probe hit the `reservedConcurrentExecutions: 1` mutex, backed off `RETRY_DELAY`, and succeeded — the §4 "run while a loop tick is in flight" row against real Lambda concurrency. +- **Read-only invariant held live.** Across the 8-minute window the log group showed exactly three fetch-length invocations (16:47:17, 16:52:19, 16:57:17) — one per scheduled tick and no others. 127 smoke runs produced zero fetches, zero Discord posts, zero Bedrock calls, and `memory.db` changed only on tick boundaries. + +**Caveat — empty-state branch is still harness-only.** The deploy landed on a bucket that already held `memory.db`, so the live runs exercised the populated branch (`Weather source lastValue: NYC: +83°F`). The `snapshotVersion: null` empty-state response is covered by the §5.1 harness but has not been observed live; doing so would require deleting the snapshot object. + +**Note for future live runs.** A tick's contention window is only ~2.3 s out of every 300 s, so a single smoke run is unlikely to observe a `429`. Catching it took repeated back-to-back runs across multiple ticks. - [x] README and `docs/01-architecture.md`, `docs/02-rehydration.md`, `docs/07-budget-protection.md` describe the Function URL as IAM-authenticated and note the on-demand token as defense in depth.