feat(pr3): reader (status op) + version-cached hydration + tutorial docs

README.md

@@ -0,0 +1,46 @@
+# sqlite-s3-agent-tutorial
+
+A working example of the **SQLite-as-a-database-for-an-agent-on-AWS, rehydrated-by-S3**
+pattern: a Discord bot that checks the weather and Bitcoin price once a day, asks an LLM
+(Amazon Bedrock) to turn the raw value into a friendly message, posts it to a Discord
+webhook, and remembers what it already posted — all state lives in a single SQLite file
+in S3. No database server, no VPC.
+
+## Quick start
+
+```bash
+npm install
+npm test
+DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..." npm run local-fetch
+```
+
+That runs the writer against a local SQLite file with no AWS involved (Phase 1). To
+deploy the real thing:
+
+```bash
+export AWS_PROFILE=your-profile
+npm run deploy
+npm run smoke
+```
+
+Before your first deploy, ensure your AWS account in `us-east-1` has an active AWS
+Marketplace subscription for `zai.glm-4.7-flash` (Bedrock enables foundation-model access
+by default in commercial Regions once the Marketplace subscription is in place; the legacy
+manual *Bedrock → Model access* console flow is no longer the gate for this model) — see
+[docs/02-rehydration.md](docs/02-rehydration.md#bedrock-setup) for what else is required and
+what breaks if you skip it.
+
+## What's here
+
+| Doc | Covers |
+|---|---|
+| [docs/01-architecture.md](docs/01-architecture.md) | The pattern, in prose: one Lambda, two ops, one bucket |
+| [docs/02-rehydration.md](docs/02-rehydration.md) | Bootstrap, conditional writes, version-cached reads |
+| [docs/03-schema.md](docs/03-schema.md) | Why three tables, not one |
+| [docs/04-extending.md](docs/04-extending.md) | Adding a third source |
+| [docs/05-from-tutorial-to-prod.md](docs/05-from-tutorial-to-prod.md) | What changes if you outgrow this |
+
+## Cost
+
+At one Discord post per day, `zai.glm-4.7-flash` costs under $0.02/year. See
+[docs/bedrock-model-comparison.md](docs/bedrock-model-comparison.md) for alternatives.

docs/01-architecture.md

@@ -0,0 +1,52 @@
+# Architecture
+
+One Lambda function. Two operations, read as `event.op`: `fetch` (the writer, run daily by
+EventBridge) and `status` (the reader, exposed by a Function URL). Both read and write the
+same single SQLite file that lives durably in one S3 object, but each keeps its own
+transient copy in `/tmp` for the lifetime of the execution environment: the writer at
+`${DB_PATH}` (default `/tmp/memory.db`), the reader at `${DB_PATH}.reader` (default
+`/tmp/memory.db.reader`). Warm Lambda invocations share that `/tmp`, which is exactly what
+lets the status reader reuse its cached database handle (see
+[docs/02-rehydration.md](02-rehydration.md)); cold starts discard it and rehydrate from S3.
+
+## Why one file in S3 instead of a database server
+
+A database server (RDS, DynamoDB) needs to exist continuously, whether or not anything is
+happening. This bot runs once a day. Provisioning a server for a workload that is asleep
+99.9% of the time is the wrong trade — you're paying for uptime a cron job doesn't need.
+SQLite has no server: it's a file format and a library. The only question the "SQLite for
+a stateful Lambda" pattern has to answer is "where does the file live between
+invocations, given Lambda's `/tmp` doesn't survive a cold start?" S3 is the answer:
+durable, versioned, and — critically for this pattern — supports conditional writes via
+`If-Match`, which is what makes concurrent writers safe (see
+[docs/02-rehydration.md](02-rehydration.md)).
+
+## Why one Lambda, not two
+
+`aws-cloud-agent`, the sibling project this tutorial is drawn from, uses two Lambdas — a
+writer and a reader — because its reader also runs semantic search backed by a vector
+index that needs its own warm-container lifecycle tuning. This tutorial's reader is a
+much smaller job: query two tables and return JSON. Splitting it into a second Lambda
+would mean a second container image, a second set of IAM grants, and a second cold-start
+budget — for a query that returns in single-digit milliseconds once hydrated. One function
+with an `op` field is simpler and the tutorial's job is to teach the storage pattern, not
+Lambda topology.
+
+## The single-writer invariant
+
+The function is deployed with `reservedConcurrentExecutions: 1`. Without it, two
+overlapping `fetch` invocations could both hydrate the same S3 version, both do their
+work, and both try to publish — the second one either overwrites the first's notification
+silently (if writes aren't conditional) or gets rejected with a 412 (because they are).
+Reserved concurrency 1 means only one invocation of this function runs at a time, so that
+race can't happen at all. The conditional write is a second line of defense that also
+protects against an out-of-band `aws s3 cp` — belt and suspenders.
+
+## EventBridge's payload
+
+The schedule's `Input` is the literal string `{"op":"fetch"}`, not a transformed event.
+EventBridge's own invocation envelope (the `detail`, `time`, `resources` fields it
+normally wraps a target's input in) is not something the handler ever has to know about —
+it reads `event.op` directly. A transformed input would produce the same behavior, but a
+constant one is unambiguous and doesn't depend on how EventBridge's wrapper shape might
+change.

docs/02-rehydration.md

@@ -0,0 +1,83 @@
+# Rehydration
+
+Three mechanisms make up the pattern this tutorial exists to teach.
+
+## 1. Bootstrap
+
+The very first `fetch` invocation finds nothing at `s3://<bucket>/memory.db` — S3 returns
+`NoSuchKey`. Rather than treating that as an error, the writer opens a brand-new, empty
+SQLite file at `/tmp/memory.db` and runs `bootstrap()`, which is nothing more than
+`CREATE TABLE IF NOT EXISTS` for the three tables in [docs/03-schema.md](03-schema.md).
+Every subsequent invocation also runs `bootstrap()` — it's a no-op against an
+already-migrated file, so there's no cost to always calling it, and it means
+`npm run deploy` produces a working bot without a manual `aws s3 cp` step first.
+
+## 2. Conditional writes
+
+When the writer is ready to publish its updated SQLite file, it doesn't just overwrite
+`s3://<bucket>/memory.db`. It sends the PUT with an `If-Match: <etag>` header, where the
+etag is the one it captured when it downloaded the file at the start of the invocation.
+S3 honors that header at the storage layer: if the object's current etag doesn't match —
+meaning someone else wrote a newer version since this invocation started — S3 rejects the
+write with `412 Precondition Failed` instead of silently clobbering it.
+
+The bootstrap case is the one exception: there's no prior etag to match against, because
+there's no prior object. The `Store` interface models this with `ifMatch: string | null` —
+`null` means "this is a fresh put, fail if the key already exists." `S3Store` translates
+that to an `If-None-Match: "*"` conditional create on the wire, not an unconditioned
+overwrite: if a concurrent deployment or `aws s3 cp` has materialized the object since
+this invocation started, S3 rejects the PUT with a 409 and the writer surfaces the same
+loud failure a 412 would. That keeps S3's HTTP semantics contained inside `S3Store`; the
+writer's orchestration code never sees a header, just a `string | null`.
+
+On a 412, the writer does not retry. A blind retry would mean re-fetching from the
+external weather/crypto API and re-posting to Discord against a snapshot that's already
+stale — the correct response to "someone else wrote first" is to abort loudly and let the
+next scheduled run pick up from the new state. Because `reservedConcurrentExecutions: 1`
+already makes two overlapping writers impossible, a 412 in practice means something else
+went wrong — a misconfiguration, or an out-of-band write — and failing loudly is what
+surfaces that in CloudWatch.
+
+## 3. Version-cached reads
+
+The reader has a different problem: it may be invoked far more often than the writer (a
+human hitting the Function URL to check on the bot), and most of those invocations happen
+against an unchanged snapshot. Re-downloading the whole SQLite file on every request would
+work, but it's wasted I/O on a warm Lambda container that already has last version on
+disk.
+
+Instead, the reader's state lives in a closure returned by `createStatusReader` — that
+closure holds the last snapshot's S3 ETag and the open read-only SQLite handle.
+`src/handler.ts` keeps a module-scope `Map<dbPath, StatusReader>` and looks up (or creates)
+one entry per `dbPath`. Because Lambda doesn't re-run module-level code on every invoke —
+only on cold starts — the Map, and therefore the reader instance it points at, survive
+warm invocations. Each request does a cheap `HEAD` first. If the returned ETag matches
+what's cached and the local file still exists, the reader reuses its already-open SQLite
+handle. Only when the ETag differs does it close the old handle, delete the stale local
+file, download the new one, and open a fresh handle.
+
+That "close, delete, re-download, reopen" sequence matters more than it looks: SQLite
+libraries like `better-sqlite3` keep an in-memory page cache tied to the open file
+descriptor. If the file on disk changes underneath an open handle — which is exactly what
+happens if you just overwrite `/tmp/memory.db` without closing first — the handle's page
+cache goes stale silently. Queries keep succeeding; they just return wrong answers. Closing
+first is what prevents that.
+
+## Bedrock setup
+
+Before the first `fetch` invocation can succeed, the deploying account in `us-east-1` needs
+two things: an active AWS Marketplace subscription for the configured `bedrockModelId`
+(default `zai.glm-4.7-flash`), and an IAM policy that grants `bedrock:InvokeModel` against
+it. Bedrock enables foundation-model access by default in commercial Regions once the
+Marketplace subscription is in place — the legacy manual *Bedrock → Model access* console
+flow is no longer the gating step for this model. If you point `bedrockModelId` at an
+Anthropic model instead, Bedrock requires a separate first-time-use EULA acceptance on the
+same page before that model will invoke; that step *is* still a manual console action and
+is the one remaining reason to open the *Model access* screen.
+
+The CDK stack's IAM policy is generated at synth time from the configured model's family
+(see `src/format/families.ts`) and is permissive enough to invoke the chosen model — but a
+Marketplace subscription must already exist on the account, or `bedrock:InvokeModel`
+returns `AccessDeniedException` regardless of what the IAM policy says. `cdk deploy` does
+not check for the subscription, so the stack deploys cleanly and the first `fetch` fails —
+which is why this tutorial calls it out before the first deploy rather than after.

docs/03-schema.md

@@ -0,0 +1,80 @@
+# Schema
+
+Three tables, prefixed `agent_` so a future migration never collides with anything else
+that might end up sharing the database.
+
+```sql
+CREATE TABLE agent_sources (
+  name TEXT PRIMARY KEY,
+  last_value TEXT,
+  last_fetched_at INTEGER,
+  last_posted_at INTEGER,
+  CONSTRAINT chk_name CHECK (name IN ('weather', 'crypto'))
+);
+
+CREATE TABLE agent_notifications (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  source TEXT NOT NULL,
+  value TEXT NOT NULL,
+  formatted_message TEXT NOT NULL,
+  posted_at INTEGER NOT NULL,
+  FOREIGN KEY (source) REFERENCES agent_sources(name) ON DELETE CASCADE
+);
+
+CREATE INDEX idx_agent_notifications_source_posted_at
+  ON agent_notifications(source, posted_at DESC);
+
+CREATE TABLE agent_runs (
+  run_id TEXT PRIMARY KEY,
+  op TEXT NOT NULL,
+  snapshot_version_in TEXT NOT NULL,
+  started_at INTEGER NOT NULL,
+  ended_at INTEGER,
+  outcome TEXT,
+  sources_checked INTEGER,
+  notifications_sent INTEGER,
+  error TEXT,
+  CONSTRAINT chk_op      CHECK (op IN ('fetch', 'status')),
+  CONSTRAINT chk_outcome CHECK (outcome IS NULL OR outcome IN ('success', 'error'))
+);
+```
+
+## Why three tables, not one
+
+`agent_sources` answers "what should I skip?" — it's the dedup state, one row per source,
+overwritten in place. `agent_notifications` answers "what did I actually post?" — it's an
+append-only history. `agent_runs` answers "did the bot itself work?" — it's an
+observability log, independent of whether any individual source produced a notification.
+Merging these would make each question harder to answer: putting `last_posted_at` only on
+`agent_notifications`, for instance, would turn "what's the current dedup state for
+weather?" into a query that has to find the most recent row and hope nothing raced it,
+instead of a primary-key lookup.
+
+## Why both `value` and `formatted_message`
+
+Dedup has to compare against something byte-for-byte stable: the same weather reading
+should always produce the same string. The LLM-formatted message is the opposite —
+non-deterministic by design, because the whole point of running it through Bedrock is to
+get natural, varied phrasing. If dedup compared `formatted_message` instead of `value`,
+an unchanged `72F` reading would produce a different message every time it was checked,
+and the dedup logic would never fire — every run would post, defeating the entire feature
+and burning an LLM call it didn't need to. `value` is what dedup reads; `formatted_message`
+is what a human reads. Storing both means the reader can show what was actually posted
+without paying for a second Bedrock call just to redisplay it.
+
+## Why `outcome` and `error` are nullable
+
+An `agent_runs` row is inserted at the *start* of a run, before any work happens, and
+updated at the end. A row where `ended_at` is still `NULL` is not missing data — it's a
+record that the process crashed or was killed mid-run, which is exactly the failure mode
+you'd otherwise have no visibility into. Defaulting `outcome` to some placeholder value
+would erase that signal.
+
+## Why `source` is a closed vocabulary
+
+The `CHECK` constraint on `agent_sources.name` accepts only `'weather'` and `'crypto'`. A
+typo like `'wether'` would otherwise silently create a third, orphaned dedup row that
+never gets checked against — the bug would look like "the weather bot stopped noticing
+changes," which is a much harder thing to debug than a constraint violation at insert
+time. Extending the tutorial to a third source means editing this one constraint plus one
+new `SourceFetcher` — see [docs/04-extending.md](04-extending.md).

docs/04-extending.md

@@ -0,0 +1,69 @@
+# Extending: adding a third source
+
+The tutorial ships with two sources — `weather` and `crypto` — deliberately, to keep the
+example small. Adding a third (say, a daily quote, or a stock price) touches exactly three
+places.
+
+## 1. The schema's closed vocabulary
+
+In `src/db/schema.ts`, add the new name to `SOURCE_NAMES`:
+
+```typescript
+export const SOURCE_NAMES = ['weather', 'crypto', 'quote'] as const;
+```
+
+The `CHECK` constraint in `AGENT_DDL` is generated from a literal SQL string, not from
+`SOURCE_NAMES` — update it too:
+
+```sql
+CONSTRAINT chk_name CHECK (name IN ('weather', 'crypto', 'quote'))
+```
+
+This is a deliberate lack of DRY: SQLite's `CHECK` clause can't reference a TypeScript
+array at schema-application time, and generating SQL from the array would obscure exactly
+the constraint a reader most needs to see when debugging a rejected insert. Keep the two
+lists next to each other and change them together.
+
+## 2. A `SourceFetcher`
+
+Add `src/sources/quote.ts`:
+
+```typescript
+import type { SourceFetcher } from './types.js';
+
+export function createQuoteFetcher(): SourceFetcher {
+  return {
+    name: 'quote',
+    async fetch() {
+      const response = await fetch('https://api.example.com/quote-of-the-day');
+      if (!response.ok) {
+        throw new Error(`quote API responded ${response.status}`);
+      }
+      const json = (await response.json()) as { quote?: string };
+      if (json.quote === undefined) {
+        throw new Error('quote API response missing quote field');
+      }
+      return json.quote;
+    },
+  };
+}
+```
+
+Register it in `src/sources/index.ts`'s `switch` statement.
+
+## 3. Nothing else
+
+`src/agent/fetch.ts`, `src/format/*`, `src/agent/status.ts`, and `infra/stack.ts` all
+operate on `SourceName` generically — none of them special-case `'weather'` or `'crypto'`
+by name. Once the schema and the fetcher exist, `SOURCES='["weather","crypto","quote"]'`
+(or the equivalent env var on the deployed function) picks up the new source with no
+further code changes. That genericity is why the closed vocabulary lives in exactly one
+place instead of being re-validated at every call site.
+
+## What this tutorial deliberately doesn't support
+
+A source registry, plugin system, or dynamic configuration of *which* sources exist at
+runtime — see the design spec's §8 ("Out of scope"). Three sources or thirty, the pattern
+is the same: edit the constraint, add a fetcher, register it. A registry would only pay
+for itself past the point where "edit two files" stops being fast enough, and this
+tutorial's job is to teach the storage pattern, not to anticipate that scale.

docs/05-from-tutorial-to-prod.md

@@ -0,0 +1,54 @@
+# From tutorial to production
+
+This tutorial's defaults are chosen for clarity, not for running a real business on. If
+you outgrow it, here's what changes — and `aws-cloud-agent`
+(github.com/equationalapplications/aws-cloud-agent), the sibling project this pattern was
+drawn from, is a working example of most of these deltas already applied.
+
+## Bucket lifecycle
+
+This tutorial's bucket is `RemovalPolicy.DESTROY` with `autoDeleteObjects: true`, so
+`cdk destroy` cleans up completely — useful for a tutorial you might spin up and tear down
+several times while learning it. A production system generally wants
+`RemovalPolicy.RETAIN`: losing the bucket should require a deliberate, separate action,
+not be a side effect of a stack deletion. `aws-cloud-agent` also versions its bucket and
+retains every version indefinitely, because it supports restoring to a prior snapshot —
+this tutorial doesn't need that if the only state that matters is "what was the last value
+posted."
+
+## More than one writer path
+
+This tutorial has exactly one thing that writes to the snapshot: the `fetch` op, on a
+fixed daily schedule. A production agent is more likely to need multiple write paths — a
+scheduled job and a manually-triggered one, say — which raises the question of whether
+`reservedConcurrentExecutions: 1` is still sufficient once two *different* Lambda
+functions might both want to write. It isn't, on its own: reserved concurrency only
+serializes invocations of one function. `aws-cloud-agent` handles this by giving every
+writer path the same conditional-write discipline this tutorial uses, so the S3 `If-Match`
+precondition — not Lambda's concurrency control — is what actually prevents two writers
+from clobbering each other, regardless of how many entry points call into that logic.
+
+## The single Lambda split
+
+This tutorial's `fetch` and `status` share one function because the reader's query is
+cheap. If your reader starts doing real work — search, aggregation, anything with its own
+latency and memory profile — split it into its own function, the way `aws-cloud-agent`
+splits writer and reader. The two functions still share the storage pattern in this
+tutorial's `docs/02-rehydration.md`; only the deployment topology changes.
+
+## Model selection
+
+This tutorial defaults to `zai.glm-4.7-flash` for cost — the whole daily notification
+costs under $0.02/year at that price point (see `docs/bedrock-model-comparison.md`). A
+production system with actual latency or quality requirements should pick a model the way
+`aws-cloud-agent`'s cost-rebalance design doc does: probe candidates against your real
+prompt, not against a price list, and pin the choice with the same "why not something
+else" reasoning that doc records.
+
+## What stays the same
+
+The rehydration protocol — bootstrap, conditional writes, version-cached reads — doesn't
+change shape as the system grows. That's the point of the pattern: it's the same mechanism
+whether the payload is a two-table dedup cache or `aws-cloud-agent`'s full knowledge graph
+with a vector index. What changes is how much work happens between hydrate and publish,
+not how hydrate and publish themselves work.