diff --git a/README.md b/README.md new file mode 100644 index 0000000..87a2d9b --- /dev/null +++ b/README.md @@ -0,0 +1,46 @@ +# sqlite-s3-agent-tutorial + +A working example of the **SQLite-as-a-database-for-an-agent-on-AWS, rehydrated-by-S3** +pattern: a Discord bot that checks the weather and Bitcoin price once a day, asks an LLM +(Amazon Bedrock) to turn the raw value into a friendly message, posts it to a Discord +webhook, and remembers what it already posted — all state lives in a single SQLite file +in S3. No database server, no VPC. + +## Quick start + +```bash +npm install +npm test +DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..." npm run local-fetch +``` + +That runs the writer against a local SQLite file with no AWS involved (Phase 1). To +deploy the real thing: + +```bash +export AWS_PROFILE=your-profile +npm run deploy +npm run smoke +``` + +Before your first deploy, ensure your AWS account in `us-east-1` has an active AWS +Marketplace subscription for `zai.glm-4.7-flash` (Bedrock enables foundation-model access +by default in commercial Regions once the Marketplace subscription is in place; the legacy +manual *Bedrock → Model access* console flow is no longer the gate for this model) — see +[docs/02-rehydration.md](docs/02-rehydration.md#bedrock-setup) for what else is required and +what breaks if you skip it. + +## What's here + +| Doc | Covers | +|---|---| +| [docs/01-architecture.md](docs/01-architecture.md) | The pattern, in prose: one Lambda, two ops, one bucket | +| [docs/02-rehydration.md](docs/02-rehydration.md) | Bootstrap, conditional writes, version-cached reads | +| [docs/03-schema.md](docs/03-schema.md) | Why three tables, not one | +| [docs/04-extending.md](docs/04-extending.md) | Adding a third source | +| [docs/05-from-tutorial-to-prod.md](docs/05-from-tutorial-to-prod.md) | What changes if you outgrow this | + +## Cost + +At one Discord post per day, `zai.glm-4.7-flash` costs under $0.02/year. See +[docs/bedrock-model-comparison.md](docs/bedrock-model-comparison.md) for alternatives. diff --git a/docs/01-architecture.md b/docs/01-architecture.md new file mode 100644 index 0000000..35a28bb --- /dev/null +++ b/docs/01-architecture.md @@ -0,0 +1,52 @@ +# Architecture + +One Lambda function. Two operations, read as `event.op`: `fetch` (the writer, run daily by +EventBridge) and `status` (the reader, exposed by a Function URL). Both read and write the +same single SQLite file that lives durably in one S3 object, but each keeps its own +transient copy in `/tmp` for the lifetime of the execution environment: the writer at +`${DB_PATH}` (default `/tmp/memory.db`), the reader at `${DB_PATH}.reader` (default +`/tmp/memory.db.reader`). Warm Lambda invocations share that `/tmp`, which is exactly what +lets the status reader reuse its cached database handle (see +[docs/02-rehydration.md](02-rehydration.md)); cold starts discard it and rehydrate from S3. + +## Why one file in S3 instead of a database server + +A database server (RDS, DynamoDB) needs to exist continuously, whether or not anything is +happening. This bot runs once a day. Provisioning a server for a workload that is asleep +99.9% of the time is the wrong trade — you're paying for uptime a cron job doesn't need. +SQLite has no server: it's a file format and a library. The only question the "SQLite for +a stateful Lambda" pattern has to answer is "where does the file live between +invocations, given Lambda's `/tmp` doesn't survive a cold start?" S3 is the answer: +durable, versioned, and — critically for this pattern — supports conditional writes via +`If-Match`, which is what makes concurrent writers safe (see +[docs/02-rehydration.md](02-rehydration.md)). + +## Why one Lambda, not two + +`aws-cloud-agent`, the sibling project this tutorial is drawn from, uses two Lambdas — a +writer and a reader — because its reader also runs semantic search backed by a vector +index that needs its own warm-container lifecycle tuning. This tutorial's reader is a +much smaller job: query two tables and return JSON. Splitting it into a second Lambda +would mean a second container image, a second set of IAM grants, and a second cold-start +budget — for a query that returns in single-digit milliseconds once hydrated. One function +with an `op` field is simpler and the tutorial's job is to teach the storage pattern, not +Lambda topology. + +## The single-writer invariant + +The function is deployed with `reservedConcurrentExecutions: 1`. Without it, two +overlapping `fetch` invocations could both hydrate the same S3 version, both do their +work, and both try to publish — the second one either overwrites the first's notification +silently (if writes aren't conditional) or gets rejected with a 412 (because they are). +Reserved concurrency 1 means only one invocation of this function runs at a time, so that +race can't happen at all. The conditional write is a second line of defense that also +protects against an out-of-band `aws s3 cp` — belt and suspenders. + +## EventBridge's payload + +The schedule's `Input` is the literal string `{"op":"fetch"}`, not a transformed event. +EventBridge's own invocation envelope (the `detail`, `time`, `resources` fields it +normally wraps a target's input in) is not something the handler ever has to know about — +it reads `event.op` directly. A transformed input would produce the same behavior, but a +constant one is unambiguous and doesn't depend on how EventBridge's wrapper shape might +change. diff --git a/docs/02-rehydration.md b/docs/02-rehydration.md new file mode 100644 index 0000000..bf62e14 --- /dev/null +++ b/docs/02-rehydration.md @@ -0,0 +1,83 @@ +# Rehydration + +Three mechanisms make up the pattern this tutorial exists to teach. + +## 1. Bootstrap + +The very first `fetch` invocation finds nothing at `s3:///memory.db` — S3 returns +`NoSuchKey`. Rather than treating that as an error, the writer opens a brand-new, empty +SQLite file at `/tmp/memory.db` and runs `bootstrap()`, which is nothing more than +`CREATE TABLE IF NOT EXISTS` for the three tables in [docs/03-schema.md](03-schema.md). +Every subsequent invocation also runs `bootstrap()` — it's a no-op against an +already-migrated file, so there's no cost to always calling it, and it means +`npm run deploy` produces a working bot without a manual `aws s3 cp` step first. + +## 2. Conditional writes + +When the writer is ready to publish its updated SQLite file, it doesn't just overwrite +`s3:///memory.db`. It sends the PUT with an `If-Match: ` header, where the +etag is the one it captured when it downloaded the file at the start of the invocation. +S3 honors that header at the storage layer: if the object's current etag doesn't match — +meaning someone else wrote a newer version since this invocation started — S3 rejects the +write with `412 Precondition Failed` instead of silently clobbering it. + +The bootstrap case is the one exception: there's no prior etag to match against, because +there's no prior object. The `Store` interface models this with `ifMatch: string | null` — +`null` means "this is a fresh put, fail if the key already exists." `S3Store` translates +that to an `If-None-Match: "*"` conditional create on the wire, not an unconditioned +overwrite: if a concurrent deployment or `aws s3 cp` has materialized the object since +this invocation started, S3 rejects the PUT with a 409 and the writer surfaces the same +loud failure a 412 would. That keeps S3's HTTP semantics contained inside `S3Store`; the +writer's orchestration code never sees a header, just a `string | null`. + +On a 412, the writer does not retry. A blind retry would mean re-fetching from the +external weather/crypto API and re-posting to Discord against a snapshot that's already +stale — the correct response to "someone else wrote first" is to abort loudly and let the +next scheduled run pick up from the new state. Because `reservedConcurrentExecutions: 1` +already makes two overlapping writers impossible, a 412 in practice means something else +went wrong — a misconfiguration, or an out-of-band write — and failing loudly is what +surfaces that in CloudWatch. + +## 3. Version-cached reads + +The reader has a different problem: it may be invoked far more often than the writer (a +human hitting the Function URL to check on the bot), and most of those invocations happen +against an unchanged snapshot. Re-downloading the whole SQLite file on every request would +work, but it's wasted I/O on a warm Lambda container that already has last version on +disk. + +Instead, the reader's state lives in a closure returned by `createStatusReader` — that +closure holds the last snapshot's S3 ETag and the open read-only SQLite handle. +`src/handler.ts` keeps a module-scope `Map` and looks up (or creates) +one entry per `dbPath`. Because Lambda doesn't re-run module-level code on every invoke — +only on cold starts — the Map, and therefore the reader instance it points at, survive +warm invocations. Each request does a cheap `HEAD` first. If the returned ETag matches +what's cached and the local file still exists, the reader reuses its already-open SQLite +handle. Only when the ETag differs does it close the old handle, delete the stale local +file, download the new one, and open a fresh handle. + +That "close, delete, re-download, reopen" sequence matters more than it looks: SQLite +libraries like `better-sqlite3` keep an in-memory page cache tied to the open file +descriptor. If the file on disk changes underneath an open handle — which is exactly what +happens if you just overwrite `/tmp/memory.db` without closing first — the handle's page +cache goes stale silently. Queries keep succeeding; they just return wrong answers. Closing +first is what prevents that. + +## Bedrock setup + +Before the first `fetch` invocation can succeed, the deploying account in `us-east-1` needs +two things: an active AWS Marketplace subscription for the configured `bedrockModelId` +(default `zai.glm-4.7-flash`), and an IAM policy that grants `bedrock:InvokeModel` against +it. Bedrock enables foundation-model access by default in commercial Regions once the +Marketplace subscription is in place — the legacy manual *Bedrock → Model access* console +flow is no longer the gating step for this model. If you point `bedrockModelId` at an +Anthropic model instead, Bedrock requires a separate first-time-use EULA acceptance on the +same page before that model will invoke; that step *is* still a manual console action and +is the one remaining reason to open the *Model access* screen. + +The CDK stack's IAM policy is generated at synth time from the configured model's family +(see `src/format/families.ts`) and is permissive enough to invoke the chosen model — but a +Marketplace subscription must already exist on the account, or `bedrock:InvokeModel` +returns `AccessDeniedException` regardless of what the IAM policy says. `cdk deploy` does +not check for the subscription, so the stack deploys cleanly and the first `fetch` fails — +which is why this tutorial calls it out before the first deploy rather than after. diff --git a/docs/03-schema.md b/docs/03-schema.md new file mode 100644 index 0000000..d143767 --- /dev/null +++ b/docs/03-schema.md @@ -0,0 +1,80 @@ +# Schema + +Three tables, prefixed `agent_` so a future migration never collides with anything else +that might end up sharing the database. + +```sql +CREATE TABLE agent_sources ( + name TEXT PRIMARY KEY, + last_value TEXT, + last_fetched_at INTEGER, + last_posted_at INTEGER, + CONSTRAINT chk_name CHECK (name IN ('weather', 'crypto')) +); + +CREATE TABLE agent_notifications ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source TEXT NOT NULL, + value TEXT NOT NULL, + formatted_message TEXT NOT NULL, + posted_at INTEGER NOT NULL, + FOREIGN KEY (source) REFERENCES agent_sources(name) ON DELETE CASCADE +); + +CREATE INDEX idx_agent_notifications_source_posted_at + ON agent_notifications(source, posted_at DESC); + +CREATE TABLE agent_runs ( + run_id TEXT PRIMARY KEY, + op TEXT NOT NULL, + snapshot_version_in TEXT NOT NULL, + started_at INTEGER NOT NULL, + ended_at INTEGER, + outcome TEXT, + sources_checked INTEGER, + notifications_sent INTEGER, + error TEXT, + CONSTRAINT chk_op CHECK (op IN ('fetch', 'status')), + CONSTRAINT chk_outcome CHECK (outcome IS NULL OR outcome IN ('success', 'error')) +); +``` + +## Why three tables, not one + +`agent_sources` answers "what should I skip?" — it's the dedup state, one row per source, +overwritten in place. `agent_notifications` answers "what did I actually post?" — it's an +append-only history. `agent_runs` answers "did the bot itself work?" — it's an +observability log, independent of whether any individual source produced a notification. +Merging these would make each question harder to answer: putting `last_posted_at` only on +`agent_notifications`, for instance, would turn "what's the current dedup state for +weather?" into a query that has to find the most recent row and hope nothing raced it, +instead of a primary-key lookup. + +## Why both `value` and `formatted_message` + +Dedup has to compare against something byte-for-byte stable: the same weather reading +should always produce the same string. The LLM-formatted message is the opposite — +non-deterministic by design, because the whole point of running it through Bedrock is to +get natural, varied phrasing. If dedup compared `formatted_message` instead of `value`, +an unchanged `72F` reading would produce a different message every time it was checked, +and the dedup logic would never fire — every run would post, defeating the entire feature +and burning an LLM call it didn't need to. `value` is what dedup reads; `formatted_message` +is what a human reads. Storing both means the reader can show what was actually posted +without paying for a second Bedrock call just to redisplay it. + +## Why `outcome` and `error` are nullable + +An `agent_runs` row is inserted at the *start* of a run, before any work happens, and +updated at the end. A row where `ended_at` is still `NULL` is not missing data — it's a +record that the process crashed or was killed mid-run, which is exactly the failure mode +you'd otherwise have no visibility into. Defaulting `outcome` to some placeholder value +would erase that signal. + +## Why `source` is a closed vocabulary + +The `CHECK` constraint on `agent_sources.name` accepts only `'weather'` and `'crypto'`. A +typo like `'wether'` would otherwise silently create a third, orphaned dedup row that +never gets checked against — the bug would look like "the weather bot stopped noticing +changes," which is a much harder thing to debug than a constraint violation at insert +time. Extending the tutorial to a third source means editing this one constraint plus one +new `SourceFetcher` — see [docs/04-extending.md](04-extending.md). diff --git a/docs/04-extending.md b/docs/04-extending.md new file mode 100644 index 0000000..d684dff --- /dev/null +++ b/docs/04-extending.md @@ -0,0 +1,69 @@ +# Extending: adding a third source + +The tutorial ships with two sources — `weather` and `crypto` — deliberately, to keep the +example small. Adding a third (say, a daily quote, or a stock price) touches exactly three +places. + +## 1. The schema's closed vocabulary + +In `src/db/schema.ts`, add the new name to `SOURCE_NAMES`: + +```typescript +export const SOURCE_NAMES = ['weather', 'crypto', 'quote'] as const; +``` + +The `CHECK` constraint in `AGENT_DDL` is generated from a literal SQL string, not from +`SOURCE_NAMES` — update it too: + +```sql +CONSTRAINT chk_name CHECK (name IN ('weather', 'crypto', 'quote')) +``` + +This is a deliberate lack of DRY: SQLite's `CHECK` clause can't reference a TypeScript +array at schema-application time, and generating SQL from the array would obscure exactly +the constraint a reader most needs to see when debugging a rejected insert. Keep the two +lists next to each other and change them together. + +## 2. A `SourceFetcher` + +Add `src/sources/quote.ts`: + +```typescript +import type { SourceFetcher } from './types.js'; + +export function createQuoteFetcher(): SourceFetcher { + return { + name: 'quote', + async fetch() { + const response = await fetch('https://api.example.com/quote-of-the-day'); + if (!response.ok) { + throw new Error(`quote API responded ${response.status}`); + } + const json = (await response.json()) as { quote?: string }; + if (json.quote === undefined) { + throw new Error('quote API response missing quote field'); + } + return json.quote; + }, + }; +} +``` + +Register it in `src/sources/index.ts`'s `switch` statement. + +## 3. Nothing else + +`src/agent/fetch.ts`, `src/format/*`, `src/agent/status.ts`, and `infra/stack.ts` all +operate on `SourceName` generically — none of them special-case `'weather'` or `'crypto'` +by name. Once the schema and the fetcher exist, `SOURCES='["weather","crypto","quote"]'` +(or the equivalent env var on the deployed function) picks up the new source with no +further code changes. That genericity is why the closed vocabulary lives in exactly one +place instead of being re-validated at every call site. + +## What this tutorial deliberately doesn't support + +A source registry, plugin system, or dynamic configuration of *which* sources exist at +runtime — see the design spec's §8 ("Out of scope"). Three sources or thirty, the pattern +is the same: edit the constraint, add a fetcher, register it. A registry would only pay +for itself past the point where "edit two files" stops being fast enough, and this +tutorial's job is to teach the storage pattern, not to anticipate that scale. diff --git a/docs/05-from-tutorial-to-prod.md b/docs/05-from-tutorial-to-prod.md new file mode 100644 index 0000000..fbbd239 --- /dev/null +++ b/docs/05-from-tutorial-to-prod.md @@ -0,0 +1,54 @@ +# From tutorial to production + +This tutorial's defaults are chosen for clarity, not for running a real business on. If +you outgrow it, here's what changes — and `aws-cloud-agent` +(github.com/equationalapplications/aws-cloud-agent), the sibling project this pattern was +drawn from, is a working example of most of these deltas already applied. + +## Bucket lifecycle + +This tutorial's bucket is `RemovalPolicy.DESTROY` with `autoDeleteObjects: true`, so +`cdk destroy` cleans up completely — useful for a tutorial you might spin up and tear down +several times while learning it. A production system generally wants +`RemovalPolicy.RETAIN`: losing the bucket should require a deliberate, separate action, +not be a side effect of a stack deletion. `aws-cloud-agent` also versions its bucket and +retains every version indefinitely, because it supports restoring to a prior snapshot — +this tutorial doesn't need that if the only state that matters is "what was the last value +posted." + +## More than one writer path + +This tutorial has exactly one thing that writes to the snapshot: the `fetch` op, on a +fixed daily schedule. A production agent is more likely to need multiple write paths — a +scheduled job and a manually-triggered one, say — which raises the question of whether +`reservedConcurrentExecutions: 1` is still sufficient once two *different* Lambda +functions might both want to write. It isn't, on its own: reserved concurrency only +serializes invocations of one function. `aws-cloud-agent` handles this by giving every +writer path the same conditional-write discipline this tutorial uses, so the S3 `If-Match` +precondition — not Lambda's concurrency control — is what actually prevents two writers +from clobbering each other, regardless of how many entry points call into that logic. + +## The single Lambda split + +This tutorial's `fetch` and `status` share one function because the reader's query is +cheap. If your reader starts doing real work — search, aggregation, anything with its own +latency and memory profile — split it into its own function, the way `aws-cloud-agent` +splits writer and reader. The two functions still share the storage pattern in this +tutorial's `docs/02-rehydration.md`; only the deployment topology changes. + +## Model selection + +This tutorial defaults to `zai.glm-4.7-flash` for cost — the whole daily notification +costs under $0.02/year at that price point (see `docs/bedrock-model-comparison.md`). A +production system with actual latency or quality requirements should pick a model the way +`aws-cloud-agent`'s cost-rebalance design doc does: probe candidates against your real +prompt, not against a price list, and pin the choice with the same "why not something +else" reasoning that doc records. + +## What stays the same + +The rehydration protocol — bootstrap, conditional writes, version-cached reads — doesn't +change shape as the system grows. That's the point of the pattern: it's the same mechanism +whether the payload is a two-table dedup cache or `aws-cloud-agent`'s full knowledge graph +with a vector index. What changes is how much work happens between hydrate and publish, +not how hydrate and publish themselves work. diff --git a/docs/superpowers/specs/2026-08-08-sqlite-s3-agent-tutorial-design.md b/docs/superpowers/specs/2026-08-08-sqlite-s3-agent-tutorial-design.md index 11ea6e1..24a68d1 100644 --- a/docs/superpowers/specs/2026-08-08-sqlite-s3-agent-tutorial-design.md +++ b/docs/superpowers/specs/2026-08-08-sqlite-s3-agent-tutorial-design.md @@ -76,7 +76,7 @@ One Lambda function, two ops, one S3 bucket, one SQLite file. **LLM message formatting.** Between the value fetch and the Discord post, the writer calls Amazon Bedrock (default `zai.glm-4.7-flash`) to turn the raw value into a friendly message. Dedup runs on the raw `value` *before* the LLM call (§3.1 step 3), so unchanged values never invoke Bedrock — the model is paid for only when there's actually something new to say. The model choice is overridable per environment via `bedrockModelId` (§11). The same `MessageFormatter` interface is implemented by `LocalTemplateFormatter` (Phase 1, no AWS) and `BedrockFormatter` (Phase 3, default), so the writer's hot path doesn't change between local and deployed. -**`fetch` is the writer; `status` is the reader.** Both share `/tmp/memory.db`. The reader's job is to make the writer's state visible — without it, the only way to inspect the bot is `aws s3 cp` and `sqlite3`, which is bad tutorial UX. The reader is a JSON endpoint, not a UI: callers (`curl`, `aws lambda invoke`, browser address bar) see `sources[]` and `recentNotifications[]`, not a dashboard. +**`fetch` is the writer; `status` is the reader.** Both read and write the same single S3 object, but each keeps its own local copy under `/tmp`: the writer at `${DB_PATH}` (default `/tmp/memory.db`), the reader at `${DB_PATH}.reader` (default `/tmp/memory.db.reader`). The split matters because the writer mutates its local file on every invocation, including the conditional-write failure path where S3 rejects the PUT with 412 and the writer records the `outcome='error'` run row locally — the S3 ETag does not change in that branch, so a reader sharing the writer's path would see an ETag cache hit and answer from the still-open reader handle against the writer's mutated bytes. Keeping the two local paths disjoint means the reader's local copy only changes when the reader itself downloads a new snapshot. The reader's job is to make the writer's state visible — without it, the only way to inspect the bot is `aws s3 cp` and `sqlite3`, which is bad tutorial UX. The reader is a JSON endpoint, not a UI: callers (`curl`, `aws lambda invoke`, browser address bar) see `sources[]` and `recentNotifications[]`, not a dashboard. **Single-writer invariant.** The function has `reservedConcurrency: 1`. Without it, two simultaneous `fetch` invocations could both hydrate the same version, both upload, and silently overwrite each other's writes. The `status` op is read-only by IAM (`s3:GetObject` only, no `s3:PutObject`), so the writer's state is safe even though it shares the role. @@ -113,9 +113,9 @@ One Lambda function, two ops, one S3 bucket, one SQLite file. ``` 1. HEAD on s3:///memory.db; capture ETag -2. If module-scope cached ETag equals current and /tmp/memory.db exists: +2. If module-scope cached ETag equals current and /tmp/memory.db.reader exists: reuse, open read-only handle -3. Else: close any open handle, rm /tmp/memory.db, GetObject → /tmp/memory.db, +3. Else: close any open handle, rm /tmp/memory.db.reader, GetObject → /tmp/memory.db.reader, cache ETag, open read-only handle 4. Query: SELECT name, last_value, last_fetched_at, last_posted_at FROM agent_sources + last N rows from agent_notifications JOIN agent_sources @@ -153,8 +153,10 @@ The writer calls `Store.put(key, body, ifMatch)`. The `ifMatch` argument is the The reader keeps the last hydrated ETag in module scope. Each invocation: 1. `HEAD s3:///memory.db`. Capture current ETag. -2. If module-scope ETag equals current ETag and `/tmp/memory.db` exists: open a read-only handle to the existing file. -3. Else: close any open handle, `rm /tmp/memory.db`, `GetObject` → `/tmp/memory.db`, open a read-only handle, update module-scope ETag. **If `GetObject` returns `NoSuchKey`** (no snapshot yet — `fetch` has never run successfully), return the empty-state JSON — `{ snapshotVersion: null, sources: [], recentNotifications: [] }` — without opening a handle. There is nothing to query until the writer has produced at least one snapshot. +2. If module-scope ETag equals current ETag and `/tmp/memory.db.reader` exists: open a read-only handle to the existing file. +3. Else: close any open handle, `rm /tmp/memory.db.reader`, `GetObject` → `/tmp/memory.db.reader`, open a read-only handle, update module-scope ETag. **If `GetObject` returns `NoSuchKey`** (no snapshot yet — `fetch` has never run successfully), return the empty-state JSON — `{ snapshotVersion: null, sources: [], recentNotifications: [] }` — without opening a handle. There is nothing to query until the writer has produced at least one snapshot. + +**Why a separate local path from the writer's.** The writer mutates its local copy on every invocation — including the conditional-write failure path, where a 412 from S3 leaves the writer's local file with the `outcome='error'` run row recorded but S3's ETag unchanged. If the reader shared the writer's local path, the reader's ETag cache hit would answer from the still-open reader handle against those locally-mutated bytes, even though the authoritative S3 snapshot did not change. Disjoint local paths keep the reader's view strictly in step with what the writer has actually published. **Why close-and-reopen rather than reuse the open handle?** `better-sqlite3` keeps a page cache in memory. If the file on disk changes underneath an open handle, the cache describes a file that no longer exists — silently wrong answers, no error. The mechanism is the same one `aws-cloud-agent` uses for the same reason. diff --git a/package.json b/package.json index 342ad36..99d2548 100644 --- a/package.json +++ b/package.json @@ -13,7 +13,8 @@ "test:watch": "vitest", "local-fetch": "tsx src/localFetch.ts", "cdk": "cdk", - "deploy": "bash scripts/deploy.sh" + "deploy": "bash scripts/deploy.sh", + "smoke": "bash scripts/smoke.sh" }, "dependencies": { "better-sqlite3": "^13.0.1", diff --git a/scripts/smoke.sh b/scripts/smoke.sh new file mode 100755 index 0000000..40fed69 --- /dev/null +++ b/scripts/smoke.sh @@ -0,0 +1,100 @@ +#!/usr/bin/env bash +set -euo pipefail + +PROFILE="${AWS_PROFILE:-default}" +REGION="${AWS_REGION:-us-east-1}" +STACK_NAME="SqliteS3AgentTutorial" + +# Both files are sensitive (the netrc holds AWS credentials, the fetch response holds the +# function output) and both live in a predictable location if hardcoded — `mktemp` gives a +# per-invocation path that an attacker on the same box can't pre-create or symlink-over. +# Exit trap cleans them up so a failure mid-run doesn't leave credentials on disk. +FETCH_RESPONSE_FILE=$(mktemp) +NETRC_FILE=$(mktemp) +chmod 600 "$NETRC_FILE" +# Populated below when the resolved credentials carry a SessionToken; kept off the +# process command line (visible in `ps aux` for the curl lifetime) by writing the +# header to a 0600 tempfile and letting curl read it via `--header "@file"`. +SECURITY_TOKEN_HEADER_FILE="" +trap 'rm -f "$FETCH_RESPONSE_FILE" "$NETRC_FILE" ${SECURITY_TOKEN_HEADER_FILE:+"$SECURITY_TOKEN_HEADER_FILE"}' EXIT + +echo "=== Fetching stack outputs ===" +outputs=$(aws cloudformation describe-stacks \ + --profile "$PROFILE" \ + --region "$REGION" \ + --stack-name "$STACK_NAME" \ + --query "Stacks[0].Outputs" \ + --output json) + +FUNCTION_NAME=$(echo "$outputs" | jq -r '.[] | select(.OutputKey == "AgentFunctionName") | .OutputValue') +FUNCTION_URL=$(echo "$outputs" | jq -r '.[] | select(.OutputKey == "AgentFunctionUrl") | .OutputValue') + +echo "Function: $FUNCTION_NAME" +echo "Function URL: $FUNCTION_URL" + +echo "" +echo "=== Invoking fetch ===" +aws lambda invoke \ + --profile "$PROFILE" \ + --region "$REGION" \ + --function-name "$FUNCTION_NAME" \ + --payload '{"op":"fetch"}' \ + --cli-binary-format raw-in-base64-out \ + "$FETCH_RESPONSE_FILE" + +echo "Fetch response:" +cat "$FETCH_RESPONSE_FILE" | jq . + +echo "" +echo "=== Waiting for the run to settle ===" +sleep 5 + +echo "" +echo "=== Querying status ===" +# Credentials go through a 0600 netrc file, not argv — `curl --user "$key:$secret"` would +# put the secret access key in `ps aux` output for the process lifetime. +# +# `aws configure export-credentials --format process` resolves through the full AWS CLI +# credential chain (env vars, SSO, `credential_process`, etc.), unlike `aws configure get` +# which only reads the static profile file. When the resolved credentials include a +# `SessionToken` — i.e. the profile is SSO or assumed-role — curl needs to send it as +# `X-Amz-Security-Token` for SigV4 to accept the signature. +FUNCTION_HOST=$(echo "$FUNCTION_URL" | sed -E 's#^https?://([^/]+).*#\1#') +credentials_json=$(aws configure export-credentials --profile "$PROFILE" --format process) +access_key=$(jq -r '.AccessKeyId' <<<"$credentials_json") +secret_key=$(jq -r '.SecretAccessKey' <<<"$credentials_json") +session_token=$(jq -r '.SessionToken // empty' <<<"$credentials_json") + +printf 'machine %s login %s password %s\n' \ + "$FUNCTION_HOST" \ + "$access_key" \ + "$secret_key" \ + > "$NETRC_FILE" + +curl_headers=(--header "Content-Type: application/json") +if [ -n "$session_token" ]; then + SECURITY_TOKEN_HEADER_FILE=$(mktemp) + chmod 600 "$SECURITY_TOKEN_HEADER_FILE" + printf 'X-Amz-Security-Token: %s\n' "$session_token" \ + > "$SECURITY_TOKEN_HEADER_FILE" + curl_headers+=(--header "@$SECURITY_TOKEN_HEADER_FILE") +fi + +status_response=$(curl -s --aws-sigv4 "aws:amz:$REGION:lambda" \ + --netrc-file "$NETRC_FILE" \ + "${curl_headers[@]}" \ + --data '{"op":"status"}' \ + "$FUNCTION_URL") + +printf '%s\n' "$status_response" | jq . + +# printf '%s\n' passes the JSON through verbatim — echo can mangle backslash escapes and +# treat a leading '-' as a flag in some shells. +weather_present=$(printf '%s\n' "$status_response" | jq '.sources[] | select(.name == "weather") | .lastValue') +if [ -z "$weather_present" ] || [ "$weather_present" = "null" ]; then + echo "FAIL: no weather source with a lastValue in status response" >&2 + exit 1 +fi + +echo "" +echo "=== Smoke test complete ===" diff --git a/src/agent/status.ts b/src/agent/status.ts new file mode 100644 index 0000000..68272f5 --- /dev/null +++ b/src/agent/status.ts @@ -0,0 +1,126 @@ +import { existsSync, rmSync, writeFileSync } from 'node:fs'; +import type Database from 'better-sqlite3'; +import { openReadOnlyDatabase } from '../db/open.js'; +import type { Store } from '../store/types.js'; + +export interface SourceStatus { + name: string; + lastValue: string | null; + lastFetchedAt: number | null; + lastPostedAt: number | null; +} + +export interface NotificationStatus { + source: string; + value: string; + formattedMessage: string; + postedAt: number; +} + +export interface StatusResult { + snapshotVersion: string | null; + sources: SourceStatus[]; + recentNotifications: NotificationStatus[]; +} + +/** Rows returned per `status` call, across all sources (spec §3.2 step 4). Not + * configurable — the reader is a diagnostic JSON endpoint, not a paginated API (spec §2). */ +const RECENT_NOTIFICATIONS_LIMIT = 10; + +interface ReaderState { + cachedEtag: string | null; + db: Database.Database | undefined; +} + +export interface StatusReader { + getStatus(store: Store, storeKey: string): Promise; +} + +function queryStatus(db: Database.Database, etag: string): StatusResult { + // ORDER BY name keeps the sources list deterministic across SQLite versions and + // vacuuming — without it, SQL does not guarantee row order. + const sources = db + .prepare(`SELECT name, last_value, last_fetched_at, last_posted_at FROM agent_sources ORDER BY name`) + .all() as Array<{ name: string; last_value: string | null; last_fetched_at: number | null; last_posted_at: number | null }>; + + // id DESC is a tie-breaker for notifications that share the same posted_at — without + // it, the LIMIT picks an arbitrary subset and the endpoint output is not stable. + const notifications = db + .prepare( + `SELECT source, value, formatted_message, posted_at FROM agent_notifications + ORDER BY posted_at DESC, id DESC LIMIT ?`, + ) + .all(RECENT_NOTIFICATIONS_LIMIT) as Array<{ + source: string; + value: string; + formatted_message: string; + posted_at: number; + }>; + + return { + snapshotVersion: etag, + sources: sources.map((row) => ({ + name: row.name, + lastValue: row.last_value, + lastFetchedAt: row.last_fetched_at, + lastPostedAt: row.last_posted_at, + })), + recentNotifications: notifications.map((row) => ({ + source: row.source, + value: row.value, + formattedMessage: row.formatted_message, + postedAt: row.posted_at, + })), + }; +} + +/** + * Creates a reader instance holding module-scope hydration state (spec §4.3): the last + * hydrated ETag and an open read-only handle. Call `getStatus` on every invocation; the + * instance itself must be created once per Lambda container (module scope in + * `src/handler.ts`), not per request — recreating it would defeat the version cache. + */ +export function createStatusReader(dbPath: string): StatusReader { + const state: ReaderState = { cachedEtag: null, db: undefined }; + + return { + async getStatus(store: Store, storeKey: string): Promise { + const head = await store.head(storeKey); + + // No snapshot yet — fetch has never run successfully (spec §4.3). Nothing to query. + if (head === null) { + return { snapshotVersion: null, sources: [], recentNotifications: [] }; + } + + const cacheHit = state.cachedEtag === head.etag && state.db !== undefined && existsSync(dbPath); + + if (!cacheHit) { + // `better-sqlite3` keeps a page cache in memory; if the file on disk changes + // underneath an open handle, the cache describes a file that no longer exists — + // silently wrong answers, no error. Close before overwriting (spec §4.3). + if (state.db !== undefined) { + state.db.close(); + state.db = undefined; + } + // `force: true` makes the delete robust against the file disappearing between the + // existsSync check and the rmSync call — `/tmp` is shared with the writer, and + // `/tmp` cleanup can race us too. With `force` the no-op case is harmless. + rmSync(dbPath, { force: true }); + + const object = await store.get(storeKey); + if (object === null) { + // HEAD succeeded but GET raced a delete between the two calls — treat as + // no-snapshot rather than throwing, since the outcome the caller cares about + // (nothing to query) is identical to the head === null branch above. + return { snapshotVersion: null, sources: [], recentNotifications: [] }; + } + + writeFileSync(dbPath, object.body); + state.cachedEtag = object.etag; + state.db = openReadOnlyDatabase(dbPath); + } + + return queryStatus(state.db as Database.Database, state.cachedEtag as string); + }, + }; +} \ No newline at end of file diff --git a/src/db/open.ts b/src/db/open.ts index d81b652..757496f 100644 --- a/src/db/open.ts +++ b/src/db/open.ts @@ -19,3 +19,12 @@ export function openDatabase(path: string): Database.Database { db.pragma('foreign_keys = ON'); return db; } + +/** + * Opens an existing SQLite file read-only. Used by the reader (spec §3.2): the reader's + * IAM grant is GetObject-only (spec §2), so a read-only DB handle matches that intent even + * though `better-sqlite3` itself has no knowledge of the S3 permission model. + */ +export function openReadOnlyDatabase(path: string): Database.Database { + return new Database(path, { readonly: true }); +} diff --git a/src/handler.ts b/src/handler.ts index a247f90..ad2bedb 100644 --- a/src/handler.ts +++ b/src/handler.ts @@ -2,6 +2,7 @@ import { BedrockRuntimeClient } from '@aws-sdk/client-bedrock-runtime'; import { S3Client } from '@aws-sdk/client-s3'; import { runFetch } from './agent/fetch.js'; +import { createStatusReader, type StatusReader } from './agent/status.js'; import { loadConfig } from './config.js'; import { createFetchDiscordPoster } from './discord/poster.js'; import { createBedrockFormatter } from './format/bedrock.js'; @@ -29,6 +30,35 @@ export interface InjectedClients { bedrockClient?: BedrockRuntimeClient; } +/** + * Module-scope, keyed by the reader's local path: the Lambda runtime may reuse the + * container across invocations, so the reader's hydration cache (spec §4.3) must + * survive warm invocations — recreating it per call would re-download the snapshot on + * every request regardless of whether its ETag changed. + */ +const statusReaders = new Map(); + +/** + * Returns a `StatusReader` whose local SQLite file is a sibling of the writer's, not + * the writer's file itself. The writer (`runFetch`) writes to `config.dbPath` on every + * invocation, including the conditional-write failure path where the S3 PutObject is + * rejected with 412 — in that case the local file is still mutated to record the + * `outcome='error'` run row, but S3's ETag is unchanged. If the reader shared the + * writer's path, a subsequent warm `status` call would see an ETag cache hit and + * answer from the still-open reader handle against the writer's mutated bytes. + * Keeping the two paths disjoint means the reader's local copy only changes when the + * reader itself downloads a new snapshot. + */ +function getStatusReader(writerDbPath: string): StatusReader { + const readerDbPath = `${writerDbPath}.reader`; + let reader = statusReaders.get(readerDbPath); + if (reader === undefined) { + reader = createStatusReader(readerDbPath); + statusReaders.set(readerDbPath, reader); + } + return reader; +} + /** * Extracts `op` from the two event shapes the function accepts: * - EventBridge-style: `{ op: 'fetch' }` (op is a top-level field). @@ -74,8 +104,11 @@ export async function runHandler( const config = loadConfig(env); if (op === 'status') { - // PR3 implements the reader op (spec §9 Phase 4). - return { statusCode: 501, body: JSON.stringify({ error: 'status op not yet implemented' }) }; + const s3Client = clients.s3Client ?? new S3Client({ region: config.region, maxAttempts: 3 }); + const store = createS3Store({ client: s3Client, bucket: config.snapshotBucket }); + const reader = getStatusReader(config.dbPath); + const result = await reader.getStatus(store, config.snapshotKey); + return { statusCode: 200, body: JSON.stringify(result) }; } const s3Client = clients.s3Client ?? new S3Client({ region: config.region, maxAttempts: 3 }); diff --git a/tests/db.test.ts b/tests/db.test.ts index 365cac8..4abedef 100644 --- a/tests/db.test.ts +++ b/tests/db.test.ts @@ -5,7 +5,7 @@ import { join } from 'node:path'; import { describe, expect, it } from 'vitest'; import Database from 'better-sqlite3'; import { bootstrap } from '../src/db/bootstrap.js'; -import { openDatabase } from '../src/db/open.js'; +import { openDatabase, openReadOnlyDatabase } from '../src/db/open.js'; import { AGENT_DDL, SOURCE_NAMES } from '../src/db/schema.js'; describe('schema DDL', () => { @@ -108,3 +108,22 @@ describe('openDatabase', () => { rmSync(dir, { recursive: true, force: true }); }); }); + +describe('openReadOnlyDatabase', () => { + it('opens an existing file without allowing writes', () => { + const dir = mkdtempSync(join(tmpdir(), 'agent-test-')); + const path = join(dir, 'memory.db'); + + const writable = openDatabase(path); + bootstrap(writable); + writable.close(); + + const readOnly = openReadOnlyDatabase(path); + expect(() => + readOnly.prepare(`INSERT INTO agent_sources (name) VALUES ('weather')`).run(), + ).toThrow(/readonly/i); + + readOnly.close(); + rmSync(dir, { recursive: true, force: true }); + }); +}); diff --git a/tests/handler.test.ts b/tests/handler.test.ts index f717317..54b2920 100644 --- a/tests/handler.test.ts +++ b/tests/handler.test.ts @@ -1,6 +1,7 @@ // tests/handler.test.ts import { GetObjectCommand, + HeadObjectCommand, PutObjectCommand, S3Client, } from '@aws-sdk/client-s3'; @@ -57,15 +58,34 @@ describe('runHandler', () => { expect(body.outcome).toBe('success'); }); - it('routes op="status" to a 501 stub (PR3 completes this op)', async () => { + it('routes op="status" through the reader and returns 200 with the empty-state shape when no snapshot exists', async () => { + s3.on(GetObjectCommand).rejects({ name: 'NoSuchKey' }); + s3.on(PutObjectCommand).resolves({ ETag: '"v1"' }); + bedrock.on(ConverseCommand).resolves({ + output: { message: { role: 'assistant', content: [{ text: 'Weather update: 72F' }] } }, + stopReason: 'end_turn', + }); + const env = { DISCORD_WEBHOOK_URL: 'https://discord.example/webhook', SNAPSHOT_BUCKET: 'test-bucket', DB_PATH: join(dir, 'memory.db'), + SOURCES: '["weather"]', }; + const clients = { s3Client: s3 as unknown as S3Client, bedrockClient: bedrock as unknown as BedrockRuntimeClient }; - const result = await runHandler({ op: 'status' }, env); - expect(result.statusCode).toBe(501); + // Publish a snapshot via fetch first, then read it via status. The S3 mock is + // stateless across commands, so this test only checks the routing and response shape, + // not that fetch's write is visible to a fresh S3 GET — status.test.ts already covers + // the version-cache hydration logic against a real LocalStore. + await runHandler({ op: 'fetch' }, env, clients, { weather: async () => '72F' }); + + s3.on(HeadObjectCommand).rejects({ name: 'NotFound' }); + const result = await runHandler({ op: 'status' }, env, clients); + + expect(result.statusCode).toBe(200); + const body = JSON.parse(result.body ?? '{}'); + expect(body).toEqual({ snapshotVersion: null, sources: [], recentNotifications: [] }); }); it('returns 400 for an unknown op', async () => { @@ -83,6 +103,7 @@ describe('runHandler', () => { // a hostile Function URL client posting `{op: 42}` or `{op: {name: 'fetch'}}` // would otherwise undermine the typed string contract. Falling through to the // body (or to the 400 path) keeps behaviour predictable. + s3.on(HeadObjectCommand).rejects({ name: 'NotFound' }); const env = { DISCORD_WEBHOOK_URL: 'https://discord.example/webhook', SNAPSHOT_BUCKET: 'test-bucket', @@ -93,7 +114,9 @@ describe('runHandler', () => { { op: 42 as any, body: '{"op":"status"}' }, env, ); - expect(result.statusCode).toBe(501); // status op (parsed from body) + expect(result.statusCode).toBe(200); // status op (parsed from body) + const body = JSON.parse(result.body ?? '{}'); + expect(body).toEqual({ snapshotVersion: null, sources: [], recentNotifications: [] }); }); it('returns 400 when event.op is a non-string and the body is empty', async () => { @@ -108,6 +131,7 @@ describe('runHandler', () => { }); it('parses op from event.body when called via a Function URL invocation', async () => { + s3.on(HeadObjectCommand).rejects({ name: 'NotFound' }); const env = { DISCORD_WEBHOOK_URL: 'https://discord.example/webhook', SNAPSHOT_BUCKET: 'test-bucket', @@ -115,7 +139,9 @@ describe('runHandler', () => { // Lambda Function URLs deliver the HTTP request body as a string under `event.body`. const result = await runHandler({ body: '{"op":"status"}' }, env); - expect(result.statusCode).toBe(501); + expect(result.statusCode).toBe(200); + const body = JSON.parse(result.body ?? '{}'); + expect(body).toEqual({ snapshotVersion: null, sources: [], recentNotifications: [] }); }); it('returns 400 when the Function URL body is not valid JSON', async () => { diff --git a/tests/status.test.ts b/tests/status.test.ts new file mode 100644 index 0000000..5c20408 --- /dev/null +++ b/tests/status.test.ts @@ -0,0 +1,116 @@ +// tests/status.test.ts +import { mkdtempSync, readFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { bootstrap } from '../src/db/bootstrap.js'; +import { openDatabase } from '../src/db/open.js'; +import { createLocalStore } from '../src/store/local.js'; +import { createStatusReader } from '../src/agent/status.js'; + +function setup() { + const dir = mkdtempSync(join(tmpdir(), 'agent-status-test-')); + const dbPath = join(dir, 'memory.db'); + const store = createLocalStore(join(dir, 'store')); + return { dir, dbPath, store }; +} + +async function seedSnapshot(dbPath: string, store: ReturnType) { + const db = openDatabase(dbPath); + bootstrap(db); + db.prepare( + `INSERT INTO agent_sources (name, last_value, last_fetched_at, last_posted_at) + VALUES ('weather', '72F', 1000, 1000)`, + ).run(); + db.prepare( + `INSERT INTO agent_notifications (source, value, formatted_message, posted_at) + VALUES ('weather', '72F', 'Looks like 72F today!', 1000)`, + ).run(); + db.close(); + return store.put('memory.db', readFileSync(dbPath), null); +} + +describe('createStatusReader', () => { + let ctx: ReturnType; + + beforeEach(() => { + ctx = setup(); + }); + + // afterEach rather than per-test cleanup calls: a failing assertion earlier in the + // body would otherwise leak the temp dir, which can bleed into the next test and + // makes failures harder to reproduce. `force: true` keeps this a no-op if the dir was + // already cleaned up by some other path. + afterEach(() => { + rmSync(ctx.dir, { recursive: true, force: true }); + }); + + it('returns the empty-state JSON when no snapshot exists yet, without opening a handle', async () => { + const reader = createStatusReader(ctx.dbPath); + const result = await reader.getStatus(ctx.store, 'memory.db'); + + expect(result).toEqual({ snapshotVersion: null, sources: [], recentNotifications: [] }); + }); + + it('downloads and returns sources + recentNotifications on first call', async () => { + await seedSnapshot(ctx.dbPath, ctx.store); + + // The reader's dbPath (reader-copy.db) does not exist yet, so the cold-start branch + // is the one under test without needing to pre-create or corrupt a file at the path. + const reader = createStatusReader(join(ctx.dir, 'reader-copy.db')); + const result = await reader.getStatus(ctx.store, 'memory.db'); + + expect(result.snapshotVersion).not.toBeNull(); + expect(result.sources).toEqual([ + { name: 'weather', lastValue: '72F', lastFetchedAt: 1000, lastPostedAt: 1000 }, + ]); + expect(result.recentNotifications).toEqual([ + { source: 'weather', value: '72F', formattedMessage: 'Looks like 72F today!', postedAt: 1000 }, + ]); + }); + + it('reuses the cached handle on a second call when the version is unchanged (no re-download)', async () => { + await seedSnapshot(ctx.dbPath, ctx.store); + const readerDbPath = join(ctx.dir, 'reader-copy.db'); + const reader = createStatusReader(readerDbPath); + + await reader.getStatus(ctx.store, 'memory.db'); + + let getCalls = 0; + const countingStore = { + ...ctx.store, + async get(key: string) { + getCalls++; + return ctx.store.get(key); + }, + }; + + const second = await reader.getStatus(countingStore, 'memory.db'); + expect(getCalls).toBe(0); // head-only, no re-download + expect(second.sources).toHaveLength(1); + }); + + it('re-downloads and re-opens when the version changes', async () => { + await seedSnapshot(ctx.dbPath, ctx.store); + const readerDbPath = join(ctx.dir, 'reader-copy.db'); + const reader = createStatusReader(readerDbPath); + + const first = await reader.getStatus(ctx.store, 'memory.db'); + + // A new fetch run changes the snapshot. + const db = openDatabase(ctx.dbPath); + db.prepare( + `UPDATE agent_sources SET last_value = '73F', last_fetched_at = 2000, last_posted_at = 2000 + WHERE name = 'weather'`, + ).run(); + db.close(); + const priorEtag = (await ctx.store.head('memory.db'))?.etag ?? null; + await ctx.store.put('memory.db', readFileSync(ctx.dbPath), priorEtag); + + const second = await reader.getStatus(ctx.store, 'memory.db'); + expect(second.snapshotVersion).not.toBe(first.snapshotVersion); + expect(second.sources).toEqual([ + { name: 'weather', lastValue: '73F', lastFetchedAt: 2000, lastPostedAt: 2000 }, + ]); + }); +}); \ No newline at end of file