feat: per-file metadata — PR/issue context, custom tags, and search#157
Merged
Conversation
Adds the D1 table and file-metadata.ts core module (validation, get/set/ delete, AND-filter search) that later tasks build the metadata CRUD endpoints and search filter on top of.
PUT splits X-Uploads-Meta-* headers: allowlisted keys still go to R2 provenance unchanged, everything else is validated (file-metadata.ts) and stored in D1 via setFileMetadata inside putObject. Invalid keys/values or a >24-key cap breach now reject the upload instead of being silently dropped. Overwriting a key does a full delete-then-set replace of its metadata; deleteObject cascades via deleteFileMetadata; dryRun writes nothing.
splitUploadMetaHeaders pre-filtered empty header values before validation, reproducing the old silent drop for X-Uploads-Meta-<key> with an empty value. Empty (and empty-key) custom entries now flow to validateMetadataEntries so the upload 400s with the typed error; the allowlisted provenance branch keeps its historical lenience.
Adds GET/PATCH /v1/:workspace/files/:key/metadata on top of Task 1's file-metadata.ts (caps, merge semantics) and Task 2's put/delete cascade, so callers can read and update an object's queryable metadata without re-uploading the file.
GET /v1/:workspace/files accepts repeatable meta.<key>=<value> params, ANDed, and switches the listing to the D1 file_metadata index instead of the R2 prefix-list when at least one is present. Combines with prefix and limit. Invalid keys or a repeated same-key param are rejected with a ValidationError. No meta.* params leaves the existing R2 path unchanged.
GET /public/files/:workspace/:key now returns the file's D1 file_metadata map (omitted when empty) and a derived github object built from gh.repo/ gh.kind/gh.number when all three parse validly. Malformed or incomplete gh.* pairs just fall back to raw metadata with no github object. The private-file 401 gate still runs before the metadata fetch, so nothing leaks on private objects.
A client could store a D1 file_metadata row named content-sha256 (via upload headers or the metadata PATCH), shadowing the server-computed integrity hash in R2 provenance. Server-set provenance keys are now a named constant (PROVENANCE_SERVER_KEYS) and validateMetadataEntries — the shared choke point for upload capture, PATCH, and future callers — rejects them with file_metadata_reserved_key. gh.* keys stay writable (system-managed by convention per the design doc).
Task 6: render the "Attached to" GitHub PR/issue row and a generic Metadata section on the /f/ file page, consuming Task 5's github/metadata fields from GET /public/files/:workspace/:key.
- put --meta k=v (repeatable, first-= split) validated client-side and sent as more X-Uploads-Meta-<key> headers alongside provenance - attach writes gh.repo/gh.kind/gh.number/gh.ref automatically from its resolved target; --meta extras merge in, target pairs win on collision - new meta get/set commands and list --meta / find k=v filter alias - client gains getMetadata/patchMetadata/findFiles; cli-args flags now support repeatable string flags via flagValues
…-wins; mirror byte cap
MCP parity for per-file metadata (Task 8). Remote worker's put tool and the local stdio MCP's put/attach tools gain a metadata param; attach keeps auto-injecting gh.* via the shared command-layer helper. New set_metadata and find_files tools mirror the CLI's meta set / find commands.
…ics, filter cap, reserved keys) Consolidated fix wave ahead of the per-file metadata PR: escape SQL LIKE wildcards in findObjectsByMetadata's prefix so `_`/`%` in a prefix match literally; make the REST PUT route pass metadata: undefined (preserve) instead of always full-replacing when a re-PUT carries no custom X-Uploads-Meta-* headers; cap GET list's meta.* filter params at META_MAX_KEYS with a typed error; register the file_metadata_duplicate_filter and file_metadata_too_many_filters error codes; reserve `visibility` as a custom metadata key so it can't shadow the real R2 visibility gate.
Mirrors the local stdio MCP's set_metadata/find_files tools on the remote worker (apps/mcp/src/tools.ts), implemented against the shared api modules (setFileMetadata/findObjectsByMetadata) so both MCP surfaces expose the same metadata CRUD/search behavior called out in the design doc.
Adds the re-upload metadata contract (--meta replaces, omitted preserves; attach always replaces since it always sends gh.*) to the CLI's put/attach --help text and the uploads-cli skill doc, per fix 7 of the metadata final-review pass.
|
Important Review skippedAuto reviews are limited based on label configuration. 🏷️ Required labels (at least one) (2)
🚫 Excluded labels (none allowed) (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis change adds queryable per-file metadata across the API, uploads client, CLI, MCP tools, and public file views. It introduces D1 persistence, validation and filtering, GitHub metadata injection and backfill, metadata management endpoints, search commands, derived public GitHub context, and comprehensive tests and documentation. ChangesQueryable file metadata
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
uploads-auth | a328f3c | Commit Preview URL Branch Preview URL |
Jul 14 2026, 12:51 AM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
uploads-api | a328f3c | Commit Preview URL Branch Preview URL |
Jul 14 2026, 12:51 AM |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
uploads-web | c0659af | Commit Preview URL Branch Preview URL |
Jul 14 2026, 12:05 AM |
…le-metadata - files.ts: keep metadata imports + adopt objectPublicUrls; meta-filtered listings now return embedUrl like the plain list path - SKILL.md: keep both post-embed paragraphs (url durability + attach metadata) - apps/mcp/tsconfig: include ../api/src/env.d.ts so Env augmentations (EMBED_PUBLIC_BASE_URL, #154) type-check from the shared api source - routes-files tests: expect embedUrl on meta-filtered items
Consolidate three copy-pasted node:sqlite fake-D1 harnesses (file-metadata-sqlite/galleries-sqlite) into apps/api/test/helpers/sqlite-d1.ts parameterized by migration path(s) and optional pragmas, and consolidate the three near-identical hand-rolled file_metadata D1 mocks in routes-files.test.ts, routes-public-files.test.ts, and usage-fake-d1.ts into a shared FileMetadataTable helper that each suite's fake D1 tries first, falling back to its own auth_tokens/workspace_usage logic. No behavior change.
…ndependent awaits Promote optStringArray/METADATA_DESCRIPTION/metadataProp from duplicated copies in apps/mcp/src/tools.ts and packages/uploads/src/mcp/tools.ts into packages/uploads/src/mcp/args.ts (re-exported from server.ts), extract a shared validateMetadataFilters (count cap + key format) in apps/api/src/file-metadata.ts used by both the REST list endpoint and the MCP find_files tool, run the independent storageConfig/findObjectsByMetadata awaits in Promise.all on both surfaces, and make the CLI's validateMetaMap validate entries directly instead of round-tripping through parseMetaFlags's "k=v" string reconstruction. Also adds "visibility" to the CLI's reserved metadata keys to match the server, with a test.
…oad path putObject's full-replace path called deleteFileMetadata then setFileMetadata, which re-reads the now-guaranteed-empty map and re-validates before writing. Add replaceFileMetadata (one validateMetadataEntries call, then a single db.batch delete-all + upserts) and use it from putObject, making delete+set atomic in one batch as a side effect. setFileMetadata's merge/PATCH path is unchanged. Adds sqlite-backed tests for replace/clear/cap-rejection/atomicity.
Member
Author
|
CodeRabbit (@coderabbitai) review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
apps/api/src/files-core.ts (1)
194-206: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winPartial write can skip usage accounting at
apps/api/src/files-core.ts:194-206.store.uploadcan succeed,replaceFileMetadatacan fail, andrecordUsageSafewill never run, leaving the object stored but the ledger under-counted and metadata rows missing. Consider moving usage accounting before the metadata batch or adding compensation on failure.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/src/files-core.ts` around lines 194 - 206, The upload path can persist the object but skip usage accounting when replaceFileMetadata fails. Update the flow around replaceFileMetadata and recordUsageSafe so usage accounting runs even if metadata replacement fails, while preserving the metadata update and existing usage values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/web/src/pages/f/`[workspace]/[...key].astro:
- Around line 43-49: The metadataEntries rendering in
apps/web/src/pages/f/[workspace]/[...key].astro lines 43-49 requires no code
change; retain the existing behavior. Update the “Custom metadata & search”
section in skills/uploads-cli/SKILL.md lines 210-235 to explicitly warn that
non-`gh.*` values supplied through `--meta` or MCP metadata are displayed on the
object’s public file page and must not contain sensitive internal notes, IDs, or
paths.
In `@packages/uploads/src/commands.ts`:
- Around line 916-930: Update runFindFiles to detect when --cursor is supplied
and reject it explicitly before calling ctx.client.findFiles, returning the
command’s established error status and message format. Preserve the existing
metadata filtering, prefix, limit, output handling, and successful return
behavior when --cursor is absent.
In `@packages/uploads/src/mcp/args.ts`:
- Around line 37-40: Update the result accumulator in the argument-validation
function to preserve an own "__proto__" key instead of invoking
Object.prototype’s setter, ensuring invalid metadata reaches validation and is
rejected. Add a regression test covering {"__proto__":"x"} and assert it fails
metadata-key validation.
In `@packages/uploads/src/mcp/tools.ts`:
- Around line 581-583: Update the metadata handling in the attachment flow to
merge optStringRecord(args, "metadata") with ghMetadataFromTarget(target) first,
then validate the complete metadata object with validateMetaMap before any file
reads; remove the validation that only checks metaExtras.
---
Outside diff comments:
In `@apps/api/src/files-core.ts`:
- Around line 194-206: The upload path can persist the object but skip usage
accounting when replaceFileMetadata fails. Update the flow around
replaceFileMetadata and recordUsageSafe so usage accounting runs even if
metadata replacement fails, while preserving the metadata update and existing
usage values.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: e76c5314-bff4-47ae-aa9c-8483d12115b8
📒 Files selected for processing (50)
.changeset/file-metadata.mdapps/api/migrations/20260713210559_file_metadata.sqlapps/api/package.jsonapps/api/scripts/backfill-gh-metadata.d.mtsapps/api/scripts/backfill-gh-metadata.mjsapps/api/src/file-metadata.tsapps/api/src/files-core.tsapps/api/src/provenance.tsapps/api/src/routes/files.tsapps/api/src/routes/public-files.tsapps/api/test/backfill-gh-metadata.test.tsapps/api/test/file-metadata-sqlite.test.tsapps/api/test/file-metadata.test.tsapps/api/test/galleries-sqlite.test.tsapps/api/test/helpers/fake-file-metadata-table.tsapps/api/test/helpers/sqlite-d1.tsapps/api/test/routes-files.test.tsapps/api/test/routes-galleries.test.tsapps/api/test/routes-key-policy.test.tsapps/api/test/routes-public-files.test.tsapps/api/test/usage-fake-d1.tsapps/mcp/src/tools.tsapps/mcp/test/mcp.test.tsapps/mcp/tsconfig.jsonapps/web/src/lib/public-file.test.tsapps/web/src/lib/public-file.tsapps/web/src/pages/f/[workspace]/[...key].astrodocs/ops.mdpackages/errors/src/codes.tspackages/uploads/README.mdpackages/uploads/src/cli-args.tspackages/uploads/src/cli.tspackages/uploads/src/client.tspackages/uploads/src/commands.tspackages/uploads/src/github.tspackages/uploads/src/index.tspackages/uploads/src/mcp/args.tspackages/uploads/src/mcp/server.tspackages/uploads/src/mcp/tools.tspackages/uploads/src/metadata.tspackages/uploads/test/cli-args.test.tspackages/uploads/test/client-metadata.test.tspackages/uploads/test/commands-attach.test.tspackages/uploads/test/commands-find.test.tspackages/uploads/test/commands-list.test.tspackages/uploads/test/commands-meta.test.tspackages/uploads/test/commands-put.test.tspackages/uploads/test/mcp.test.tspackages/uploads/test/metadata.test.tsskills/uploads-cli/SKILL.md
- SKILL.md: warn that non-gh.* --meta/MCP metadata values render publicly on the /f/ file page. - CLI: reject --cursor with metadata search (list --meta / find), which previously accepted and silently ignored it. - mcp/args.ts optStringRecord: use a null-prototype accumulator so a __proto__ metadata key can't be silently dropped by the inherited setter. - mcp/tools.ts attach + commands.ts runAttach: validate the merged metadata (extras + gh.* pairs) against the key/byte caps, not just the extras alone, so the merge can't silently exceed the cap server-side. - files-core.ts putObject: record usage before writing custom metadata, so a failing metadata batch no longer leaves the ledger under-counted.
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In plain terms
Files now carry searchable key-value metadata.
uploads attachrecords which PR or issue a screenshot belongs to, you can tag any upload with your own pairs (app,url,device,resolution, …), the/f/file pages show that context (including an "Attached to owner/repo#123" link), and you can search by it — "show me every screenshot attached to PR 123" is nowuploads find gh.number=123.What it does
file_metadatatable (one row per pair, indexed for equality search) — the queryable-tag tier, modeled on S3 object tags / Azure blob index tags. R2 custom metadata (provenance, visibility) is untouched.X-Uploads-Meta-<key>header convention: non-provenance keys become metadata rows instead of being silently dropped; invalid keys/values now reject with typed errors.^[a-z][a-z0-9._-]{0,63}$, values ≤512 printable chars, ≤24 keys per file/request, ≤8 KB total. Reserved keys:content-sha256,visibility(would shadow server-controlled values).GET/PATCH /v1/:ws/files/:key/metadata(merge semantics,deletelist) andGET /v1/:ws/files?meta.<k>=<v>(repeatable, ANDed, combines withprefix/limit).uploads attachwritesgh.repo/gh.kind/gh.number/gh.refautomatically (canonical lowercase). New CLI:put/attach --meta k=v,meta get|set,ls --meta, anduploads find k=v ….metadataonput, plusset_metadata/find_filestools./f/pages render the attachment link + a metadata list (public files only; private files still 401 before metadata is fetched).gh.*for existinggh/<owner>/<repo>/…keys.What it is not
/v1/signwork).--metaheaders replaces the file's whole metadata set; without them the existing metadata is preserved (same as the MCP tools). Useuploads meta setto edit individual keys.How to try it
uploads put ./shot.png --meta app=web --meta device=mobile uploads attach ./after.png --pr 123 uploads find gh.number=123 uploads meta set screenshots/shot.png page=/settingsTechnical notes
Single validation choke point (
apps/api/src/file-metadata.ts) covers upload capture, PATCH, and both MCP workers. LIKE prefix search escapes%/_with anESCAPEclause. Metadata deletion cascades from object deletes.meta.*filter count capped at 24. Backfill:node apps/api/scripts/backfill-gh-metadata.mjs --dry-run(see docs/ops.md).Follow-ups deliberately deferred:
set_metadata's bare-Error 404 in the remote worker, client-side reserved-key list omitsvisibility(server rejects with a clear 400),getFileMetadatadoesn't wrap raw D1 errors, visibility annotation absent from metadata-filtered listings (documented caveat).Test plan
pnpm --filter @uploads/api test(329 passing, incl. node:sqlite-backed metadata suite)pnpm --filter @uploads/mcp test(32) ·pnpm --filter @buildinternet/uploads test(261) ·pnpm --filter @uploads/errors test(15) · web (63)pnpm checkandpnpm typecheckclean--dry-runfirst) after mergeSummary by CodeRabbit
New Features
--meta, metadata retrieval and updates, and metadata-basedfindandlistfiltering.Documentation