Virtual cells bind to warehouse objects; stop the context document lying

[scottypate]

Resolves #16 and #17. Does not close #18 — see "What is not here".

Filed by the MNTN APE team against 0.0.14 while building their first virtual cell. Fixing them surfaced six more document divergences, one deploy defect that made the whole cell class undeployable, and a model problem underneath all of it.

The model change

materialize: never is retired. A virtual export declares bind: <source>, naming an existing warehouse object.

The reasoning: a never transform's SQL was executed by nothing. It ran inside verify's DuckDB session for a few milliseconds and was discarded, so every semantic in it was a promise nobody kept — a derived column was unreachable, a rename was unreachable, and a WHERE was unenforced row filtering, which on a PII surface is silent over-disclosure. A contract must describe a relation something actually produces. Transforms now require materialization.

This was found by building a non-passthrough never transform and following the document's own instructions: verify passed, context published all three columns, and there was no path by which an agent could obtain the derived one. Every never fixture in the tree was SELECT *, which is why it had never been caught.

Document correctness

• served_here derives from the mounted route list, computed once and fed to both the interface digest and the document, so they cannot disagree. Worth noting: the fix proposed in served_here is derived from --no-data, not from whether any route is mounted #17 used discoverable_routes, which deliberately includes bound exports, and would have been a no-op.
• Hosted /context surfaces .cell/source_check.json, gated on digest and profile and fail-closed when the profile is absent, so a staging attestation cannot validate a prod server. The record also now ships in the deploy artifact and folds into content_hash — without that half the fix reached a laptop and not a pod.
• grain_verified is false when any discoverable export declares no grain. verify skips the uniqueness check in that case, so the field was true for a check that never ran.
• The query block is unconditional. It is interface grammar, not mount state. Reverses ADR 0012 §4, amended here.
• A two-doors regression test asserts the portable and hosted documents agree modulo an explicit allowlist, each entry justified and asserted-present before being stripped. It found two undocumented asymmetries on its first run, and later accepted the source_check work with no new entries.
• The /context ETag now covers the startup-fixed observed inputs. Without it a client honoring If-None-Match would 304 straight past the verified_at_source upgrade this PR delivers.

Types and metadata

A bound export's declared types are checked against the warehouse's own types rather than DuckDB's rendering. Wide BigQuery NUMERIC renders as VARCHAR, so verify was rejecting decimal and forcing authors to declare string for a column the warehouse returns as a number — publishing a false type to a machine.

Warehouse column descriptions surface under observed.source_descriptions, scoped to sources an export actually binds. They are a machine fact: outside the interface digest, and outside description_digest, so an upstream comment edit cannot move our release gate. Contract prose stays authored and reviewed in declared.

Only BigQuery populates this today; Postgres and Snowflake run no metadata job, and the absence is explicit rather than silent.

Deploy

A bound cell can deploy a Server — its /context is the integration surface for agent consumers, which a static file cannot authenticate. No init Job is rendered, since datamk run refuses the cell; both share one predicate. schedule: on a bound cell is refused, closing a hole the relax would otherwise have opened. The anonymous-access refusal now names the document itself as the payload.

Behavior changes

• A zero-transform cell refuses run.
• materialize: never is rejected, with a migration error naming both exits and hinting bind: only when the transform is a genuine passthrough.
• The interface digest moves once, for served_here and the query block. No datamk_context bump — nothing is renamed or re-meant.

What is not here

datamk interface import (#18). Its selector depends on the binding model, and shipping a generator before that settled risked mass-producing contracts we would then have had to migrate by hand. #18 stays open.

The thin-interface lint is also deferred: schema: is optional, an empty one makes serve emit SELECT *, and verify then checks nothing. Real, but a new refusal surface with no spec, and it should not land before import makes the honest path cheap.

Verification

624 tests, 0 failures. cargo fmt --check and cargo clippy --all-targets --all-features -- -D warnings clean.

Two review passes ran before this was committed. They found nine issues, all fixed here — including one blocker where the deploy relax was a no-op because the init Job was still rendered unconditionally, and one case where a deleted test had genuinely lost coverage rather than being converted. The BigQuery metadata query is covered by a credential-gated test that skips without DATAMK_TEST_BQ_PROJECT, per the existing convention; it does not run in default CI.

🤖 Generated with Claude Code