fix(drivers): a declared field written as an explicit undefined is indistinguishable from one never written (#12641)

* fix(drivers): a declared field written as an explicit `undefined` is indistinguishable from one never written

A row has two states to say about a field — the key is absent, or the key
holds a value. An own key holding `undefined` is neither, so every consumer
had to invent a reading of it, and measured on `origin/main` they disagreed:
CEL (`has(record.f)` is `false`) and `materializeDeclaredFields` read it as
absent, a bare `f in row` reads it as present.

Both JS-backed drivers were measured separately and did NOT match.
`driver-memory` emitted the own key from `create` and `find` alike, while its
own projection path and its own matcher already read the shape as absent.
`driver-mongodb` split: `create()` returned an own key holding `undefined`
while BSON stored `null` for the same field (MongoClient default
`ignoreUndefined: false`, no override here), so `find()` answered with a
value — one write, two answers, from one driver.

Both drivers now drop own keys holding `undefined` on the way into storage.
`null` is untouched and stays a value. No accept set moves and no exported
name changes. On `driver-mongodb` the scope is the insert doors and the
returned values; `$set`-shaped patches are left alone, and on `driver-memory`
the normalisation is post-merge, so neither driver answers "what does a patch
carrying `undefined` mean" — that is a storage-contract question this repair
does not reopen.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o

* wip(drivers): recovered uncommitted own-key-undefined pins from a killed seat

NOT REVIEWED, NOT VERIFIED. The resumed dispatch was killed before it
committed these; recovered so they survive the container.

Both files are the card's two outstanding obligations by name:
memory-own-key-undefined.test.ts and mongodb-own-key-undefined.test.ts —
the second being the driver-mongodb measurement the card asked for
explicitly and told the dev not to assume matched driver-memory.

No gate was run against them, no ablation exists, and the fail-OPEN
consumer sweep the card stars as a separate, more urgent card is not
evidenced anywhere in this branch.

* record: correcting e87cf37 — the seat it rescued was alive, and its verification exists

e87cf37 was written by a worktree-rescue actor at 2026-08-27T02:23:47Z. Its
content is correct and is kept as-is; its MESSAGE is not, and it cannot be
amended from this seat without a force-push, which the dev contract forbids.
So the correction is additive, here.

That message says the dispatch "was killed before it committed these", and
that "No gate was run against them, no ablation exists, and the fail-OPEN
consumer sweep ... is not evidenced anywhere in this branch."

The seat was not killed. It was mid-task, in the same session that had already
run all three of those things against exactly the content e87cf37 committed,
and it continued from there to finish this branch. On that content:

  * 23 gate families derived by scripts/pm/dispatch-gates.mjs --repo, all green,
    each exit code captured before any pipe and each quoted from the gate's own
    verdict line;
  * two ablation legs with the direction AND the exact failure count written
    down first — 8 of 10 in driver-memory, 4 of 6 in driver-mongodb, both as
    predicted, mutation proved on disk with anchored grep -cF counts, restored
    under `trap ... EXIT INT TERM` with an empty `git diff` verified;
  * the fail-OPEN consumer sweep, over 110 non-test presence tests under
    packages/, with a synthetic positive control that fires in the same scan.
    No fail-open consumer found.

Recorded because the branch is squash-merged and `git log` otherwise ends on a
commit stating that this branch's verification does not exist.

The mechanism defect is filed as #12637, a sub-issue
of #12627 (which proposes the rescue step that has no liveness precondition).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01W6HFzyH98W1YaQXhJUJt6o

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-warren

.changeset/driver-own-key-undefined-normalisation.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/driver-memory": minor
+"@objectstack/driver-mongodb": minor
+---
+
+fix(drivers): a declared field written as an explicit `undefined` is indistinguishable from one never written (#9276)
+
+A row has exactly two states to say about a field, each with a defined meaning:
+**the key is absent** ("no value was ever written") or **the key holds a
+value**. An own key holding `undefined` is neither. Only a JS-backed driver can
+emit it — a SQL NULL arrives as `null`, which is a value — and every consumer
+downstream had to invent a reading of it. Measured on `origin/main`, they did
+not agree: `has(record.f)` on the real `@objectstack/formula` CEL engine reads
+it as ABSENT, `materializeDeclaredFields` reads it as ABSENT by documented
+design, and a bare `f in row` reads it as PRESENT.
+
+Both JS-backed drivers were measured separately, and they did **not** match:
+
+- **`driver-memory`** preserved the own key holding `undefined` through
+  `create` and handed it back from `find`. Its own projection path and its own
+  matcher already read the shape as absent (`projectFields` skips `undefined`
+  values, `{f: {$exists: true}}` excluded it, `{f: {$null: true}}` included it)
+  — so the returned row was the only surface in the driver still claiming the
+  key was present, and the same stored row answered `'f' in row` differently
+  depending on whether a projection was requested.
+- **`driver-mongodb`** SPLIT. `create()` returns the object it built in
+  process, so the field came back as an own key holding `undefined`; but the
+  MongoClient default is `ignoreUndefined: false` and this driver sets no
+  override, so BSON stored `null` for that same field and a subsequent `find()`
+  answered `null` — a value. One write, two answers, from one driver.
+
+Both drivers now drop own keys holding `undefined` on the way into storage, so
+a declared field written as `undefined` and one never written are the same row:
+deep-equal, same own keys, same answer to every presence test. `null` is
+untouched and stays a value.
+
+Fixed at the producer rather than at each consumer: converging one consumer
+resolves one seam, but the next consumer that reasons about key presence
+re-acquires the problem.
+
+**Behaviour that changes, precisely.** What these two packages RETURN for one
+input class, and what `driver-mongodb` STORES for it. A caller passing an
+explicitly-`undefined` property to `create`/`bulkCreate`/`update`/`updateMany`
+(or seeding `initialData`) no longer sees that key in the returned row, and no
+`null` is written for it in MongoDB. `undefined` does not survive JSON, so this
+shape cannot arrive over the wire — reaching it requires in-process code.
+
+**What does NOT change.** No accept set moves: no schema, refine, validator or
+public type is touched, nothing that parsed before is refused now, and no
+exported name is added, removed or moved. Filter results are unchanged in both
+drivers — measured identical before and after for `$null` / `$exists` /
+equality on `driver-memory`, and on `driver-mongodb` `$null: true` lowers to
+`$eq: null` and `$null: false` to `$ne: null`, which MongoDB matches
+identically against a missing field and a stored `null`.
+
+Scope on `driver-mongodb` is the INSERT doors and the values returned.
+`$set`-shaped patches are deliberately untouched: changing them would answer
+"what does a patch carrying `undefined` mean — clear the field, or leave the
+prior value standing" which is a storage-contract question, not this repair's
+to settle. On `driver-memory` the normalisation is applied POST-merge for the
+same reason — it keeps today's answer (every measured consumer read the merged
+own-key-`undefined` as "absent", and the row now says absent outright) rather
+than silently turning such a patch into a no-op.

packages/drivers/driver-memory/src/memory-driver.ts

@@ -95,6 +95,65 @@ export interface InMemoryDriverConfig {
   };
 }
 
+/**
+ * Drop every own key whose value is `undefined` (#9276).
+ *
+ * ## The rule, and why the driver owes it
+ *
+ * A row has exactly two states to say about a field, and each has a defined
+ * meaning: **the key is absent** ("no value was ever written") or **the key
+ * holds a value**. An own key holding `undefined` is NEITHER, so every consumer
+ * downstream has to pick a reading of it, and measured on `origin/main` they do
+ * not agree — `has(record.f)` on the real `@objectstack/formula` CEL engine and
+ * `materializeDeclaredFields` both read it as ABSENT, while a bare `f in row`
+ * reads it as PRESENT. That disagreement is the whole cost: a third state that
+ * nothing declares, that no consumer can resolve locally, and that only a
+ * JS-backed driver can even emit (a SQL NULL arrives as `null`, which is a
+ * value).
+ *
+ * This driver ALREADY holds "a value of `undefined` means the key is not
+ * emitted" in two of its own places, which is why normalising here is
+ * convergence rather than a new rule:
+ *
+ *  - `projectFields` skips `undefined` values, so the same stored row answered
+ *    `'status' in row === false` under a projection and `true` without one;
+ *  - the matcher reads it as absent — measured, `{ status: { $exists: true } }`
+ *    excludes it and `{ status: { $null: true } }` includes it, exactly as for
+ *    a row that never carried the key at all.
+ *
+ * So the returned row was the only surface still claiming the key was present.
+ *
+ * ## Where it is applied, and what that preserves
+ *
+ * On the way INTO the backing table (see {@link InMemoryDriver.toStoredRecord}
+ * and the `initialData` seeding door), which is post-merge on the update path.
+ * That placement is load-bearing: `update(id, { f: undefined })` today merges
+ * an own key holding `undefined` over the stored value, and every measured
+ * consumer reads the result as "the field is absent". Dropping the key AFTER
+ * the merge keeps that reading byte for byte; dropping it BEFORE would make the
+ * same call a no-op that leaves the prior value standing, which is a different
+ * answer to "what does a patch carrying `undefined` mean" — a storage-contract
+ * question this normalisation deliberately does not reopen.
+ *
+ * Returns the input unchanged (same reference) when there is nothing to drop,
+ * so the common case allocates nothing — the same convention
+ * {@link InMemoryDriver.toStorageForms} follows.
+ *
+ * `@objectstack/driver-mongodb` carries a structural twin of this function on
+ * its insert doors, for the same reason its `toStorageForms` is a twin rather
+ * than an import: the two driver packages share no code. This doc comment is
+ * the canonical statement of the rule; that copy defers to it.
+ */
+function withoutUndefinedOwnKeys<T extends Record<string, any>>(record: T): T {
+  let out: Record<string, any> | undefined;
+  for (const key of Object.keys(record)) {
+    if (record[key] !== undefined) continue;
+    out ??= { ...record };
+    delete out[key];
+  }
+  return (out as T) ?? record;
+}
+
 /**
  * Snapshot for in-memory transactions.
  */
@@ -252,7 +311,7 @@ export class InMemoryDriver implements IDataDriver {
         const table = this.getTable(objectName);
         for (const record of records) {
           const id = (record as any).id || this.generateId(objectName);
-          table.push({ ...record, id });
+          table.push(withoutUndefinedOwnKeys({ ...record, id }));
         }
       }
       this.logger.info('InMemory Database Connected with initial data', {
@@ -399,7 +458,7 @@ export class InMemoryDriver implements IDataDriver {
 
     const table = this.getTable(object);
 
-    const newRecord = this.toStorageForms(object, {
+    const newRecord = this.toStoredRecord(object, {
       id: data.id || this.generateId(object),
       ...data,
       created_at: data.created_at || new Date().toISOString(),
@@ -426,7 +485,7 @@ export class InMemoryDriver implements IDataDriver {
       return null;
     }
 
-    const updatedRecord = this.toStorageForms(object, {
+    const updatedRecord = this.toStoredRecord(object, {
       ...table[index],
       ...data,
       id: table[index].id, // Preserve original ID
@@ -525,7 +584,7 @@ export class InMemoryDriver implements IDataDriver {
       for (const record of targetRecords) {
           const index = table.findIndex(r => r.id === record.id);
           if (index !== -1) {
-              const updated = this.toStorageForms(object, {
+              const updated = this.toStoredRecord(object, {
                   ...table[index],
                   ...data,
                   updated_at: new Date().toISOString()
@@ -1467,6 +1526,18 @@ export class InMemoryDriver implements IDataDriver {
     return new RegExp(this.escapeRegex(value as string));
   }
 
+  /**
+   * The form a record takes in the backing table: no own key holding
+   * `undefined`, then every declared temporal field in its storage form.
+   *
+   * Every write door goes through here rather than through
+   * {@link toStorageForms} directly, so the two normalisations cannot drift
+   * apart door by door.
+   */
+  private toStoredRecord<T extends Record<string, any>>(object: string, record: T): T {
+    return this.toStorageForms(object, withoutUndefinedOwnKeys(record));
+  }
+
   /**
    * Put every declared temporal field of a record into its storage form — the
    * write half of the convention the filter path reads against. Returns the