fix(spec): withdraw the ADR-0087 field-required-notnull-explicit conversion — required: true stops stamping storage.notNull

.changeset/adr-0113-withdraw-required-notnull-conversion.md

@@ -0,0 +1,23 @@
+---
+"@objectstack/spec": patch
+"@objectstack/driver-sql": patch
+---
+
+fix(spec): withdraw the `field-required-notnull-explicit` ADR-0087 conversion — `required: true` no longer stamps `storage.notNull: true` on anybody's fields (#16693)
+
+ADR-0113 split the pre-17 `required` tri-binding on purpose: `required` is the **write-time contract** and is NOT a column constraint, and `storage.notNull` alone binds the physical column (`sql-driver.ts#createColumn` has keyed off it alone since that ADR's P0). The `field-required-notnull-explicit` conversion asserted exactly the implication the ADR abolished — it added `storage: { notNull: true }` to every field it found `required: true` on — so it is removed from the conversion registry and from protocol 17's ADR-0087 ledger entry.
+
+**Who this was reaching, and why it was not confined to old artifacts.** The entry carried `retiredFromLoadPath: true` and a docblock stating that "only `os migrate meta --from <16 or lower>` may apply it". That was not true of this tree. The artifact-ingestion door replays the whole chain with `includeRetired: true` (`applyArtifactForwardConversions`, `@objectstack/metadata-core`) and keys the replay off the artifact's declared `engines.protocol` **floor**, not its age — so any artifact declaring `^17.0.0`, which is the range `create-objectstack` stamps, was converted at boot. Measured on this tree at that seam: an artifact declaring `^17.0.0` on a 17.3.0 runtime came back from the door with `storage.notNull: true` written onto a field its author wrote as nullable-and-write-gated, and the boot logged `converted N site(s) forward` with a remedy sentence telling the author to write the same tightening into the source. On a populated database that instruction is a `tighten_not_null` / `severity: error` / `category: destructive` migration — prescribed as the remedy for a deprecation notice.
+
+**What moves for consumers.**
+
+- `applyConversions(stack, { includeRetired: true })` — the artifact-ingestion door and `os migrate meta` — no longer emits or applies this rewrite. The default load posture (`includeRetired: false`) is unchanged: the conversion was already skipped there.
+- `os migrate meta --from 16` no longer lists it, and a `required: true` field crosses 16 → 17 carrying its write contract and nothing else.
+- Boot no longer warns about it, so an artifact whose only conversion was this one now boots with that warning gone.
+- Nothing is authored differently and nothing is refused that was accepted before. `required` and `storage.notNull` both remain authorable and both keep their ADR-0113 meanings. A column is NOT NULL because its author wrote `storage: { notNull: true }`, and for no other reason.
+
+**No migration is owed to anyone** (maintainer ruling, 2026-09-08, decision batch #85, option A). Genuinely pre-ADR-0113 artifacts are not measured to exist, existing columns are left exactly as they are, and an app that wants NOT NULL columns declares `storage.notNull` deliberately — which is what the app that reported this had already done.
+
+The protocol-17 ledger entry and the generated upgrade guide now say this in the other direction too, and the falsified sentence in `sql-driver.ts` — "sources authored before protocol 17 carry `storage.notNull` explicitly via the `field-required-notnull-explicit` conversion, so their columns come out exactly as they always did" — is corrected where it stood.
+
+Two sentences in `@objectstack/driver-sql` that this withdrawal falsifies are corrected with it, and no drift behaviour changes. The `relax_not_null` finding — raised when a column is NOT NULL and the metadata declares no `storage` constraint — used to prescribe "(pre-protocol-17 sources: `os migrate meta` stamps it for every previously-required field)"; it now says the constraint has to be declared by its author, because nothing supplies it any more. The comment beside it, which closed with "`os migrate meta` ratifies it whenever the source is next migrated", says so too. The deliberate SILENCE for a `required: true` field whose column is already NOT NULL is unchanged — this corrects the sentences, never the finding.

docs/protocol-upgrade-guide.md

@@ -136,7 +136,7 @@ The same close-out retires the four inert tool authoring keys (`category`, `perm
 
 The AppSchema sheds its seven dead authoring keys (2026-06 liveness audit, #4001 app step): `version` (apps are versioned by manifest.version), `aria`, `objects`/`apis` (the self-described "config file convenience" — nothing read them; the chatbot derives an app's objects from its nav items), `sharing`/`embed` (a declared-but-unenforced public surface — the only live path is FormView.sharing; ADR-0049), and `mobileNavigation` (fully unimplemented). Pure lossless deletes — none ever had a runtime effect; each key is tombstoned with its prescription.
 
-ADR-0113 splits the `required` tri-binding: post-17, `required` is ONLY the write-time contract (insert must provide; update may not null out; legacy null rows rest), and the physical NOT NULL is the explicit `storage.notNull`. The `field-required-notnull-explicit` conversion preserves every pre-17 source verbatim-in-meaning by stamping `storage.notNull: true` onto each required field — under the old semantics that column WAS created NOT NULL, so the rewrite writes down what the text already meant. Migration-chain-only (retired from the load path): this is a default flip, not a rename, and a loader that auto-applied it would stamp the constraint onto 17-authored sources that deliberately omit it.
+ADR-0113 splits the `required` tri-binding: post-17, `required` is ONLY the write-time contract (insert must provide; update may not null out; legacy null rows rest), and the physical NOT NULL is the explicit `storage.notNull`. ⚠️ NOTHING converts the column half for you, and nothing tightens a column you already have. A `field-required-notnull-explicit` conversion did stamp `storage.notNull: true` onto every `required: true` field; it was WITHDRAWN (maintainer ruling 2026-09-08), because stamping the constraint wherever `required: true` appears is exactly the implication the ADR abolished — and because the artifact-ingestion door replays retired conversions, so the LOADER applied it to 17-authored sources that deliberately omit it and then told their authors to write the same tightening into the source, which on a populated database is a destructive `tighten_not_null` migration prescribed as the remedy for a deprecation notice. Post-17 a column is NOT NULL because its author wrote `storage: { notNull: true }`, and for no other reason. If you are upgrading a pre-17 source whose columns ARE NOT NULL and you want them to stay that way, add `storage: { notNull: true }` to those fields yourself — deliberately, and knowing that doing it to a field whose column is currently nullable is a destructive migration with a backfill ceremony.
 
 On the wire contract it also retires the `/analytics/query` request ENVELOPE (#3878): `AnalyticsQueryRequestSchema` used to describe `{ cube, query: {...}, format }` — the dialect of the retired degraded analytics shim (#3891) that the real engine never understood (an envelope body inferred a column-less cube and died as an SQL syntax error). The canonical request body is now the BARE AnalyticsQuery — `cube` + `measures` at the top level — which is what every real caller already sends; the schema tombstones `query`/`format`, and the dispatcher entry validates bodies and answers 400 with the prescription. No stored metadata carries this shape (it was HTTP-only), so the change is two semantic TODOs for API callers rather than a stack conversion.
 
@@ -261,7 +261,6 @@ Finally it removes the 'pdf' member of `view.exportOptions` formats (#8010, main
 | `tool-inert-authoring-keys-removed` | `tool.category / tool.permissions / tool.active / tool.builtIn` | tool keys 'category'/'permissions'/'active'/'builtIn' removed (#3896 close-out — authorable and inert; permissions gated nothing, active:false withdrew nothing) | retired — `migrate meta` only |
 | `app-dead-authoring-keys-removed` | `app.version / app.aria / app.objects / app.apis / app.sharing / app.embed / app.mobileNavigation / app.contextSelectors.includeAll / app.contextSelectors.placement / app.homePageId / app.areas.order` | app keys 'version'/'aria'/'objects'/'apis'/'sharing'/'embed'/'mobileNavigation'/'homePageId' plus contextSelectors 'includeAll'/'placement' and areas 'order' removed (liveness audits #4001, #4509, #4667 — unread or wrongly encoded; sharing/embed declared a public surface no route enforced, mobileNavigation was fully unimplemented, includeAll was deliberately disobeyed because an 'All' row would clear a mandatory scope, homePageId WAS read by objectui's console before v17 but encoded the landing page as an ID cross-reference that silently fell back when it dangled — the landing page is the first nav item (premise corrected in #4709; the retirement stands), and no renderer ever sorted areas) | retired — `migrate meta` only |
 | `app-area-fail-open-gates-removed` | `app.areas.visible / app.areas.requiredPermissions` | navigation-area keys 'visible'/'requiredPermissions' removed (#4651, ADR-0049 — FAIL-OPEN access gates: no layer ever read them, so a 'hidden' or permission-gated area was served and rendered to every user, while the identically named keys on a navigation ITEM and on the APP are enforced; gate the items inside the area, or gate the app) | retired — `migrate meta` only |
-| `field-required-notnull-explicit` | `object.fields.*.required / object.fields.*.storage.notNull` | required fields gain explicit 'storage.notNull: true' (ADR-0113 — pre-17 'required' implied the column constraint; post-17 it is only the write contract) | retired — `migrate meta` only |
 | `action-inert-keys-removed` | `action.shortcut / action.bulkEnabled` | action keys 'shortcut'/'bulkEnabled' removed (#3896 close-out — no keydown path dispatches shortcuts; the multi-select toolbar reads the view's bulkActions) | retired — `migrate meta` only |
 | `flow-inert-keys-removed` | `flow.active / flow.template / flow.nodes[].outputSchema / flow.errorHandling.fallbackNodeId` | flow keys 'active'/'template', node 'outputSchema' and errorHandling 'fallbackNodeId' removed (#3896 close-out — active:false never stopped a flow; status is the enforced lifecycle) | retired — `migrate meta` only |
 | `view-inert-keys-removed` | `view.list.responsive / view.list.performance / view.form.defaultSort / view.form.aria` | view keys removed (#3896 close-out): list 'responsive'/'performance', form 'defaultSort'/'aria' — no renderer read them (list aria/data and form data stay live) | retired — `migrate meta` only |

packages/cli/test/migrate-meta.e2e.test.ts

@@ -34,8 +34,7 @@ const TSX = resolve(HERE, '../../../node_modules/.bin/tsx');
 /**
  * A stack authored against protocol 16: every line marked `// 16:` is a shape
  * the v17 chain must rewrite, spanning each conversion family — renames
- * (action execute→target, sharing full→edit), the required→storage.notNull
- * explicitization, and the #3896 close-out removals (rls.priority, the four
+ * (action execute→target, sharing full→edit) and the #3896 close-out removals (rls.priority, the four
  * tool keys, flow active/template/outputSchema/fallbackNodeId, view/dashboard
  * inert keys, agent.knowledge, skill.triggerPhrases).
  */
@@ -48,7 +47,7 @@ export default {
     name: 'e2e_ticket',
     label: 'Ticket',
     fields: {
-      title: { type: 'text', label: 'Title', required: true },      // 16: required implied NOT NULL
+      title: { type: 'text', label: 'Title', required: true },      // 16: required implied NOT NULL — v17 does NOT write that down (#16693)
       notes: { type: 'textarea', label: 'Notes' },
     },
   }],
@@ -147,7 +146,6 @@ const EXPECTED_CONVERSIONS = [
   'skill-trigger-phrases-removed',
   'tool-inert-authoring-keys-removed',
   'permission-rls-priority-removed',
-  'field-required-notnull-explicit',
   'sharing-rule-access-level-full-to-edit',
 ];
 
@@ -187,6 +185,21 @@ describe('os migrate meta --from 16 (e2e over the real CLI)', () => {
     }
   });
 
+  /**
+   * ⛔ The negative half of the assertion above (#16693). A conversion that
+   * stamps `storage: { notNull: true }` onto every `required: true` field
+   * asserts the implication ADR-0113 abolished, so `migrate meta` must
+   * attribute NOTHING to it — on a source (`title`) that would have triggered
+   * it, which is what keeps this from passing vacuously.
+   */
+  it('attributes nothing to the withdrawn required→storage.notNull conversion', () => {
+    const ids = new Set(out.parsed.applied.map((a: any) => a.conversionId));
+    expect(ids.has('field-required-notnull-explicit')).toBe(false);
+    // Anti-vacuity: the same run DID attribute rewrites, so an empty `applied`
+    // cannot be what makes the line above green.
+    expect(ids.size).toBeGreaterThan(0);
+  });
+
   it('surfaces the semantic TODOs instead of auto-applying them', () => {
     expect(Array.isArray(out.parsed.todos)).toBe(true);
     expect(out.parsed.todos.length).toBeGreaterThan(0);
@@ -217,9 +230,16 @@ describe('os migrate meta --from 16 (e2e over the real CLI)', () => {
     expect(snap.tools[0].category).toBeUndefined();
     expect(snap.permissions[0].rowLevelSecurity[0].priority).toBeUndefined();
     expect(snap.sharingRules[0].accessLevel).toBe('edit');
-    // ADR-0113 explicitization: the pre-17 required field carries its column
-    // constraint in writing; the optional field gains nothing.
-    expect(snap.objects[0].fields.title.storage).toEqual({ notNull: true });
+    // ⛔ ADR-0113, #16693: the chain does NOT write a column constraint for the
+    // author. `required: true` crosses 16→17 as the write-time contract and
+    // nothing else, so `title` comes out with NO `storage` block — exactly like
+    // the optional field beside it. The withdrawn `field-required-notnull-
+    // explicit` conversion used to make this line read `{ notNull: true }`, and
+    // an app that followed the boot warning it raised was performing a
+    // destructive `tighten_not_null` migration on a populated database while
+    // believing it was clearing a deprecation notice.
+    expect(snap.objects[0].fields.title.required, 'the write contract survives the chain').toBe(true);
+    expect(snap.objects[0].fields.title.storage, 'no column constraint is invented').toBeUndefined();
     expect(snap.objects[0].fields.notes.storage).toBeUndefined();
   });
 

packages/drivers/driver-sql/src/schema-drift.ts

@@ -858,8 +858,18 @@ export function diffManagedTable(args: {
       // deliberately SILENT: that is every pre-protocol-17 source after a
       // runtime upgrade, the write gate makes the column constraint
       // unreachable (harmless belt-and-suspenders), and nagging every legacy
-      // required field would bury real drift. `os migrate meta` ratifies it
-      // whenever the source is next migrated.
+      // required field would bury real drift.
+      //
+      // ⚠️ Nothing ratifies it later either, and this block used to say the
+      // opposite — "`os migrate meta` ratifies it whenever the source is next
+      // migrated". That died with the ADR-0087 `field-required-notnull-explicit`
+      // conversion, WITHDRAWN in #16693 (maintainer ruling 2026-09-08): no
+      // chain step writes `storage.notNull` for anybody, at any protocol floor.
+      // So the silence above is permanent until the author declares the
+      // constraint themselves — which is the ADR-0113 posture (`required` is
+      // the write contract; `storage.notNull` alone binds the column), not a
+      // gap. The SILENCE itself is unchanged: this PR corrects the sentence,
+      // never the behaviour.
       out.push({
         kind: 'nullability_mismatch',
         remoteName: table,
@@ -873,8 +883,8 @@ export function diffManagedTable(args: {
         message:
           `${table}.${fieldName}: the column is NOT NULL but the metadata declares no ` +
           `storage constraint. Ratify it by declaring \`storage: { notNull: true }\` ` +
-          `(pre-protocol-17 sources: \`os migrate meta\` stamps it for every ` +
-          `previously-required field), or deliberately relax the column via "os migrate".`,
+          `yourself — nothing supplies it for you, and \`required: true\` does not imply ` +
+          `it (ADR-0113) — or deliberately relax the column via "os migrate".`,
       });
     } else if (!expectNullable && col.nullable) {
       out.push({

packages/drivers/driver-sql/src/sql-driver.ts

@@ -16587,9 +16587,21 @@ export class SqlDriver implements IDataDriver {
       // constraint, not from `required` — `required` is the write-time
       // contract enforced by the record validator at the engine seam, and
       // binding the DDL to it made every post-deploy tightening a
-      // destructive migration. Sources authored before protocol 17 carry
-      // `storage.notNull` explicitly via the `field-required-notnull-explicit`
-      // conversion, so their columns come out exactly as they always did.
+      // destructive migration.
+      //
+      // ⚠️ NOTHING supplies `storage.notNull` on an author's behalf. The
+      // sentence that used to close this block said the opposite — that
+      // pre-17 sources "carry `storage.notNull` explicitly via the
+      // `field-required-notnull-explicit` conversion, so their columns come
+      // out exactly as they always did" — and it was measured false from both
+      // ends (#16693): a real `^17.0.0` app logged that conversion and got
+      // NULLABLE columns anyway, and the conversion itself has since been
+      // WITHDRAWN (maintainer ruling 2026-09-08), because stamping the
+      // constraint wherever `required: true` appears is the implication
+      // ADR-0113 abolished. So a column reaches `notNullable()` here because
+      // its author wrote `storage: { notNull: true }`, and for no other
+      // reason; a `required: true` field with no `storage` block gets a
+      // nullable column, at every protocol floor, on every dialect.
       if ((field as { storage?: { notNull?: boolean } }).storage?.notNull) col.notNullable();
       this.applyDeclaredColumnDefault(col, field, type);
     }