Skip to content

fix(knowledge): waive source_* requirements for source_kind: internal (and teach the doc generator about it) - #1684

Closed
elhoim wants to merge 1 commit into
danielmiessler:mainfrom
elhoim:fix/kb-schema-internal-source-waiver
Closed

fix(knowledge): waive source_* requirements for source_kind: internal (and teach the doc generator about it)#1684
elhoim wants to merge 1 commit into
danielmiessler:mainfrom
elhoim:fix/kb-schema-internal-source-waiver

Conversation

@elhoim

@elhoim elhoim commented Jul 28, 2026

Copy link
Copy Markdown

PER_TYPE_REQUIRED demands source_url on every research note. That is wrong for notes carrying source_kind: internal — those have no external origin. They are the primary artifact: an own-prose capture, a design decision, a corpus the system itself built. The rule asks for a URL that cannot exist.

Found on a real archive

Migrating a 1,801-note archive onto kb-v3 left exactly two violation classes. The second was 10 research notes failing this rule with nothing to backfill — I checked every one, and none contained a single URL anywhere in its body. Three were LifeOS's own artifacts; seven were OCR captures whose source URL was lost upstream.

Editing ten notes to satisfy a rule that shouldn't apply to them is the wrong direction. Fixing the rule took one condition.

The change

const isInternal = deq(get("source_kind")) === "internal";
for (const key of PER_TYPE_REQUIRED[effType] ?? []) {
  if (isInternal && key.startsWith("source_")) continue;
  if (missingOrEmpty(key)) v.push({ key, problem: `required for type ${effType}` });
}

Externally-sourced notes are unaffected and still must carry their provenance.

Second half: the generator

GenerateKnowledgeSchemaDoc reads PER_TYPE_REQUIRED — a data table — while the new rule lives in validate(). Left alone, the generated _schema.md would state a requirement the linter does not enforce.

That is the exact failure this schema exists to prevent, so the generator now emits the waiver alongside the table.

Verified

On the archive this was found on:

Before After
Envelope-conformant 1791 / 1801 (99.4%) 1801 / 1801 (100%)
Fully conformant 1781 (98.9%) 1801 (100%)
Completeness gaps 10 0

_schema.md regenerated and carries the waiver text. One caveat worth stating: my first attempt at this used field(), which returns an FmField object rather than a string — KnowledgeLint threw a TypeError immediately rather than silently mis-validating, which is the behaviour you want from a linter. The committed version uses deq(get(...)), matching the surrounding code.

PER_TYPE_REQUIRED demands source_url on every research note. That is wrong
for notes with source_kind: internal — those have no external origin, they
ARE the primary artifact (an own-prose capture, a design decision, a corpus
the system itself built). The rule asks for a URL that cannot exist.

On a real archive this was 10 notes permanently unable to pass lint, with
nothing to backfill: none of the ten had a single URL anywhere in its body.
The fix is the schema, not the notes.

Skip source_* per-type requirements when source_kind is internal.
Externally-sourced notes still must carry their provenance.

Also teach GenerateKnowledgeSchemaDoc to emit the waiver. The generator
reads PER_TYPE_REQUIRED (a data table) while the rule lives in validate(),
so the generated doc would otherwise state a requirement the linter does
not enforce — the same doc-vs-behaviour drift this schema exists to prevent.
@danielmiessler

Copy link
Copy Markdown
Owner

Ported into source with credit — thank you.

The public repo is generated from a private tree by rsync at release time, so a merge here would be reverted at the next cut. Instead the intent landed in source, your handle and this PR number are in the code comment at the fix site, and it ships in the next release. Closing this PR as ported, not rejected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants