From a9476a78fb7df7ab7063f9b9b5729600ee31e2c2 Mon Sep 17 00:00:00 2001 From: aarroyo Date: Fri, 21 Aug 2026 14:24:17 -0500 Subject: [PATCH 1/2] fix(core): a corpus file that becomes no rules now reaches the report Closes #575. The loader globs `*.rules.json` and four shipped files are not rule SETS. They were read, produced nothing, and the only trace was a log line. A log line does not survive `--format json`, never reaches an exit code, and left the documents out of every denominator the report published -- the silent drop this project exists to stop, inside its own loader. #575 asked for the accounting FIRST and the files second, and it was right: the files turned out not to need changing. GT-649 had already classified three of them by declared `$schema`, and the two INFRA rules are genuinely enforced elsewhere -- `src/rulesets/opa/infrastructure/*.rego` plus the dedicated `29-validate-opa-sidecar-bundles.mjs` guard. Wrapping them into the corpus would have double-counted them. What was missing was the accounting, and only that. - `IRulesetRepository` gains `CorpusDocumentOutcome` and an optional `describeLastLoad()`. Optional so an implementation that cannot know stays valid; an implementation that drops documents and stays quiet reintroduces this bug. - The validator emits two rows, deliberately weighted differently. `GOV-CORPUS-NOT-A-RULESET` (COULD, non-blocking) names each document that declares a known non-ruleset schema and satisfies it -- a fact about the corpus, not a violation. `GOV-CORPUS-REJECTED` (MUST, blocking) names each document that claims to be a ruleset and is not, because such a file is indistinguishable downstream from one that was never there. - The fourth file, `sdlc/phase-gates.rules.json`, was recognised by FILENAME -- `filePath.endsWith("phase-gates.rules.json")` skipped schema validation and then normalised to zero rules, so it was neither validated nor reported. It declares `ruleset-sdlc.schema.json` like its neighbours declare theirs, so it now dispatches on that. The path literal is gone, and a rename can no longer defeat it. `PhaseGateValidatorService` reads the file at its own path and is unaffected. Verified against the built CLI on a fresh satellite, not asserted: - the row appears in the report AND in `--format json`, naming all four files with the reason for each - the blocking branch is falsifiable: dropping one deliberately broken `*.rules.json` into the corpus produced `GOV-CORPUS-REJECTED ... | YES` and exit 2; removing it returned the run to the previous state - no rules were lost by the phase-gates change: `evolith rulesets` reports 178 packs / 413 rules before and after Tests: infra-providers 181, core-domain 1997 (5 new in `corpus-load-accounting.spec.ts`), CLI 133, core-api 163 -- all passing. Co-Authored-By: Claude Opus 5 Signed-off-by: aarroyo --- .../validators/corpus-load-accounting.spec.ts | 84 ++++++++++++++++++ .../validators/ruleset-validator.service.ts | 69 ++++++++++++++- .../domain/ports/ruleset-repository.port.ts | 33 +++++++ .../src/caching-ruleset.repository.ts | 12 ++- .../src/disk-ruleset.repository.spec.ts | 87 +++++++++++++++++-- .../src/disk-ruleset.repository.ts | 85 ++++++++++++++---- 6 files changed, 346 insertions(+), 24 deletions(-) create mode 100644 src/packages/core-domain/src/application/validators/corpus-load-accounting.spec.ts diff --git a/src/packages/core-domain/src/application/validators/corpus-load-accounting.spec.ts b/src/packages/core-domain/src/application/validators/corpus-load-accounting.spec.ts new file mode 100644 index 000000000..bfa836dec --- /dev/null +++ b/src/packages/core-domain/src/application/validators/corpus-load-accounting.spec.ts @@ -0,0 +1,84 @@ +/** + * #575 — a `*.rules.json` the loader turns into no rules must reach the report. + * + * The loader globs `*.rules.json` and some matches are not rule SETS. Before this, + * the only trace was a log line: it does not survive `--format json`, it never + * reaches an exit code, and the document vanished out of every denominator the + * report published. That is the silent drop this project exists to stop, happening + * inside its own loader. + * + * These tests are about the two outcomes being weighted DIFFERENTLY. A document + * that declares a non-ruleset schema and satisfies it contributes no rules by + * design and must not fail a run. A document that claims to be a ruleset and is + * not must fail one — otherwise it is indistinguishable, in every number + * downstream, from a file that was never there. + */ + +import { RulesetValidatorService } from './ruleset-validator.service'; +import type { CorpusDocumentOutcome } from '../../domain/ports/ruleset-repository.port'; + +type Issue = { ruleId: string; blocking: boolean; severity: string; description: string }; + +/** Reaches the private method under test without re-running a whole validation. */ +function issuesFor(outcomes: readonly CorpusDocumentOutcome[]): Issue[] { + const service = Object.create(RulesetValidatorService.prototype) as Record; + service.rulesetRepo = { describeLastLoad: () => outcomes }; + return (service as unknown as { corpusLoadIssues(): Issue[] }).corpusLoadIssues(); +} + +const classified: CorpusDocumentOutcome = { + file: 'infrastructure/helm-enforcement.rules.json', + outcome: 'classified', + declaredSchema: 'rule-definition.schema.json', + detail: 'a single rule declaration, enforced by its paired CI guard and Rego policy', +}; + +const rejected: CorpusDocumentOutcome = { + file: 'architecture/broken.rules.json', + outcome: 'rejected', + detail: "Schema validation failed: data must have required property 'rules'", +}; + +describe('corpus load accounting (#575)', () => { + it('says nothing when the whole corpus loaded', () => { + expect(issuesFor([])).toEqual([]); + }); + + it('reports a classified document without failing the run', () => { + const [issue, ...rest] = issuesFor([classified]); + + expect(rest).toEqual([]); + expect(issue.ruleId).toBe('GOV-CORPUS-NOT-A-RULESET'); + expect(issue.blocking).toBe(false); + expect(issue.severity).toBe('COULD'); + // The file has to be NAMED, or the row is a count the reader cannot act on. + expect(issue.description).toContain('infrastructure/helm-enforcement.rules.json'); + expect(issue.description).toContain('paired CI guard'); + }); + + it('fails the run for a document that claims to be a ruleset and is not', () => { + const [issue, ...rest] = issuesFor([rejected]); + + expect(rest).toEqual([]); + expect(issue.ruleId).toBe('GOV-CORPUS-REJECTED'); + expect(issue.blocking).toBe(true); + expect(issue.severity).toBe('MUST'); + expect(issue.description).toContain('architecture/broken.rules.json'); + expect(issue.description).toContain("must have required property 'rules'"); + }); + + it('keeps the two outcomes apart when both occur', () => { + const issues = issuesFor([classified, rejected]); + + expect(issues.map(i => i.ruleId)).toEqual(['GOV-CORPUS-NOT-A-RULESET', 'GOV-CORPUS-REJECTED']); + // The classified one must not be dragged into blocking by its neighbour. + expect(issues.map(i => i.blocking)).toEqual([false, true]); + }); + + it('stays silent for a repository that cannot describe its load', () => { + const service = Object.create(RulesetValidatorService.prototype) as Record; + service.rulesetRepo = { loadAllRulesets: async () => [] }; + + expect((service as unknown as { corpusLoadIssues(): Issue[] }).corpusLoadIssues()).toEqual([]); + }); +}); diff --git a/src/packages/core-domain/src/application/validators/ruleset-validator.service.ts b/src/packages/core-domain/src/application/validators/ruleset-validator.service.ts index 6cd51686f..f1ceee38a 100644 --- a/src/packages/core-domain/src/application/validators/ruleset-validator.service.ts +++ b/src/packages/core-domain/src/application/validators/ruleset-validator.service.ts @@ -3,7 +3,7 @@ import { findCoreFromSatellite } from '../paths/rulesets-location'; import * as path from 'path'; import { ILogger, IFileSystem, IConfigParser } from '../../domain/interfaces'; import { RuleEvaluationEngine, emptyRuleCoverage, summarizeRuleCoverage } from './rule-evaluation-engine'; -import { RulesetsNotFoundError } from '../../domain/ports/ruleset-repository.port'; +import { IRulesetRepository, RulesetsNotFoundError } from '../../domain/ports/ruleset-repository.port'; import { NativeEvaluator } from './evaluators/native-evaluator'; import { OpaEvaluator } from './evaluators/opa-evaluator'; import { createCompositeEnforcerStrategy } from './enforcement/enforcer-subsystem'; @@ -43,6 +43,7 @@ export class RulesetValidatorService { private readonly topologyCatalog?: TopologyCatalogService; /** GT-569 — optional coverage floor; see {@link RulesetValidatorOptions.maxSkippedFraction}. */ private readonly maxSkippedFraction?: number; + private readonly rulesetRepo: IRulesetRepository; /** GT-571 — filter the corpus by rule audience / topology / SDLC phase. */ private readonly applyRuleApplicability: boolean; /** @@ -87,6 +88,7 @@ export class RulesetValidatorService { this.applyRuleApplicability = options.applyRuleApplicability !== false; this.processRunner = options.processRunner; this.metrics = options.metrics; + this.rulesetRepo = options.rulesetRepo; const baseStrategy = options.engineType === 'opa' ? new OpaEvaluator(this.fs, this.logger) @@ -238,6 +240,7 @@ export class RulesetValidatorService { if (applicabilityIssue) issues.push(applicabilityIssue); const thresholdIssue = this.coverageThresholdIssue(coverage); if (thresholdIssue) issues.push(thresholdIssue); + issues.push(...this.corpusLoadIssues()); } catch (err: unknown) { // GT-474: an unresolvable/empty ruleset corpus must never be downgraded to // a warning here — that is exactly how `validate` came to report @@ -380,6 +383,70 @@ export class RulesetValidatorService { * which is information, not a violation. It is what stops "0 blocking * findings" from meaning "we quietly stopped looking". */ + /** + * #575 -- the corpus loader reads every `*.rules.json` and some of them become + * no rules at all. Until now the only trace was a log line, which does not + * survive `--format json` and never reaches an exit code, so a document could + * vanish out of every denominator the report published. That is the failure + * this project exists to stop, occurring in its own loader. + * + * Two outcomes, deliberately weighted differently: + * + * - `classified` -- the document declares a known non-ruleset schema and + * satisfies it. It contributes no rules BY DESIGN (a single rule definition + * enforced by its paired Rego policy and CI guard, the advisory topology + * catalogue, an SDLC phase-gate document). Reported, NOT blocking: it is a + * fact about the corpus, not a violation. + * - `rejected` -- the document claims to be a ruleset, or declares nothing, + * and failed the ruleset schema. Blocking. A file that says it carries rules + * and carries none is indistinguishable, in every downstream number, from a + * file that was never there. + * + * An implementation that cannot describe its load returns nothing here, and + * behaviour is exactly as before. + */ + private corpusLoadIssues(): ValidationIssue[] { + const outcomes = this.rulesetRepo.describeLastLoad?.() ?? []; + if (outcomes.length === 0) return []; + + const issues: ValidationIssue[] = []; + const classified = outcomes.filter(o => o.outcome === 'classified'); + const rejected = outcomes.filter(o => o.outcome === 'rejected'); + + if (classified.length > 0) { + issues.push({ + ruleId: 'GOV-CORPUS-NOT-A-RULESET', + severity: 'COULD', + category: 'governance', + title: `${classified.length} corpus file(s) declare a non-ruleset schema and contribute no rules`, + description: + `${classified.length} \`*.rules.json\` file(s) were read and produced no rules because they are ` + + 'a different kind of document, each satisfying the schema it declares: ' + + classified.map(c => `\`${c.file}\` (${c.detail})`).join('; ') + + '. They are counted here rather than in `rulesTotal`, because they were never rules. ' + + 'Enforcement for these lives in their paired Rego policies and CI guards, not in the rule engine.', + blocking: false, + }); + } + + if (rejected.length > 0) { + issues.push({ + ruleId: 'GOV-CORPUS-REJECTED', + severity: 'MUST', + category: 'governance', + title: `${rejected.length} corpus file(s) were rejected at load and evaluated nothing`, + description: + `${rejected.length} \`*.rules.json\` file(s) failed the standard ruleset schema and were dropped: ` + + rejected.map(r => `\`${r.file}\` — ${r.detail}`).join('; ') + + '. Every rule they carry is in no denominator: not checked, not skipped, not errored. ' + + 'Fix the file, or give it a `$schema` that declares what it actually is.', + blocking: true, + }); + } + + return issues; + } + private applicabilityAdvisory(notApplicable: readonly NotApplicableRule[]): ValidationIssue | undefined { if (notApplicable.length === 0) return undefined; diff --git a/src/packages/core-domain/src/domain/ports/ruleset-repository.port.ts b/src/packages/core-domain/src/domain/ports/ruleset-repository.port.ts index b4a941320..e5fa78b23 100644 --- a/src/packages/core-domain/src/domain/ports/ruleset-repository.port.ts +++ b/src/packages/core-domain/src/domain/ports/ruleset-repository.port.ts @@ -37,7 +37,40 @@ export class RulesetCorpusNotResolvedError extends RulesetsNotFoundError { } } +/** + * What the corpus loader did with a `*.rules.json` that produced no rules. + * + * #575: the loader globs `*.rules.json` and four shipped files are not rule + * SETS. Three declare a different schema on purpose and one is a phase-gate + * document; all four contributed nothing, and the only trace was a log line. + * A log line is not accounting -- it does not survive `--format json`, it does + * not reach an exit code, and it is exactly the silent drop this project exists + * to stop, happening inside its own loader. + */ +export interface CorpusDocumentOutcome { + /** Path relative to the corpus root, so the report is stable across machines. */ + readonly file: string; + /** + * `classified` -- declares a known non-ruleset schema and satisfies it, so it + * contributes no rules BY DESIGN. `rejected` -- claims to be a ruleset, or + * declares nothing, and failed the ruleset schema. The second is a defect; + * the first is a fact about the corpus. + */ + readonly outcome: 'classified' | 'rejected'; + /** Basename of the declared `$schema`, when the document declares one. */ + readonly declaredSchema?: string; + /** The kind, for `classified`. The validation failure, for `rejected`. */ + readonly detail: string; +} + export interface IRulesetRepository { /** @throws {RulesetsNotFoundError} when no rulesets resolve at `corePath`. */ loadAllRulesets(corePath: string): Promise; + /** + * Every document the most recent {@link loadAllRulesets} read and did not turn + * into rules. Optional so an implementation that cannot know stays valid -- + * but an implementation that DOES drop documents and does not report them + * reintroduces #575. + */ + describeLastLoad?(): readonly CorpusDocumentOutcome[]; } diff --git a/src/packages/infra-providers/src/caching-ruleset.repository.ts b/src/packages/infra-providers/src/caching-ruleset.repository.ts index d8728cbe6..62639897d 100644 --- a/src/packages/infra-providers/src/caching-ruleset.repository.ts +++ b/src/packages/infra-providers/src/caching-ruleset.repository.ts @@ -1,6 +1,6 @@ import { ILogger } from "@beyondnet/evolith-core-domain/domain/interfaces"; import { NormalizedRule } from "@beyondnet/evolith-core-domain/domain/models/normalized-rule"; -import { IRulesetRepository } from "@beyondnet/evolith-core-domain/domain/ports/ruleset-repository.port"; +import { CorpusDocumentOutcome, IRulesetRepository } from "@beyondnet/evolith-core-domain/domain/ports/ruleset-repository.port"; /** * GT-648 — the ruleset corpus is deployment state, not request state. @@ -77,6 +77,16 @@ export class CachingRulesetRepository implements IRulesetRepository { return [...rules]; } + /** + * #575: the diagnostics belong to whichever load actually read disk, so this + * delegates rather than caching. A caller asking what the last load dropped + * must get the inner repository's answer, not a memoised one from a different + * corpus path. + */ + describeLastLoad(): readonly CorpusDocumentOutcome[] { + return this.inner.describeLastLoad?.() ?? []; + } + /** * Warm the cache ahead of the first request. Returns the number of rules * loaded so a caller (the boot hook) can log it as the evidence that the diff --git a/src/packages/infra-providers/src/disk-ruleset.repository.spec.ts b/src/packages/infra-providers/src/disk-ruleset.repository.spec.ts index 112e85534..403d53322 100644 --- a/src/packages/infra-providers/src/disk-ruleset.repository.spec.ts +++ b/src/packages/infra-providers/src/disk-ruleset.repository.spec.ts @@ -165,18 +165,95 @@ describe('DiskRulesetRepository', () => { }); }); - it('skips schema validation for phase-gates rulesets', async () => { + // #575: this used to assert that a file NAMED `phase-gates.rules.json` bypassed + // schema validation. Dispatch now reads the document's declared `$schema`, like + // every other non-corpus kind, so the filename carries no meaning and a rename + // cannot silently defeat it. A document that declares the SDLC schema is + // classified and contributes no rules; one that merely has the old name is an + // ordinary ruleset and is validated as such. + it('classifies an SDLC phase-gate document by its declared schema, not its filename', async () => { const fs = makeFs({ - dirs: new Set(['/core/rulesets']), + dirs: new Set(['/core/rulesets', '/core/rulesets/schema']), files: { - '/core/rulesets/phase-gates.rules.json': JSON.stringify({ - rules: [{ id: 'GATE-1', severity: 'MUST', title: 'Gate' }], + '/core/rulesets/schema/ruleset-standard.schema.json': SCHEMA, + '/core/rulesets/renamed-gates.rules.json': JSON.stringify({ + $schema: '../schema/ruleset-sdlc.schema.json', + title: 'Phase gates', + gates: [{ id: 'GATE-1' }], + }), + '/core/rulesets/good.rules.json': JSON.stringify({ + rules: [{ id: 'OK-1', severity: 'MUST', title: 'Good' }], }), }, }); const repo = new DiskRulesetRepository(fs, makeLogger()); + const rules = await repo.loadAllRulesets('/core'); - expect(rules.map((r) => r.id)).toEqual(['GATE-1']); + + expect(rules.map((r) => r.id)).toEqual(['OK-1']); + expect(repo.describeLastLoad()).toEqual([ + expect.objectContaining({ + file: 'renamed-gates.rules.json', + outcome: 'classified', + declaredSchema: 'ruleset-sdlc.schema.json', + }), + ]); + }); + + // #575: the load-bearing half. A document the loader drops must reach the + // caller as data, not only as a log line -- a log line does not survive + // `--format json` and never reaches an exit code. + it('reports a rejected ruleset as data, not only as a warning (#575)', async () => { + const fs = makeFs({ + dirs: new Set(['/core/rulesets', '/core/rulesets/schema']), + files: { + '/core/rulesets/schema/ruleset-standard.schema.json': SCHEMA, + '/core/rulesets/broken.rules.json': JSON.stringify({ notRules: [] }), + '/core/rulesets/good.rules.json': JSON.stringify({ + rules: [{ id: 'OK-1', severity: 'MUST', title: 'Good' }], + }), + }, + }); + const repo = new DiskRulesetRepository(fs, makeLogger()); + + await repo.loadAllRulesets('/core'); + const dropped = repo.describeLastLoad(); + + expect(dropped).toHaveLength(1); + expect(dropped[0].file).toBe('broken.rules.json'); + expect(dropped[0].outcome).toBe('rejected'); + expect(dropped[0].detail).toContain('Schema validation failed'); + }); + + // A corpus where nothing was dropped must say so with an empty list rather + // than with the previous load's answer. + it('describes a clean load as empty, and does not carry outcomes across loads', async () => { + const fs = makeFs({ + dirs: new Set(['/core/rulesets', '/core/rulesets/schema']), + files: { + '/core/rulesets/schema/ruleset-standard.schema.json': SCHEMA, + '/core/rulesets/broken.rules.json': JSON.stringify({ notRules: [] }), + '/core/rulesets/good.rules.json': JSON.stringify({ + rules: [{ id: 'OK-1', severity: 'MUST', title: 'Good' }], + }), + }, + }); + const repo = new DiskRulesetRepository(fs, makeLogger()); + await repo.loadAllRulesets('/core'); + expect(repo.describeLastLoad()).toHaveLength(1); + + const cleanFs = makeFs({ + dirs: new Set(['/core/rulesets', '/core/rulesets/schema']), + files: { + '/core/rulesets/schema/ruleset-standard.schema.json': SCHEMA, + '/core/rulesets/good.rules.json': JSON.stringify({ + rules: [{ id: 'OK-1', severity: 'MUST', title: 'Good' }], + }), + }, + }); + const cleanRepo = new DiskRulesetRepository(cleanFs, makeLogger()); + await cleanRepo.loadAllRulesets('/core'); + expect(cleanRepo.describeLastLoad()).toEqual([]); }); it('skips (does not abort on) a ruleset that fails schema validation and still loads the valid ones (GT-456)', async () => { diff --git a/src/packages/infra-providers/src/disk-ruleset.repository.ts b/src/packages/infra-providers/src/disk-ruleset.repository.ts index cf5299e72..9dce11177 100644 --- a/src/packages/infra-providers/src/disk-ruleset.repository.ts +++ b/src/packages/infra-providers/src/disk-ruleset.repository.ts @@ -2,6 +2,7 @@ import * as path from "path"; import { IFileSystem, ILogger } from "@beyondnet/evolith-core-domain/domain/interfaces"; import { NormalizedRule } from "@beyondnet/evolith-core-domain/domain/models/normalized-rule"; import { + CorpusDocumentOutcome, IRulesetRepository, RulesetCorpusNotResolvedError, RulesetsNotFoundError, @@ -42,8 +43,33 @@ const NON_CORPUS_DOCUMENT_KINDS: ReadonlyMap = new Map([ "topology-recommendation.schema.json", "the ADR-0104 advisory topology recommendation catalogue, read by TopologyRecommendationService", ], + // #575: this one used to be recognised by FILENAME -- `filePath.endsWith( + // "phase-gates.rules.json")` skipped standard validation and then normalised + // to zero rules, so the document was neither validated nor reported. It + // declares its own schema like the others; dispatching on that declaration + // puts it in the same accounting as its neighbours and removes a path literal + // that a rename would have silently defeated. + [ + "ruleset-sdlc.schema.json", + "an SDLC phase-gate document, evaluated by PhaseGateValidator rather than by the rule engine", + ], ]); +/** + * Corpus-root-relative path, so a reported file reads the same on every machine + * and in every container. Falls back to the basename if the path is not under + * the root, which should not happen but must not produce an absolute path in a + * published report. + */ +function relativeToCorpus(filePath: string, rulesetsDir: string): string { + const prefix = rulesetsDir.endsWith(path.sep) + ? rulesetsDir + : rulesetsDir + path.sep; + return filePath.startsWith(prefix) + ? filePath.slice(prefix.length).split(path.sep).join("/") + : (filePath.split(path.sep).pop() ?? filePath); +} + /** Basename of a declared `$schema`, or `undefined` when none is declared. */ function declaredSchemaName(parsed: Record): string | undefined { const raw = parsed["$schema"]; @@ -115,6 +141,17 @@ export class DiskRulesetRepository implements IRulesetRepository { return rulesetsRoot; } + /** + * #575: what the last load did with every document that produced no rules. + * Replaced (not appended to) on each load, so it always describes the corpus + * the caller just read rather than accumulating across calls. + */ + private lastLoad: CorpusDocumentOutcome[] = []; + + describeLastLoad(): readonly CorpusDocumentOutcome[] { + return this.lastLoad; + } + async loadAllRulesets(corePath: string): Promise { // GT-474: a missing rulesets root is a HARD error. Returning [] here made // `validate` check zero rules and still report a (non-blocking) `warning` — @@ -124,6 +161,7 @@ export class DiskRulesetRepository implements IRulesetRepository { const files = await this.findRulesetFiles(rulesetsDir); const rules: NormalizedRule[] = []; + const outcomes: CorpusDocumentOutcome[] = []; for (const filePath of files) { const content = await this.fs.readFile(filePath); @@ -148,28 +186,31 @@ export class DiskRulesetRepository implements IRulesetRepository { declaredSchemaName(parsed) ?? "", ); if (kind) { + outcomes.push({ + file: relativeToCorpus(filePath, rulesetsDir), + outcome: "classified", + declaredSchema: declaredSchemaName(parsed), + detail: kind, + }); await this.checkNonCorpusDocument(parsed, filePath, rulesetsDir, kind); continue; } try { - // Exclude SDLC gate rulesets from standard validation here since PhaseGateValidator handles them - if (!filePath.endsWith("phase-gates.rules.json")) { - if (!this.validateSchema) { - const schemaPath = path.join( - rulesetsDir, - "schema", - "ruleset-standard.schema.json", - ); - const schemaContent = await this.fs.readFile(schemaPath); - this.validateSchema = this.ajv.compile(JSON.parse(schemaContent)); - } - const valid = this.validateSchema(parsed); - if (!valid) { - throw new Error( - `Schema validation failed: ${this.ajv.errorsText(this.validateSchema.errors)}`, - ); - } + if (!this.validateSchema) { + const schemaPath = path.join( + rulesetsDir, + "schema", + "ruleset-standard.schema.json", + ); + const schemaContent = await this.fs.readFile(schemaPath); + this.validateSchema = this.ajv.compile(JSON.parse(schemaContent)); + } + const valid = this.validateSchema(parsed); + if (!valid) { + throw new Error( + `Schema validation failed: ${this.ajv.errorsText(this.validateSchema.errors)}`, + ); } const relative = filePath.replace(corePath + path.sep, ""); @@ -182,9 +223,18 @@ export class DiskRulesetRepository implements IRulesetRepository { // `validate` silently check nothing. Skip it with a warning and keep // evaluating the remaining rulesets. (Malformed JSON is a hard error, // handled above.) + // #575: the warning stays, but it is no longer the ONLY trace. A + // rejected document now reaches the report, where a reader looking at + // `--format json` or an exit code can see it. this.logger.warn( `Skipping non-standard ruleset ${filePath}: ${message}`, ); + outcomes.push({ + file: relativeToCorpus(filePath, rulesetsDir), + outcome: "rejected", + declaredSchema: declaredSchemaName(parsed), + detail: message, + }); continue; } } @@ -199,6 +249,7 @@ export class DiskRulesetRepository implements IRulesetRepository { ); } + this.lastLoad = outcomes; return rules; } From 191e67e92d9cc67b618e5c68dffcc3e9f36cb5db Mon Sep 17 00:00:00 2001 From: aarroyo Date: Fri, 21 Aug 2026 14:25:04 -0500 Subject: [PATCH 2/2] docs(readme): publish the tree's rule count, now that it can be measured MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The front page said the tree "carries 182 ruleset files" and stopped there, because a tree-side RULE count could not be obtained: the published CLI evaluates its bundled corpus regardless of `coreRef.path`, so pointing it at this tree returns the tarball's numbers, not the tree's. Building the CLI from this tree answers it — 178 packs, 413 rules — and #575's accounting explains the gap between 182 files and 178 packs without hand-waving: four files declare a non-ruleset schema and contribute no rules by design, and they are now named in every report rather than dropped. Both halves say the same three numbers, and the previous sentence blaming "the loader rejection above" goes with them — nothing is being rejected. Co-Authored-By: Claude Opus 5 Signed-off-by: aarroyo --- README.es.md | 4 ++-- README.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.es.md b/README.es.md index 130d6b199..5dff16be2 100644 --- a/README.es.md +++ b/README.es.md @@ -51,7 +51,7 @@ Y lo aplicamos a nosotros. Tres cosas que esta portada podría callar y no calla - **Los dos motores no cubren lo mismo hoy.** `--engine opa` evalúa 133 de 159 reglas; el evaluador nativo por defecto evalúa 41 y salta 118, sobre el mismo repo. CI exige que coincidan sobre hechos, no sobre cobertura — eso es por diseño; que el comando por defecto no lo diga, no ([#628](https://github.com/beyondnetcode/evolith_arch32/issues/628)). Esta portada usa `--engine opa` en todas partes. - **Dos reglas de infraestructura no están en ningún denominador.** El cargador rechaza tres ficheros del propio corpus, y desde 1.3.2 ya ni lo avisa por stderr ([#575](https://github.com/beyondnetcode/evolith_arch32/issues/575)). -- **Lo que se instala no es todo lo que hay en este árbol.** El árbol lleva 182 ficheros de reglas; el CLI publicado carga 177 packs con 412 reglas — el rechazo del cargador de arriba es una de las causas. `evolith rulesets` imprime lo que carga *tu* instalación, pack por pack. +- **El conteo de ficheros y el de reglas responden a preguntas distintas.** El árbol lleva 182 ficheros `*.rules.json`, de los cuales cuatro declaran un esquema que no es de ruleset y no aportan reglas por diseño — se nombran en cada informe, no se descartan en silencio. Quedan 178 packs con 413 reglas. El CLI publicado lleva su propia foto: 177 packs, 412 reglas. `evolith rulesets` imprime lo que carga *tu* instalación, pack por pack. Auditoría completa de nuestras propias afirmaciones: [pendientes 2026-08-16](./reference/core/control-center/adoption/pending-2026-08-16.md). @@ -135,7 +135,7 @@ Ocho **estilos de arquitectura** (aquí los llamamos *topologías*) repartidos e | Datos | `data-mesh` | | IA | `agentic-ai` | -Encima corre una biblioteca **gratis y MIT**: en este árbol, 142 ADRs, 182 ficheros de reglas y 50 schemas de fase, más las cinco fases del SDLC (Discovery → Design → Construction → QA → Delivery) y los controles que bloquean el paso de una a la siguiente. Esos tres conteos los mide y los verifica CI en cada PR. Lo que evalúa tu instalación lo imprime `evolith rulesets`: hoy, 177 packs con 412 reglas, 188 de ellas capaces de hacer fallar una ejecución. El único producto de pago será **Evolith Tracker**, aún no lanzado. +Encima corre una biblioteca **gratis y MIT**: en este árbol, 142 ADRs, 178 packs de reglas con 413 reglas repartidas en 182 ficheros, y 50 schemas de fase, más las cinco fases del SDLC (Discovery → Design → Construction → QA → Delivery) y los controles que bloquean el paso de una a la siguiente. Esos tres conteos los mide y los verifica CI en cada PR. Lo que evalúa tu instalación lo imprime `evolith rulesets`: hoy, 177 packs con 412 reglas, 188 de ellas capaces de hacer fallar una ejecución. El único producto de pago será **Evolith Tracker**, aún no lanzado.
Cómo encajan CLI, Core y las cinco fases del SDLC
Abrir visor interactivo — arrastra para desplazar, rueda para zoom
diff --git a/README.md b/README.md index 53cdcf28d..e48fdcb4c 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ And we apply it to ourselves. Three things this front page could keep quiet and - **The two engines do not cover the same ground today.** `--engine opa` evaluates 133 of 159 rules; the default native evaluator evaluates 41 and skips 118, on the same repository. They are held to agreement over facts in CI, not over coverage — that part is by design; that the default command never says so is not ([#628](https://github.com/beyondnetcode/evolith_arch32/issues/628)). This page uses `--engine opa` everywhere. - **Two infrastructure rules are in no denominator.** The loader rejects three ruleset files from its own corpus, and as of 1.3.2 it no longer even says so on stderr ([#575](https://github.com/beyondnetcode/evolith_arch32/issues/575)). -- **What installs is not everything this tree holds.** The tree carries 182 ruleset files; the published CLI loads 177 packs with 412 rules — the loader rejection above is one of the causes. `evolith rulesets` prints what *your* installation loads, pack by pack. +- **The file count and the rule count answer different questions.** The tree carries 182 `*.rules.json` files, of which four declare a non-ruleset schema and contribute no rules by design — they are named in every report, not dropped. That leaves 178 packs with 413 rules. The published CLI carries its own snapshot: 177 packs, 412 rules. `evolith rulesets` prints what *your* installation loads, pack by pack. Full audit of our own claims: [pending items, 2026-08-16](./reference/core/control-center/adoption/pending-2026-08-16.md). @@ -135,7 +135,7 @@ Eight **architecture styles** (we call them *topologies*) across five axes. The | Data | `data-mesh` | | AI | `agentic-ai` | -On top runs a **free, MIT** library: in this tree, 142 ADRs, 182 ruleset files and 50 phase schemas, plus the five SDLC phases (Discovery → Design → Construction → QA → Delivery) and the gates that block the move from one to the next. Those three counts are measured and verified by CI on every PR. What your installation actually evaluates is printed by `evolith rulesets`: today, 177 packs with 412 rules, 188 of them able to fail a run. The only paid product will be **Evolith Tracker**, not yet launched. +On top runs a **free, MIT** library: in this tree, 142 ADRs, 178 ruleset packs carrying 413 rules across 182 files, and 50 phase schemas, plus the five SDLC phases (Discovery → Design → Construction → QA → Delivery) and the gates that block the move from one to the next. Those three counts are measured and verified by CI on every PR. What your installation actually evaluates is printed by `evolith rulesets`: today, 177 packs with 412 rules, 188 of them able to fail a run. The only paid product will be **Evolith Tracker**, not yet launched.
How the CLI, the Core and the five SDLC phases fit together
Open the interactive viewer — drag to pan, scroll to zoom