Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.es.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Y lo aplicamos a nosotros. Tres cosas que esta portada podría callar y no calla

- **Los dos motores no cubren lo mismo hoy.** `--engine opa` evalúa 133 de 159 reglas; el evaluador nativo por defecto evalúa 41 y salta 118, sobre el mismo repo. CI exige que coincidan sobre hechos, no sobre cobertura — eso es por diseño; que el comando por defecto no lo diga, no ([#628](https://github.com/beyondnetcode/evolith_arch32/issues/628)). Esta portada usa `--engine opa` en todas partes.
- **Dos reglas de infraestructura no están en ningún denominador.** El cargador rechaza tres ficheros del propio corpus, y desde 1.3.2 ya ni lo avisa por stderr ([#575](https://github.com/beyondnetcode/evolith_arch32/issues/575)).
- **Lo que se instala no es todo lo que hay en este árbol.** El árbol lleva 182 ficheros de reglas; el CLI publicado carga 177 packs con 412 reglas — el rechazo del cargador de arriba es una de las causas. `evolith rulesets` imprime lo que carga *tu* instalación, pack por pack.
- **El conteo de ficheros y el de reglas responden a preguntas distintas.** El árbol lleva 182 ficheros `*.rules.json`, de los cuales cuatro declaran un esquema que no es de ruleset y no aportan reglas por diseño — se nombran en cada informe, no se descartan en silencio. Quedan 178 packs con 413 reglas. El CLI publicado lleva su propia foto: 177 packs, 412 reglas. `evolith rulesets` imprime lo que carga *tu* instalación, pack por pack.

Auditoría completa de nuestras propias afirmaciones: [pendientes 2026-08-16](./reference/core/control-center/adoption/pending-2026-08-16.md).

Expand Down Expand Up @@ -135,7 +135,7 @@ Ocho **estilos de arquitectura** (aquí los llamamos *topologías*) repartidos e
| Datos | `data-mesh` |
| IA | `agentic-ai` |

Encima corre una biblioteca **gratis y MIT**: en este árbol, 142 ADRs, 182 ficheros de reglas y 50 schemas de fase, más las cinco fases del SDLC (Discovery → Design → Construction → QA → Delivery) y los controles que bloquean el paso de una a la siguiente. Esos tres conteos los mide y los verifica CI en cada PR. Lo que evalúa tu instalación lo imprime `evolith rulesets`: hoy, 177 packs con 412 reglas, 188 de ellas capaces de hacer fallar una ejecución. El único producto de pago será **Evolith Tracker**, aún no lanzado.
Encima corre una biblioteca **gratis y MIT**: en este árbol, 142 ADRs, 178 packs de reglas con 413 reglas repartidas en 182 ficheros, y 50 schemas de fase, más las cinco fases del SDLC (Discovery → Design → Construction → QA → Delivery) y los controles que bloquean el paso de una a la siguiente. Esos tres conteos los mide y los verifica CI en cada PR. Lo que evalúa tu instalación lo imprime `evolith rulesets`: hoy, 177 packs con 412 reglas, 188 de ellas capaces de hacer fallar una ejecución. El único producto de pago será **Evolith Tracker**, aún no lanzado.

<div align="center"><a href="https://beyondnetcode.github.io/evolith_arch32/master-view.html" title="Abrir el diagrama interactivo"><img src="./reference/core/sdlc/assets/master-view.svg" alt="Cómo encajan CLI, Core y las cinco fases del SDLC" width="820" /></a><br/><sub><b><a href="https://beyondnetcode.github.io/evolith_arch32/master-view.html">Abrir visor interactivo</a></b> — arrastra para desplazar, rueda para zoom</sub></div>

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ And we apply it to ourselves. Three things this front page could keep quiet and

- **The two engines do not cover the same ground today.** `--engine opa` evaluates 133 of 159 rules; the default native evaluator evaluates 41 and skips 118, on the same repository. They are held to agreement over facts in CI, not over coverage — that part is by design; that the default command never says so is not ([#628](https://github.com/beyondnetcode/evolith_arch32/issues/628)). This page uses `--engine opa` everywhere.
- **Two infrastructure rules are in no denominator.** The loader rejects three ruleset files from its own corpus, and as of 1.3.2 it no longer even says so on stderr ([#575](https://github.com/beyondnetcode/evolith_arch32/issues/575)).
- **What installs is not everything this tree holds.** The tree carries 182 ruleset files; the published CLI loads 177 packs with 412 rules — the loader rejection above is one of the causes. `evolith rulesets` prints what *your* installation loads, pack by pack.
- **The file count and the rule count answer different questions.** The tree carries 182 `*.rules.json` files, of which four declare a non-ruleset schema and contribute no rules by design — they are named in every report, not dropped. That leaves 178 packs with 413 rules. The published CLI carries its own snapshot: 177 packs, 412 rules. `evolith rulesets` prints what *your* installation loads, pack by pack.

Full audit of our own claims: [pending items, 2026-08-16](./reference/core/control-center/adoption/pending-2026-08-16.md).

Expand Down Expand Up @@ -135,7 +135,7 @@ Eight **architecture styles** (we call them *topologies*) across five axes. The
| Data | `data-mesh` |
| AI | `agentic-ai` |

On top runs a **free, MIT** library: in this tree, 142 ADRs, 182 ruleset files and 50 phase schemas, plus the five SDLC phases (Discovery → Design → Construction → QA → Delivery) and the gates that block the move from one to the next. Those three counts are measured and verified by CI on every PR. What your installation actually evaluates is printed by `evolith rulesets`: today, 177 packs with 412 rules, 188 of them able to fail a run. The only paid product will be **Evolith Tracker**, not yet launched.
On top runs a **free, MIT** library: in this tree, 142 ADRs, 178 ruleset packs carrying 413 rules across 182 files, and 50 phase schemas, plus the five SDLC phases (Discovery → Design → Construction → QA → Delivery) and the gates that block the move from one to the next. Those three counts are measured and verified by CI on every PR. What your installation actually evaluates is printed by `evolith rulesets`: today, 177 packs with 412 rules, 188 of them able to fail a run. The only paid product will be **Evolith Tracker**, not yet launched.

<div align="center"><a href="https://beyondnetcode.github.io/evolith_arch32/master-view.html" title="Open the interactive diagram"><img src="./reference/core/sdlc/assets/master-view.svg" alt="How the CLI, the Core and the five SDLC phases fit together" width="820" /></a><br/><sub><b><a href="https://beyondnetcode.github.io/evolith_arch32/master-view.html">Open the interactive viewer</a></b> — drag to pan, scroll to zoom</sub></div>

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
/**
* #575 — a `*.rules.json` the loader turns into no rules must reach the report.
*
* The loader globs `*.rules.json` and some matches are not rule SETS. Before this,
* the only trace was a log line: it does not survive `--format json`, it never
* reaches an exit code, and the document vanished out of every denominator the
* report published. That is the silent drop this project exists to stop, happening
* inside its own loader.
*
* These tests are about the two outcomes being weighted DIFFERENTLY. A document
* that declares a non-ruleset schema and satisfies it contributes no rules by
* design and must not fail a run. A document that claims to be a ruleset and is
* not must fail one — otherwise it is indistinguishable, in every number
* downstream, from a file that was never there.
*/

import { RulesetValidatorService } from './ruleset-validator.service';
import type { CorpusDocumentOutcome } from '../../domain/ports/ruleset-repository.port';

type Issue = { ruleId: string; blocking: boolean; severity: string; description: string };

/** Reaches the private method under test without re-running a whole validation. */
function issuesFor(outcomes: readonly CorpusDocumentOutcome[]): Issue[] {
const service = Object.create(RulesetValidatorService.prototype) as Record<string, unknown>;
service.rulesetRepo = { describeLastLoad: () => outcomes };
return (service as unknown as { corpusLoadIssues(): Issue[] }).corpusLoadIssues();
}

const classified: CorpusDocumentOutcome = {
file: 'infrastructure/helm-enforcement.rules.json',
outcome: 'classified',
declaredSchema: 'rule-definition.schema.json',
detail: 'a single rule declaration, enforced by its paired CI guard and Rego policy',
};

const rejected: CorpusDocumentOutcome = {
file: 'architecture/broken.rules.json',
outcome: 'rejected',
detail: "Schema validation failed: data must have required property 'rules'",
};

describe('corpus load accounting (#575)', () => {
it('says nothing when the whole corpus loaded', () => {
expect(issuesFor([])).toEqual([]);
});

it('reports a classified document without failing the run', () => {
const [issue, ...rest] = issuesFor([classified]);

expect(rest).toEqual([]);
expect(issue.ruleId).toBe('GOV-CORPUS-NOT-A-RULESET');
expect(issue.blocking).toBe(false);
expect(issue.severity).toBe('COULD');
// The file has to be NAMED, or the row is a count the reader cannot act on.
expect(issue.description).toContain('infrastructure/helm-enforcement.rules.json');
expect(issue.description).toContain('paired CI guard');
});

it('fails the run for a document that claims to be a ruleset and is not', () => {
const [issue, ...rest] = issuesFor([rejected]);

expect(rest).toEqual([]);
expect(issue.ruleId).toBe('GOV-CORPUS-REJECTED');
expect(issue.blocking).toBe(true);
expect(issue.severity).toBe('MUST');
expect(issue.description).toContain('architecture/broken.rules.json');
expect(issue.description).toContain("must have required property 'rules'");
});

it('keeps the two outcomes apart when both occur', () => {
const issues = issuesFor([classified, rejected]);

expect(issues.map(i => i.ruleId)).toEqual(['GOV-CORPUS-NOT-A-RULESET', 'GOV-CORPUS-REJECTED']);
// The classified one must not be dragged into blocking by its neighbour.
expect(issues.map(i => i.blocking)).toEqual([false, true]);
});

it('stays silent for a repository that cannot describe its load', () => {
const service = Object.create(RulesetValidatorService.prototype) as Record<string, unknown>;
service.rulesetRepo = { loadAllRulesets: async () => [] };

expect((service as unknown as { corpusLoadIssues(): Issue[] }).corpusLoadIssues()).toEqual([]);
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { findCoreFromSatellite } from '../paths/rulesets-location';
import * as path from 'path';
import { ILogger, IFileSystem, IConfigParser } from '../../domain/interfaces';
import { RuleEvaluationEngine, emptyRuleCoverage, summarizeRuleCoverage } from './rule-evaluation-engine';
import { RulesetsNotFoundError } from '../../domain/ports/ruleset-repository.port';
import { IRulesetRepository, RulesetsNotFoundError } from '../../domain/ports/ruleset-repository.port';
import { NativeEvaluator } from './evaluators/native-evaluator';
import { OpaEvaluator } from './evaluators/opa-evaluator';
import { createCompositeEnforcerStrategy } from './enforcement/enforcer-subsystem';
Expand Down Expand Up @@ -43,6 +43,7 @@ export class RulesetValidatorService {
private readonly topologyCatalog?: TopologyCatalogService;
/** GT-569 — optional coverage floor; see {@link RulesetValidatorOptions.maxSkippedFraction}. */
private readonly maxSkippedFraction?: number;
private readonly rulesetRepo: IRulesetRepository;
/** GT-571 — filter the corpus by rule audience / topology / SDLC phase. */
private readonly applyRuleApplicability: boolean;
/**
Expand Down Expand Up @@ -87,6 +88,7 @@ export class RulesetValidatorService {
this.applyRuleApplicability = options.applyRuleApplicability !== false;
this.processRunner = options.processRunner;
this.metrics = options.metrics;
this.rulesetRepo = options.rulesetRepo;

const baseStrategy = options.engineType === 'opa'
? new OpaEvaluator(this.fs, this.logger)
Expand Down Expand Up @@ -238,6 +240,7 @@ export class RulesetValidatorService {
if (applicabilityIssue) issues.push(applicabilityIssue);
const thresholdIssue = this.coverageThresholdIssue(coverage);
if (thresholdIssue) issues.push(thresholdIssue);
issues.push(...this.corpusLoadIssues());
} catch (err: unknown) {
// GT-474: an unresolvable/empty ruleset corpus must never be downgraded to
// a warning here — that is exactly how `validate` came to report
Expand Down Expand Up @@ -380,6 +383,70 @@ export class RulesetValidatorService {
* which is information, not a violation. It is what stops "0 blocking
* findings" from meaning "we quietly stopped looking".
*/
/**
* #575 -- the corpus loader reads every `*.rules.json` and some of them become
* no rules at all. Until now the only trace was a log line, which does not
* survive `--format json` and never reaches an exit code, so a document could
* vanish out of every denominator the report published. That is the failure
* this project exists to stop, occurring in its own loader.
*
* Two outcomes, deliberately weighted differently:
*
* - `classified` -- the document declares a known non-ruleset schema and
* satisfies it. It contributes no rules BY DESIGN (a single rule definition
* enforced by its paired Rego policy and CI guard, the advisory topology
* catalogue, an SDLC phase-gate document). Reported, NOT blocking: it is a
* fact about the corpus, not a violation.
* - `rejected` -- the document claims to be a ruleset, or declares nothing,
* and failed the ruleset schema. Blocking. A file that says it carries rules
* and carries none is indistinguishable, in every downstream number, from a
* file that was never there.
*
* An implementation that cannot describe its load returns nothing here, and
* behaviour is exactly as before.
*/
private corpusLoadIssues(): ValidationIssue[] {
const outcomes = this.rulesetRepo.describeLastLoad?.() ?? [];
if (outcomes.length === 0) return [];

const issues: ValidationIssue[] = [];
const classified = outcomes.filter(o => o.outcome === 'classified');
const rejected = outcomes.filter(o => o.outcome === 'rejected');

if (classified.length > 0) {
issues.push({
ruleId: 'GOV-CORPUS-NOT-A-RULESET',
severity: 'COULD',
category: 'governance',
title: `${classified.length} corpus file(s) declare a non-ruleset schema and contribute no rules`,
description:
`${classified.length} \`*.rules.json\` file(s) were read and produced no rules because they are ` +
'a different kind of document, each satisfying the schema it declares: ' +
classified.map(c => `\`${c.file}\` (${c.detail})`).join('; ') +
'. They are counted here rather than in `rulesTotal`, because they were never rules. ' +
'Enforcement for these lives in their paired Rego policies and CI guards, not in the rule engine.',
blocking: false,
});
}

if (rejected.length > 0) {
issues.push({
ruleId: 'GOV-CORPUS-REJECTED',
severity: 'MUST',
category: 'governance',
title: `${rejected.length} corpus file(s) were rejected at load and evaluated nothing`,
description:
`${rejected.length} \`*.rules.json\` file(s) failed the standard ruleset schema and were dropped: ` +
rejected.map(r => `\`${r.file}\` — ${r.detail}`).join('; ') +
'. Every rule they carry is in no denominator: not checked, not skipped, not errored. ' +
'Fix the file, or give it a `$schema` that declares what it actually is.',
blocking: true,
});
}

return issues;
}

private applicabilityAdvisory(notApplicable: readonly NotApplicableRule[]): ValidationIssue | undefined {
if (notApplicable.length === 0) return undefined;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,40 @@ export class RulesetCorpusNotResolvedError extends RulesetsNotFoundError {
}
}

/**
* What the corpus loader did with a `*.rules.json` that produced no rules.
*
* #575: the loader globs `*.rules.json` and four shipped files are not rule
* SETS. Three declare a different schema on purpose and one is a phase-gate
* document; all four contributed nothing, and the only trace was a log line.
* A log line is not accounting -- it does not survive `--format json`, it does
* not reach an exit code, and it is exactly the silent drop this project exists
* to stop, happening inside its own loader.
*/
export interface CorpusDocumentOutcome {
/** Path relative to the corpus root, so the report is stable across machines. */
readonly file: string;
/**
* `classified` -- declares a known non-ruleset schema and satisfies it, so it
* contributes no rules BY DESIGN. `rejected` -- claims to be a ruleset, or
* declares nothing, and failed the ruleset schema. The second is a defect;
* the first is a fact about the corpus.
*/
readonly outcome: 'classified' | 'rejected';
/** Basename of the declared `$schema`, when the document declares one. */
readonly declaredSchema?: string;
/** The kind, for `classified`. The validation failure, for `rejected`. */
readonly detail: string;
}

export interface IRulesetRepository {
/** @throws {RulesetsNotFoundError} when no rulesets resolve at `corePath`. */
loadAllRulesets(corePath: string): Promise<NormalizedRule[]>;
/**
* Every document the most recent {@link loadAllRulesets} read and did not turn
* into rules. Optional so an implementation that cannot know stays valid --
* but an implementation that DOES drop documents and does not report them
* reintroduces #575.
*/
describeLastLoad?(): readonly CorpusDocumentOutcome[];
}
12 changes: 11 additions & 1 deletion src/packages/infra-providers/src/caching-ruleset.repository.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { ILogger } from "@beyondnet/evolith-core-domain/domain/interfaces";
import { NormalizedRule } from "@beyondnet/evolith-core-domain/domain/models/normalized-rule";
import { IRulesetRepository } from "@beyondnet/evolith-core-domain/domain/ports/ruleset-repository.port";
import { CorpusDocumentOutcome, IRulesetRepository } from "@beyondnet/evolith-core-domain/domain/ports/ruleset-repository.port";

/**
* GT-648 — the ruleset corpus is deployment state, not request state.
Expand Down Expand Up @@ -77,6 +77,16 @@ export class CachingRulesetRepository implements IRulesetRepository {
return [...rules];
}

/**
* #575: the diagnostics belong to whichever load actually read disk, so this
* delegates rather than caching. A caller asking what the last load dropped
* must get the inner repository's answer, not a memoised one from a different
* corpus path.
*/
describeLastLoad(): readonly CorpusDocumentOutcome[] {
return this.inner.describeLastLoad?.() ?? [];
}

/**
* Warm the cache ahead of the first request. Returns the number of rules
* loaded so a caller (the boot hook) can log it as the evidence that the
Expand Down
Loading
Loading