Skip to content

Adversarial fixtures: citations that mention a topic but do not support the claim #3

Description

@alsoleg89

Problem

The current 12-case benchmark proves structural traceability and safe rendering. It deliberately does not prove that an excerpt entails the attached evidence statement.

We need inspectable fixtures for the next benchmark layer.

Fixture families

Contributions can add one or more minimal examples:

  • a citation that merely mentions the subject;
  • a source whose scope is narrower than the claim;
  • correlation presented as causation;
  • a stale source used for a current-state claim;
  • two sources that appear contradictory only because populations or time windows differ;
  • an excerpt that drops a decisive negation or qualifier.

Acceptance criteria

  • Each fixture includes the claim, bounded source excerpt, locator, and expected verdict.
  • The failure is explainable without a model-specific score.
  • Copyrighted excerpts stay short and link to the source.
  • At least one nearby valid control prevents an over-broad rule.
  • Fixtures contain no secrets, personal data, or unverifiable private sources.
  • Benchmark documentation states which checks are deterministic and which require a reviewer/model.

See benchmarks/README.md and skill/doubt/references/evidence-ladder.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    evidence-contractChanges to validation rules or the map schemahelp wantedExtra attention is neededneeds-evidenceClaim or fixture needs stronger source support

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions