Manuscript Forensics

Catch statistical impossibilities before peer review—without overclaiming the evidence.

Manuscript Forensics is a local Python toolkit for checking whether the numbers, citations, figures, and software claims inside a scientific manuscript can coexist. Its central feature is not another LLM opinion: every finding carries an evidence tier that programmatically limits how strongly it may be worded.

reported tables + citations + figures + software versions
                              │
                              ▼
                    reproducible checks
                              │
                              ▼
        algebraic fact │ distribution dilemma │ measurement bound

What it catches

Check	Question answered	Evidence tier
Subgroup pooling	Do subgroup means and SDs reconstruct the reported total?	Algebraic identity
ANOVA reconstruction	Is the reported p-value plausible from the tabulated summaries?	Distribution-dependent
Bernoulli sum SD	Does a reported total-score SD survive an independence comparison and rounding guard?	Distribution-dependent
Expected counts	Are threshold counts compatible with the reported mean and SD under normality?	Distribution-dependent
DOI resolution	Is the DOI genuinely unregistered, or did the lookup itself fail?	Verified metadata
Citation values	Do numbers attributed to a source match that source?	Algebraic identity
Threshold provenance	Does the cited manual actually contain the attributed cutoff?	Algebraic identity
Figure recovery	What can be concluded when only part of a plotted dataset is recoverable?	Incomplete measurement
Software probes	Could the declared package/version reproduce the claimed behavior?	Check-specific

The evidence-tier contract

Tier	Allowed conclusion	Rejected conclusion
ALGEBRAIC	“These values cannot both hold.”	—
DISTRIBUTION	“Either the distribution is unusual, or this summary is misleading.”	“This is impossible.”
INCOMPLETE	“Even the worst-case bound does not reach the claim.”	Any point estimate

This is enforced in code. A distribution-dependent Finding that claims an “impossibility,” or an incomplete measurement that reports a point estimate, raises TierLanguageError before the language can reach an author.

Thirty-second demo

git clone https://github.com/odafeng/manuscript-forensics
cd manuscript-forensics
python -m venv .venv && source .venv/bin/activate
pip install -e .

manuscript-forensics pooling examples/pooling_mismatch.json

The synthetic example reports two subgroups whose pooled SD is far above the stated total SD. The output records the reconstructed values, both sample- and population-SD conventions, the full rounding sweep, and the surviving evidence tier as JSON.

{
  "tier": "algebraic_identity",
  "holds": true,
  "claim": "The subgroup table and the total table cannot both hold...",
  "data": {
    "mean_reconciled": true,
    "sd_reconciled": false,
    "ddof0_reconciled": false
  }
}

Use --fail-on-finding to make a surviving discrepancy return exit code 1 in an audit pipeline.

Python API

from checks.stats import Group, pooling_identity_check

groups = [
    Group(label="Group A", mean=10.0, sd=1.0, n=50),
    Group(label="Group B", mean=12.0, sd=1.2, n=50),
]

finding = pooling_identity_check(
    groups,
    reported_mean=11.0,
    reported_sd=0.5,
    decimals=1,
)

print(finding.tier, finding.holds)
print(finding.claim)
print(finding.data)

The modules under checks/ expose the statistical, reference, figure, DOCX, and software-probing functions independently. Network fetchers are injected, so reference checks can be cached, mocked, and tested without hiding transport failures.

Why rounding comes first

A printed value of 0.91 represents an interval, not an infinitely precise point. Before a gap is reported, the checks expand printed values across their rounding intervals and, where relevant, try both sample and population SD conventions. A difference that disappears under either guard is reconciled—not promoted into a finding.

Confidentiality

This public repository was built as a clean-room export and contains synthetic fixtures only. No manuscript, author information, unpublished summary statistics, review draft, or original git history is included.

• .doc, .docx, and .pdf are ignored by default.
• Run checks locally when a manuscript is confidential.
• Do not paste manuscript text into issues or public CI logs.
• DOI and package lookups receive identifiers only when the caller explicitly supplies a fetcher.

Limitations

• Summary statistics cannot reconstruct ranks or arbitrary distributions.
• An ANOVA reconstruction is a diagnostic comparison, not proof about a rank-based test.
• Independence is not a lower bound for a sum of correlated Bernoulli variables.
• Digitised figures require explicit calibration and legend masking.
• A transport error is never evidence that a DOI or citation is invalid.
• Findings support human peer review; they do not replace domain judgment.

Development

pip install -e ".[dev]"
ruff check .
ruff format --check .
pyright
pytest -q

License

MIT — see LICENSE.