feat: duplicate_mark, and the key builders collapse a repeated mark (#835)

[raeq]

Closes #835. Refs #762.

The gap

UTS #39 §5.4 lists "forbid sequences of the same nonspacing mark" as an optional detection. None of it was implemented, and nothing else in the library covers it: a + two acutes is two marks, so no zalgo threshold reaches it, and it is one script, so mixed_script cannot see it. It renders indistinguishably from a + one acute.

>>> disarm.canonicalize("á́") == disarm.canonicalize("á")
False                      # two spellings of one word, two keys
>>> disarm.has_anomalies("á́")
False                      # and nothing says so

Both are now the other way round.

Two parts, and why they are separate

AnomalyKind::DuplicateMark reports after the zalgo rule. #724's enclosing-mark rule could go first because one enclosing mark per base is below every count threshold by construction, so the two can never both fire. A repeat has no such bound — four identical acutes are a repeat and a stack — and putting this first made every zalgo finding report as duplicate_mark instead.

Step::DropRepeatedMarks is a new pipeline step, deliberately not folded into the zalgo cap. #788 pairs strip_zalgo with is_zalgo so the cap never removes a mark from text the predicate calls ordinary — and two identical acutes are ordinary by that threshold. My first attempt put the dedupe inside strip_zalgo_into and broke the pairing on 540 strings. As its own step, both contracts hold:

>>> marks(disarm.strip_zalgo("Z" + "́" * 8))   # the cap, unchanged
3
>>> marks(disarm.canonicalize("Z" + "́" * 8))  # the cap + the repeat rule
1

It runs before the cap so the cap counts marks a reader can distinguish: a + five acutes + five graves capped first keeps three acutes and loses the grave entirely; deduplicating first keeps one of each.

Class-0 marks are out of scope, matching #842's discriminator — positioned rather than stacked, so a doubled Devanagari matra is an orthography question rather than this one.

Key movement

builder	rows moved (of 22,977)
canonicalize	11
canonicalize_strict	13
sort_key	11
search_key, catalog_key, strip_obfuscation, normalize_confusables, fold_case	0

No row grew. KEY_SCHEMA_VERSION 2 → 3, fixture regenerated in the same commit, Upgrade note written.

Gate

The #850/#862 ordering gate now checks the new step as well as the cap — it has the identical hazard, since a zero-width between two acutes hides the repeat from it exactly as it hides the count from the cap. Mutation-checked: moving the step before the zero-width strip fails it with

canonicalize: Step::DropRepeatedMarks runs before Step::StripZeroWidth, which can delete a character from between two mark runs

Two existing tests were updated rather than weakened: test_real_zalgo_is_still_stripped now asserts the 3-vs-1 split above (and adds a distinct-marks case proving the cap itself is untouched inside canonicalize), and sort_key_zalgo_cap_runs_after_the_zero_width_strip switched to four distinct class-230 marks — its four-identical-acutes input would otherwise collapse for the new reason and stop testing the ordering it was written for.

Full gate green: cargo test 723, pytest 6,222, doc tests 41 pages, clippy both feature sets, perf_lint.sh, cargo doc, mkdocs --strict.

🤖 Generated with Claude Code

[github-actions]

📄 Docs preview: https://46cd4e11.disarm-docs.pages.dev

[Copilot]

🟡 Changes recommended

drop_repeated_marks_into currently performs redundant NFC normalization despite being immediately followed by the zalgo NFD→NFC step in key-builder pipelines, introducing avoidable hot-path overhead.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Implements UTS #39 §5.4’s optional detection for repeated nonspacing marks by adding a new anomaly kind and a dedicated key-builder pipeline step that collapses immediately repeated stacking marks, ensuring visually indistinguishable spellings (e.g. á́ vs á) collide in canonical keys while keeping strip_zalgo’s cap semantics unchanged.

Changes:

• Add AnomalyKind::DuplicateMark detection (reported after zalgo) and document it.
• Add Step::DropRepeatedMarks to the key-builder pipelines (canonicalize, canonicalize_strict, sort_key) and extend the ordering gate accordingly.
• Bump KEY_SCHEMA_VERSION to 3 and update tests, docs, and Node binding typings for the new anomaly kind.

File summaries

File	Description
tests/test_zalgo_cap.py	Updates assertions to reflect dedupe-vs-cap behavior in canonicalize.
tests/test_anomalies.py	Adds a reachability sample for the new duplicate_mark anomaly kind.
src/zalgo.rs	Introduces drop_repeated_marks_into and its streaming predicate.
src/presets.rs	Adds DropRepeatedMarks step, wires it into key builders, and expands ordering gate coverage.
src/api/metadata.rs	Bumps KEY_SCHEMA_VERSION and documents key movement impact.
src/anomalies.rs	Adds DuplicateMark kind, message text, and classification logic.
docs/user-guide/anomaly-detection.md	Documents the new duplicate_mark anomaly in the user guide table.
docs/RUST_API.md	Updates measured false-positive count and notes the change driver.
CHANGELOG.md	Adds upgrade note + changelog entries for duplicate-mark key collapse and anomaly kind.
bindings/node/index.ts	Extends the AnomalyKind union type with duplicate_mark.

Review details

• Files reviewed: 10/11 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[raeq]

Rebased onto #873. Two conflicts, both resolved rather than picked:

• CHANGELOG — strip_obfuscation is not a fixed point: ы folds to ƅi, and the exhaustive idempotence gate tests the one function that iterates #723's upgrade note and this one are separate movements and both stay. The KEY_SCHEMA_VERSION bump to 3 in this PR now covers both.
• golden_keys.tsv.gz — regenerated against the rebased code rather than resolved to either side, since strip_obfuscation is not a fixed point: ы folds to ƅi, and the exhaustive idempotence gate tests the one function that iterates #723 moved strip_obfuscation on 90 rows and this PR moves three other builders. Picking a side would have committed a fixture matching neither.

While resolving that second one: #873 moved key output and left KEY_SCHEMA_VERSION at 2. tests/test_key_stability.py did not catch it, and cannot — it checks that the fixture header matches the constant, and gen_key_fixture.py writes the current constant into that header, so regenerating always agrees with whatever the constant happens to be. The counter's own doc comment names this exact failure ("Regenerating the fixture without bumping the constant — the exact way a counter like this goes stale") and the check aimed at it is anchored to the thing that drifts.

No action needed here — this PR's bump to 3 covers #723's movement too, so nothing shipped wrong. I'll raise the gate separately.