fix: the detector and the neutralizer agreed on nothing invisible (#812, #813, #814)

[raeq]

Closes #812, closes #813, closes #814. Refs #762.

Three issues, one channel: the neutralizer and the detector held independent definitions of "invisible", and they disagreed in both directions.

What was measured on main before the change

input	strip_format	canonicalize	llm_guardrail	has_anomalies
pay + U+1D173 + pal	unchanged	unchanged	unchanged	False
pay + U+1BCA0 + pal	unchanged	unchanged	unchanged	False
Hello + U+E000×4	kept (correct)	stripped	kept	False
Hello + tag payload	stripped	stripped	stripped	True

The tag row is included because #812's tag half was already fixed by #700 — the issue was written against 0.14.1. Re-measuring first is what kept this PR to the part that is still broken.

#813 — invisible by property, not by name

Twelve Cf code points marked Default_Ignorable_Code_Point=Yes: Duployan U+1BCA0–U+1BCA3, musical U+1D173–U+1D17A. Not zero-width by name, not Tags, not fillers, not bidi — so no predicate covered them.

pay split by an invisible is the worked example in the anomaly guide, and it is caught with U+200B. It was not caught with U+1D173.

Scoped to the twelve, not the category: of the 41 Cf that survive strip_format, the other 29 render and carry meaning (Arabic and Kaithi number signs, Egyptian hieroglyph layout controls) and are still kept. There is a test for that direction too.

#812 — the PUA run

canonicalize deleted it, the detector said nothing. Added as a Carrier with a run threshold of 4, deliberately not a neighbour rule: a single PUA code point beside a letter is an icon-font glyph, which is the same reason #413 has strip_format keep the block.

#814 — the profile could not, even in principle

ProfileSpec had twelve fields and none was PUA. Adds PipelineSteps::STRIP_PUA and the spec field; seven of eight profiles set it. code_context does not — #413's rule applied to the profiles, not an omission, since it is the one profile whose job is preserving its input.

>>> get_pipeline("llm_guardrail")("Summarize." + chr(0xF0000))
'summarize.'                      # was 'summarize.\U000f0000'
>>> get_pipeline("code_context")("Menu " + chr(0xE000))
'Menu '                     # preserved, by design

One definition

The twelve live in invisibles.rs and the strip path, the preset fast-path mask and the detector all read that function. #700 found the detector's own eight-character list had drifted from the strip functions; restating a class in a second place is how that happens.

One consequence is public and stated in the CHANGELOG: strip_zero_width_chars widens by those twelve on every surface. It is the function whose description is "strip zero-width and invisible characters", and a code point that renders as nothing belongs to that set by that description.

What caught it independently

docs/security/watermarks.md had already measured this exact class, and its "removed by nothing, reported by nothing" row held exactly these 12. test_watermark_page.py failed the moment the fix landed, with neither: 12 -> 0. The page is updated; that row is now empty.

Tests

tests/test_invisible_channel.py, 42 cases, every one failing on 0.14.1. Its closing gate enumerates Cf and asks which survivors are Default_Ignorable — anchored to the property rather than to the list of twelve, so it asks the issue's question instead of restating the answer.

Two existing tests asserted the defect and were updated rather than deleted: test_ml_normalize_leaves_private_use_area keeps its name because the preset half is still open, and the two profile step-order locks gained strip_pua.

Full gate green: cargo test 723, pytest 6,290, doc tests 41 pages, clippy both feature sets, perf_lint.sh, cargo doc, mkdocs --strict. No bindings/ file changed and no src/api signature moved.

🤖 Generated with Claude Code

[github-actions]

📄 Docs preview: https://cecd1337.disarm-docs.pages.dev

[Copilot]

🟡 Changes recommended

There are a couple of small but concrete issues in src/pipeline.rs (misleading flag comment and a missing buffer reserve in the new step) that should be corrected before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Unifies the definition of “invisible” characters across stripping (neutralization), anomaly detection, and LLM pipelines so that characters removed by canonicalization/stripping are also detectable (and vice versa), closing the gaps described in #812/#813/#814.

Changes:

• Adds Default_Ignorable Cf coverage (the 12 Duployan + musical controls) to the stripping and detection paths via a shared predicate.
• Introduces strip_pua as a pipeline step and wires it into most policy profiles (while keeping code_context preserving PUA by design).
• Expands anomaly detection to treat PUA runs as an “invisible” carrier class and updates docs/CHANGELOG and tests accordingly.

File summaries

File	Description
tests/test_policy_profiles.py	Updates expected named-profile step lists to include strip_pua.
tests/test_llm_presets.py	Locks step order for llm_guardrail/rag_ingest to include strip_pua.
tests/test_invisible_channel.py	New regression tests asserting stripper/detector agreement on the invisible channel.
tests/test_cve_vectors.py	Updates CVE-vector expectations around profiles vs presets for PUA handling.
src/whitespace.rs	Extends zero-width stripping to include the new default-ignorable format predicate.
src/presets.rs	Ensures preset fast-path classification accounts for the new invisible class member.
src/pipeline.rs	Adds STRIP_PUA step bitflag, ordering, profile wiring, and execution logic.
src/invisibles.rs	Adds is_default_ignorable_format and uses it in invisible-class stripping.
src/anomalies.rs	Uses the shared predicate for in-word invisibles and adds PUA run detection.
docs/user-guide/llm-pipelines.md	Documents the guardrail/profile PUA behavior change and code_context carve-out.
docs/user-guide/anomaly-detection.md	Updates the “invisible” kind definition and false-positive guards.
docs/security/watermarks.md	Updates the watermark measurement counts now that the “neither” class is closed.
CHANGELOG.md	Announces the unified invisible definition and the widened strip_zero_width_chars surface.

Review details

• Files reviewed: 13/13 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.