Refresh bundled LanguageTool rules to 61bf3ff8, and fix the Java ground truth

[dchaplinsky]

Closes #8.

The pin moves d24cd528 → 61bf3ff8 (2026-07-23) — not the ed14f8572b the issue quotes, which was already two commits stale when the watchdog opened it. Refreshed per PROVENANCE.md.

The interesting part

Upstream deleted the 63 inline (?U) flags. That looks cosmetic; it isn't. LanguageTool now compiles every rule with UNICODE_CHARACTER_CLASS instead — SrxTools passes segment's DEFAULT_PATTERN_FLAGS_PARAMETER — so the prefixes became redundant.

Python is Unicode-aware by default, so choppa needs no change. But the segment 2.0.3 CLI we use as ground truth compiles under the JDK's own defaults, and since JDK 19 that means an ASCII-only \b. The first differential run came back 2465 vs 2394 segments, every difference a Cyrillic abbreviation exception (р. Десна, ім. Леніна, проф. Є.) that stopped firing on the Java side. The harness was wrong, not the port.

2.0.3 can't be told to use those flags — defaultPatternFlags arrived in 2.0.4, which was never released as a binary — so scripts/lt_java_srx.py writes a copy with (?U) prefixed onto each pattern, which compiles to the same thing. With it the ground truth became JDK-independent: 17, 21 and 24 now agree byte for byte, where the pristine file does not.

A latent bug, fixed by accident

ByTwoLineBreaks was rewritten to spell out ASCII whitespace instead of \s, so global (?U) couldn't let a paragraph break span an NBSP. Python's \s was always Unicode-aware — so choppa used to break paragraphs on \n\xa0\n where Java kept them whole. Confirmed against old-rules Java on JDK 17: 150 segments vs 141. Invisible on all six benchmark corpora.

tests/test_paragraph_breaks.py pins it — and it's a real regression test: 15 failures against the old ruleset, clean against the new. Every expectation was byte-verified against the Java CLI first, which is how the uk_two/en_two asymmetry on U+2029 turned out to be Java's behaviour too, not a quirk.

Verification

Test suite	124 passed, 851 LT subtests (was 848 — fixtures re-extracted at the same commit; the 3 new ones cover the new Catalan rules)
Differential	10 corpora, ~84,500 segments, zero differences
Corpora	uk Wikipedia + Militarny news, 3 Gutenberg novels, Catalan + Russian Wikipedia, synthetic Unicode-whitespace stress
Impact	uk/en/ru output unchanged; Catalan gains breaks (… a X. No és …, … Sargó I. El nom …)

CI

The gate now asserts both halves of the contract at the pin: the rules match, and LanguageTool still compiles them Unicode-aware (greps SrxTools.java for UNICODE_CHARACTER_CLASS). A refresh that silently changed the second would leave the gate measuring nothing — which is exactly the failure this PR exists to fix, one level up.

Worth knowing: the old gate would still have passed, because CI pins JDK 17, where \b is Unicode-aware. It fails on 19+. So this wasn't a red build — it was a gate one java-version: bump away from breaking while silently validating against abandoned semantics.

Follow-up, deliberately not here

The Java invocation now lives in four places (CI, README, benchmark.py docstring, PROVENANCE by reference) and this PR widened all four in lockstep. Folding it into benchmark.py --segment-bin would collapse them to one, but that's a new subprocess-driving feature, not a rules refresh.

🤖 Generated with Claude Code

https://claude.ai/code/session_01XNTo45TUJea3HnXNMwLq6D