Copied PDF text has decomposed diacritics

[dstillman]

https://forums.zotero.org/discussion/133014/bug-report-copying-text-from-zotero-splits-combining-diacritics-causing-spell-check-failures

I thought this was just a misunderstanding of NFKD, that we were copying whatever was in the document, and that it was up to external programs to normalize if they wanted to, but it looks like we're genuinely producing mixed-normalization output. (And even if we were copying directly, it seems reasonable to always normalize extracted text to NFC. Text extraction from a PDF is different from, say, copying from a text editor, where copying the original Unicode faithfully seems appropriate.)

Claude's summary:

This is reproducible and isn't passthrough of the PDF's own encoding. Testing with two sample PDFs containing "Educação da nação" — one with fully precomposed text (U+00E3/U+00E7), one with a ToUnicode CMap mapping to fully decomposed sequences — a real double-click + copy in the reader produces the identical mixed-form output for both:

E d u c a U+00E7 U+0061 U+0303 o
         ç (NFC) ã (decomposed)

ç is composed and ã is decomposed regardless of the source PDF. (poppler extracts both files faithfully.)

Cause

normalizeChar() in getStructuredText char building (evaluator.js#L3244-L3282) NFKD-decomposes every glyph's unicode (intentionally, to decompose ligatures) and then re-composes via a hard-coded specialCases map. The map covers acute/grave/circumflex/diaeresis vowels plus ç and ñ, but omits ã/õ — and everything else (ẽ, ũ, š, č, å, macrons, ogoneks, …). Anything not in the map stays decomposed in char.c, which feeds text selection, copy, and highlight annotation text. pdf-worker builds from the same fork, so extracted full-text content carries the same mixed form.

Suggested fix

Replace the specialCases table with canonical recomposition:

char.normalize('NFKD').normalize('NFC')

NFC recomposes every canonical base + combining-mark pair (all current specialCases entries produce exactly their NFKD → NFC result), while compatibility decompositions such as ligatures stay decomposed, preserving the intended behavior. The existing map already collapses two codepoints to one, so this doesn't introduce a new class of offset change relative to the pdf.js text layer.

Also, FYI:

Related

While testing this: double-clicking on a page whose only word has no wordBreakAfter throws TypeError: head.rects is undefined (getClosestWord() returns undefined offsets when no char on the page has wordBreakAfter, and getSelectionRanges() then dereferences head.rects). Doesn't occur with normal multi-word pages.