Reject nameprep-stripped code points in unicodeToASCII#429
Conversation
garydgregory
left a comment
There was a problem hiding this comment.
Hello @sahvx655-wq
What about:
00AD: Soft hyphen200B->200D: Zero width space, zero width non-joiner, zero width joinerFEFF: Zero width no-break space (Byte Order Mark)
?
|
Those five are all general category Cf, so the FORMAT check from #420 (which this PR keeps) already rejects them before the new branch is even consulted. When I walked RFC 3454 Table B.1 against Character.getType, the Cf entries were exactly what the existing guard caught; the leftovers were U+034F (Mn), U+1806 (Pd), U+180B..U+180D and U+FE00..U+FE0F (Mn), and those are what isNameprepMappedToNothing adds. Any of them alone is enough to make a byte-distinct host validate as the clean label, so the two checks together should now cover the whole mapped-to-nothing table. testIDNFormatCodePoints already pinned 00AD, 200B, 200D and FEFF; 200C was the one from your list without an explicit assertion, so I've pushed a one-line addition covering it. DomainValidatorTest is green with all of them asserted. |
|
Those five are all general category Cf, so the FORMAT check merged in #420 already rejects them; I re-checked the categories to be sure (00AD, 200B..200D and FEFF all report Character.FORMAT) and testIDNFormatCodePoints pins them. 200C was the one from your list without an explicit assertion, so I've pushed a commit adding it and the class is green (22 tests). This PR is about the remainder of the RFC 3454 Table B.1 set that is not Cf: U+034F, U+1806, U+180B..U+180D and U+FE00..U+FE0F report as Mn/Pd, so the getType check waves them through while IDN.toASCII still deletes them during nameprep, which is how exa͏mple.com was converting to and validating as example.com. The isNameprepMappedToNothing guard covers exactly that leftover set; without it the invisible-character homograph hole from #420 stays half open for those code points. |
unicodeToASCII already returns the host unconverted when it carries a Unicode FORMAT code point, because IDN.toASCII deletes those during nameprep and the label would otherwise validate as a clean string. Walking the nameprep mapping showed the FORMAT category is only part of RFC 3454 Table B.1 ("commonly mapped to nothing"): the combining grapheme joiner U+034F, the Mongolian TODO soft hyphen and free variation selectors U+1806 and U+180B..U+180D, and the variation selectors U+FE00..U+FE0F are stripped as well but report as Mn/Pd, so the getType check waves them through. isValid("exa͏mple.com"), plus the variation-selector and Mongolian forms, convert to example.com and return true, so a byte-distinct host validates as example.com and EmailValidator.isValidDomain and UrlValidator.isValidAuthority inherit the hole; left alone it is an invisible-character homograph and an allow-list poisoning vector.
The guard now rejects those non-FORMAT code points beside the existing FORMAT check, so the label regex sees the original invisible character and fails it. Keeping it inside unicodeToASCII closes the domain, email and URL callers together rather than each re-testing for stripped code points, and legitimate IDN letters (bücher.ch, президент.рф) and the hangul fillers that punycode to a real label are untouched. testIDNMappedToNothing covers the four representative code points; it fails on master and passes with the guard.