Skip to content

Reject nameprep-stripped code points in unicodeToASCII#429

Merged
garydgregory merged 2 commits into
apache:masterfrom
sahvx655-wq:domain-reject-nameprep-stripped
Jul 23, 2026
Merged

Reject nameprep-stripped code points in unicodeToASCII#429
garydgregory merged 2 commits into
apache:masterfrom
sahvx655-wq:domain-reject-nameprep-stripped

Conversation

@sahvx655-wq

Copy link
Copy Markdown
Contributor

unicodeToASCII already returns the host unconverted when it carries a Unicode FORMAT code point, because IDN.toASCII deletes those during nameprep and the label would otherwise validate as a clean string. Walking the nameprep mapping showed the FORMAT category is only part of RFC 3454 Table B.1 ("commonly mapped to nothing"): the combining grapheme joiner U+034F, the Mongolian TODO soft hyphen and free variation selectors U+1806 and U+180B..U+180D, and the variation selectors U+FE00..U+FE0F are stripped as well but report as Mn/Pd, so the getType check waves them through. isValid("exa͏mple.com"), plus the variation-selector and Mongolian forms, convert to example.com and return true, so a byte-distinct host validates as example.com and EmailValidator.isValidDomain and UrlValidator.isValidAuthority inherit the hole; left alone it is an invisible-character homograph and an allow-list poisoning vector.

The guard now rejects those non-FORMAT code points beside the existing FORMAT check, so the label regex sees the original invisible character and fails it. Keeping it inside unicodeToASCII closes the domain, email and URL callers together rather than each re-testing for stripped code points, and legitimate IDN letters (bücher.ch, президент.рф) and the hangul fillers that punycode to a real label are untouched. testIDNMappedToNothing covers the four representative code points; it fails on master and passes with the guard.

@garydgregory garydgregory left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello @sahvx655-wq

What about:

  • 00AD: Soft hyphen
  • 200B -> 200D: Zero width space, zero width non-joiner, zero width joiner
  • FEFF: Zero width no-break space (Byte Order Mark)

?

@sahvx655-wq

Copy link
Copy Markdown
Contributor Author

Those five are all general category Cf, so the FORMAT check from #420 (which this PR keeps) already rejects them before the new branch is even consulted. When I walked RFC 3454 Table B.1 against Character.getType, the Cf entries were exactly what the existing guard caught; the leftovers were U+034F (Mn), U+1806 (Pd), U+180B..U+180D and U+FE00..U+FE0F (Mn), and those are what isNameprepMappedToNothing adds. Any of them alone is enough to make a byte-distinct host validate as the clean label, so the two checks together should now cover the whole mapped-to-nothing table.

testIDNFormatCodePoints already pinned 00AD, 200B, 200D and FEFF; 200C was the one from your list without an explicit assertion, so I've pushed a one-line addition covering it. DomainValidatorTest is green with all of them asserted.

@sahvx655-wq

Copy link
Copy Markdown
Contributor Author

Those five are all general category Cf, so the FORMAT check merged in #420 already rejects them; I re-checked the categories to be sure (00AD, 200B..200D and FEFF all report Character.FORMAT) and testIDNFormatCodePoints pins them. 200C was the one from your list without an explicit assertion, so I've pushed a commit adding it and the class is green (22 tests).

This PR is about the remainder of the RFC 3454 Table B.1 set that is not Cf: U+034F, U+1806, U+180B..U+180D and U+FE00..U+FE0F report as Mn/Pd, so the getType check waves them through while IDN.toASCII still deletes them during nameprep, which is how exa͏mple.com was converting to and validating as example.com. The isNameprepMappedToNothing guard covers exactly that leftover set; without it the invisible-character homograph hole from #420 stays half open for those code points.

@garydgregory garydgregory changed the title reject nameprep-stripped code points in unicodeToASCII Reject nameprep-stripped code points in unicodeToASCII Jul 23, 2026
@garydgregory
garydgregory merged commit b41a464 into apache:master Jul 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants