Skip to content

reject out-of-range code points in UTF-32 wide-string input - #5348

Open
Angadi56 wants to merge 2 commits into
nlohmann:developfrom
Angadi56:utf32-eof-sentinel
Open

reject out-of-range code points in UTF-32 wide-string input#5348
Angadi56 wants to merge 2 commits into
nlohmann:developfrom
Angadi56:utf32-eof-sentinel

Conversation

@Angadi56

@Angadi56 Angadi56 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

wide_string_input_helper<BaseInputAdapter, 4>::fill_buffer converts UTF-32 input into the UTF-8 the lexer reads, and any code point above U+10FFFF falls into an "unknown character" branch that passes the code unit straight through. That pass-through narrows a 32-bit unit to int, so the single unit 0xFFFFFFFF arrives as -1, which is exactly std::char_traits<char>::eof(). The lexer reads that as end of input, so the parse stops there and the strict trailing check in parser::parse still succeeds: json::parse(std::u32string(U"[1]") + char32_t(0xFFFFFFFF) + U"anything") returns [1] and json::accept returns true, with every unit after the sentinel silently discarded. I found it while lining this branch up against the sibling UTF-16 helper, whose pass-through is safe only because its units cannot exceed 0xFFFF. The neighboring out-of-range values already behave correctly, since 0x110000 and a lone surrogate both stay positive after the cast and raise parse_error.101, so only the one value that collides with the sentinel is mis-decoded. The fix has that branch emit a byte that is never valid UTF-8 instead of a narrowed unit, which is the same collision input_stream_adapter::get_character already documents guarding against, and it leaves the two in-range casts in the file alone because they are bounded to 0..0x7F. Valid input is unchanged and the regression test goes in the existing invalid std::u32string section.

  • The changes are described in detail, both the what and why.
  • If applicable, an existing issue is referenced.
  • The Code coverage remained at 100%. A test case for every new line of code.
  • If applicable, the documentation is updated.
  • The source code is amalgamated by running make amalgamate.

@nlohmann nlohmann left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! The change looks good, but please fix the self-cast warning from GCC.

@nlohmann

nlohmann commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Clang was updated and the latest develop branch fixes a CI issue. Please rebase to the latest develop branch.

@nlohmann nlohmann added the please rebase Please rebase your branch to origin/develop label Aug 4, 2026
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
@Angadi56
Angadi56 force-pushed the utf32-eof-sentinel branch from 7647568 to 0b552a4 Compare August 4, 2026 18:11
@Angadi56

Angadi56 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Done. Rebased onto the latest develop and dropped the cast in the out-of-range branch. char_traits::int_type is int and 0xFF already is one, so the plain assignment avoids the useless-cast warning.

@nlohmann nlohmann removed the please rebase Please rebase your branch to origin/develop label Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants