Sort JCS object keys by UTF-16 code unit, not code point - #3
Merged
Conversation
RFC 8785 section 3.2.3 orders object keys by UTF-16 code unit. json.dumps(sort_keys=True) orders them by Unicode code point. The two agree throughout the Basic Multilingual Plane and diverge above it, where a supplementary-plane key encodes as a surrogate pair beginning U+D800 and therefore sorts below keys in U+E000..U+FFFF. No published hash changes. Every key in every shipped example is ASCII, so the two orderings already agreed on all of them; spec_hash, criteria_hash and vtc_hash are byte-identical before and after. This closes a latent defect rather than correcting a live one. The module caveat previously disclosed only the number-serialization limitation and said nothing about key order, which was the more misleading omission of the two: a canonicalizer built on the shortcut passes an ASCII vector by accident and would fail on the first non-BMP key it ever saw. Two vectors now pin the behaviour, one at the root and one nested, and the caveat names what is fixed and what is still not implemented. Prompted by a canonicalization vector raised by @wowlegend in x402-foundation/x402#3065, which pins the same divergence class.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
jcs()intools/validate.pywas built onjson.dumps(sort_keys=True), which ordersobject keys by Unicode code point. RFC 8785 section 3.2.3 orders them by UTF-16
code unit.
The two agree throughout the Basic Multilingual Plane and diverge above it, because a
supplementary-plane key encodes as a surrogate pair beginning
U+D800and thereforesorts below keys in
U+E000..U+FFFF:No published hash changes
Every key in every shipped example is ASCII, so the two orderings already agreed on all
of them. Recomputed with the fix:
spec_hashsha256:bb0e87ce…c47adf35criteria_hashsha256:d9205d4f…6d66c69f7vtc_hashsha256:3184cbd5…dc8e1b705This closes a latent defect, not a live one. Nothing already published is wrong.
Why it was worth fixing rather than documenting
The module caveat disclosed only the number-serialization limitation and said nothing
about key order, which was the more misleading of the two omissions. A canonicalizer
carrying this shortcut passes an ASCII or BMP vector by accident, so the defect is
invisible to exactly the tests an implementer would think to write, and would surface
on the first non-BMP key the implementation ever saw.
Changes
tools/validate.py:_utf16_key_order()reorders keys recursively beforeserialization;
jcs()no longer passessort_keys.== canonicalization ==section, one at the root and onenested, so the property is pinned at every depth rather than only at the top.
unimplemented (ECMAScript number serialization over the full float range).
Verified that the previous implementation fails both new vectors and the new one passes,
so these are not vacuous.
Provenance
Prompted by a canonicalization vector raised by @wowlegend in
x402-foundation/x402#3065, which pins the same divergence class from the JavaScript
side. The vector that catches this shape is theirs; the bug is ours.