feat(stdlib): implement string.pack, string.unpack, string.packsize#217
Merged
Conversation
Adds full Lua 5.3 §6.4.2 support for packing and unpacking binary data via a new Lua.VM.Stdlib.String.Pack module. Handles all format options including arbitrary integer widths (i1-i16), endianness directives (< > =), platform alignment (!N, X<op>), fixed-size and length-prefixed strings (c<n>, s<n>, z), and the IEEE ±Infinity bit-pattern round-trip needed because the BEAM uses 1.0e308 as a finite stand-in for ±1/0. Promotes tpack.lua from skipped to ready in the Lua 5.3 suite. Two small in-scope discoveries: string.reverse was UTF-8-oriented (broken for binaries containing NUL); string.rep rejected float counts even though Lua 5.3's '^' always returns a float. Both surfaced via tpack.lua, both fixed in the same commit. See Discoveries section in the plan file. Plan: .agents/plans/A25-string-pack-unpack.md
Bumps the suite-triage goal to 6/29 (tpack.lua now passes) and removes the string.pack/unpack/packsize deferral now that it's implemented. Plan: A25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement string.pack, string.unpack, string.packsize
Plan:
.agents/plans/A25-string-pack-unpack.mdGoal
Implement
string.pack/2,string.unpack/2,3, andstring.packsize/1per Lua 5.3 §6.4.2. These are the only stdlib functions that previously
raised
string.pack not yet implemented, blockingtpack.luaandparts of
strings.lua.Success criteria
string.pack,string.unpack,string.packsizeexist inlib/lua/vm/stdlib/string.exand are reachable from Lua(delegating to
Lua.VM.Stdlib.String.Pack)- [x]
<>=!(endian and alignment)- [x]
b B h H i I l L j J T(signed/unsigned ints, sized variants)- [x]
f d n(floats)- [x]
ss1-s8(length-prefixed string)- [x]
z(zero-terminated string)- [x]
x(padding byte)- [x]
X<op>(alignment to op's natural alignment)- [x]
c<n>(fixed-size string)- [x]
(space, ignored)tpack.luapasses (promoted to@ready_tests; suite count 5/24 → 6/24)mix testcount goes up by at least 5 (added 41 new unit tests intest/lua/vm/stdlib/string_pack_test.exs)Changes
The bulk of the implementation is in the new
Lua.VM.Stdlib.String.Packmodule. The format-string parser walks the format once and emits a list
of operations (
{:int, …},{:float, …},{:fixed_string, n},{:lstring, …},{:zstring},{:padding, n},{:align, n}); thepack/unpack/packsize drivers then evaluate that list while tracking
their own running byte position. This split is necessary because
variable-length ops (
s,z) advance the position by an amount onlyknown at evaluation time, and subsequent alignment requirements depend
on that runtime position (matches PUC-Lua's
getdetails/totalsizedesign).Discoveries
Documented in the plan file under
## Discoveries. Briefly:string.reversewas UTF-8-oriented, not byte-oriented. Fixedin scope —
tpack.luaexercises byte reversal of packed integerbytes via
s2:reverse(), so the bug was on the critical path.string.reprejected float counts. Lua 5.3's^always returnsa float, so
string.rep("c…", 2^3)was rejected as "number expected,got number". Fixed in scope by accepting floats with integer values.
variable-length ops (
s,z).unpackcast semantics for size > SZINT and size == SZINT matchPUC-Lua's
(lua_Integer)resbit-pattern reinterpretation. Notably,unpack("<J", pack("<j", -1)) == -1requires wrapping unsigned 8-bytevalues above 2^63-1 to their signed-64-bit equivalents.
1/0through
pack("f", …). Handled by detecting the four ±Inf bitpatterns on decode and remapping them to the Lua VM's
±1.0e308stand-ins (consistent with
Lua.VM.Executor.safe_divide/4).Verification
Out of scope (intentional)
string.formatimprovementsstring.packextensions beyond Lua 5.3 §6.4.2 (no custom format options)NaN through pack/unpack; consistent with the existing
safe_divide/4accepted-divergence note)