Skip to content

feat(stdlib): implement string.pack, string.unpack, string.packsize#217

Merged
davydog187 merged 4 commits into
mainfrom
feat/string-pack-unpack
May 9, 2026
Merged

feat(stdlib): implement string.pack, string.unpack, string.packsize#217
davydog187 merged 4 commits into
mainfrom
feat/string-pack-unpack

Conversation

@davydog187
Copy link
Copy Markdown
Contributor

Implement string.pack, string.unpack, string.packsize

Plan: .agents/plans/A25-string-pack-unpack.md

Goal

Implement string.pack/2, string.unpack/2,3, and string.packsize/1
per Lua 5.3 §6.4.2. These are the only stdlib functions that previously
raised string.pack not yet implemented, blocking tpack.lua and
parts of strings.lua.

Success criteria

  • string.pack, string.unpack, string.packsize exist in
    lib/lua/vm/stdlib/string.ex and are reachable from Lua
    (delegating to Lua.VM.Stdlib.String.Pack)
  • All format options from Lua 5.3 §6.4.2 supported:
    - [x] < > = ! (endian and alignment)
    - [x] b B h H i I l L j J T (signed/unsigned ints, sized variants)
    - [x] f d n (floats)
    - [x] s s1-s8 (length-prefixed string)
    - [x] z (zero-terminated string)
    - [x] x (padding byte)
    - [x] X<op> (alignment to op's natural alignment)
    - [x] c<n> (fixed-size string)
    - [x] (space, ignored)
  • tpack.lua passes (promoted to @ready_tests; suite count 5/24 → 6/24)
  • mix test count goes up by at least 5 (added 41 new unit tests in
    test/lua/vm/stdlib/string_pack_test.exs)
  • No regression elsewhere (1585 → 1626 passing, 0 failures)

Changes

.agents/plans/A25-string-pack-unpack.md |  77 ++++-
lib/lua/vm/stdlib/string.ex             | 124 +++--
lib/lua/vm/stdlib/string/pack.ex        | 619 +++++++++++++++++++++++++ (new)
test/lua/vm/stdlib/string_pack_test.exs | 312 +++++++++++++ (new)
test/lua53_suite_test.exs               |   2 +-

The bulk of the implementation is in the new Lua.VM.Stdlib.String.Pack
module. The format-string parser walks the format once and emits a list
of operations ({:int, …}, {:float, …}, {:fixed_string, n},
{:lstring, …}, {:zstring}, {:padding, n}, {:align, n}); the
pack/unpack/packsize drivers then evaluate that list while tracking
their own running byte position. This split is necessary because
variable-length ops (s, z) advance the position by an amount only
known at evaluation time, and subsequent alignment requirements depend
on that runtime position (matches PUC-Lua's getdetails /
totalsize design).

Discoveries

Documented in the plan file under ## Discoveries. Briefly:

  1. string.reverse was UTF-8-oriented, not byte-oriented. Fixed
    in scope — tpack.lua exercises byte reversal of packed integer
    bytes via s2:reverse(), so the bug was on the critical path.
  2. string.rep rejected float counts. Lua 5.3's ^ always returns
    a float, so string.rep("c…", 2^3) was rejected as "number expected,
    got number". Fixed in scope by accepting floats with integer values.
  3. Alignment had to move from parse-time to runtime to handle
    variable-length ops (s, z).
  4. unpack cast semantics for size > SZINT and size == SZINT match
    PUC-Lua's (lua_Integer)res bit-pattern reinterpretation. Notably,
    unpack("<J", pack("<j", -1)) == -1 requires wrapping unsigned 8-byte
    values above 2^63-1 to their signed-64-bit equivalents.
  5. BEAM has no IEEE ±Infinity, but the suite round-trips 1/0
    through pack("f", …). Handled by detecting the four ±Inf bit
    patterns on decode and remapping them to the Lua VM's ±1.0e308
    stand-ins (consistent with Lua.VM.Executor.safe_divide/4).

Verification

mix format --check-formatted          # clean
mix compile --warnings-as-errors      # clean
mix test                               # 1626 passing, 0 failing, 30 skipped
mix test --only lua53                  # 6/24 ready: simple_test, api,
                                       #   bitwise, code, tpack (new), vararg

Out of scope (intentional)

  • string.format improvements
  • string.pack extensions beyond Lua 5.3 §6.4.2 (no custom format options)
  • Performance optimisation beyond "passes the suite without timing out"
  • Full IEEE NaN bit-pattern round-tripping (the suite doesn't exercise
    NaN through pack/unpack; consistent with the existing
    safe_divide/4 accepted-divergence note)

davydog187 added 4 commits May 9, 2026 05:53
Adds full Lua 5.3 §6.4.2 support for packing and unpacking binary data
via a new Lua.VM.Stdlib.String.Pack module. Handles all format options
including arbitrary integer widths (i1-i16), endianness directives
(< > =), platform alignment (!N, X<op>), fixed-size and length-prefixed
strings (c<n>, s<n>, z), and the IEEE ±Infinity bit-pattern round-trip
needed because the BEAM uses 1.0e308 as a finite stand-in for ±1/0.

Promotes tpack.lua from skipped to ready in the Lua 5.3 suite.

Two small in-scope discoveries: string.reverse was UTF-8-oriented
(broken for binaries containing NUL); string.rep rejected float counts
even though Lua 5.3's '^' always returns a float. Both surfaced via
tpack.lua, both fixed in the same commit. See Discoveries section in
the plan file.

Plan: .agents/plans/A25-string-pack-unpack.md
Bumps the suite-triage goal to 6/29 (tpack.lua now passes) and removes
the string.pack/unpack/packsize deferral now that it's implemented.

Plan: A25
@davydog187 davydog187 merged commit 781a617 into main May 9, 2026
4 checks passed
@davydog187 davydog187 deleted the feat/string-pack-unpack branch May 9, 2026 13:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant