erts: Fix out-of-bounds read on distributed priority send of a fragmented message

[potatosalad]

Fix out-of-bounds read on distributed priority send of a fragmented message

• Heap-buffer-overflow receiving a large distributed priority message #11416

What happens

A distributed [priority] send larger than 32 KiB to an erlang:alias([priority]) triggers an out-of-bounds read while the receiving node saves the message: the emulator reads 8 bytes past a 40-byte allocation. It's deterministic on the first such message, on stock OTP with no third-party code. Under ASan it aborts with heap-buffer-overflow; on a normal build the read lands in unrelated memory and the node can crash.

Root cause

When a distributed message arrives fragmented, handle_altact_msg() (erts/emulator/beam/erl_proc_sig_queue.c) allocates a bare message reference and attaches the fragment separately:

mp = erts_alloc_message(0, NULL);      /* no embedded heap fragment */
...
mp->data.heap_frag = sig->hfrag.next;  /* fragment is SEPARATELY attached */

erts_alloc_message(0, NULL) returns a 40-byte ErtsMessage with no room for an embedded ErlHeapFragment.

The priority path then inserts that message through insert_prepared_prio_msg(), a thin wrapper that hard-codes the "combined" (embedded-fragment) case:

return insert_prepared_prio_msg_attached(c_p, tracing, sig,
                                         ERTS_MSG_COMBINED_HFRAG, ...);

So a message whose fragment is separately attached gets tagged ERTS_MSG_COMBINED_HFRAG, which claims the fragment is embedded in the message allocation. erts_save_message_in_proc() (erts/emulator/beam/erl_message.c:981) trusts that tag and reads the supposedly-embedded fragment's off_heap.first at offset 48 of the 40-byte region — hence the 8-byte over-read.

Only sends above the 32 KiB fragmentation threshold reach this branch, which is why smaller priority messages are fine.

The fix

erts/emulator/beam/erl_proc_sig_queue.c — pass the message's actual attachment through instead of reclassifying it as combined:

cnt += insert_prepared_prio_msg_attached(c_p, tracing, mp,
                                         mp->data.attached,
                                         ERL_MESSAGE_TERM(mp), token,
                                         next_nm_sig);

mp->data.attached is exactly what the branch above just set, so the fragment keeps its real ownership and the save path no longer reads a fragment that was never embedded. The non-fragmented priority paths are untouched and still go through insert_prepared_prio_msg().

Regression test

erts/emulator/test/signal_SUITE.erl — priority_messages_order only ever sent small priority messages before, so it never crossed the fragmentation threshold. Its distributed priority message is now 1 MiB:

Large = binary:copy(<<0>>, 1 bsl 20),
...
erlang:send(PrioAlias, {prio_msg, Large}, [priority]),

with the expected-message list updated to match. The case already runs twice — once against a local process, once against a ?CT_PEER() node — so the 1 MiB message crosses a real distribution link, and it still asserts the ordering of that message against the surrounding link/monitor/exit signals. Unpatched, the case aborts; patched, it passes.

How it was verified

Everything below ran against a Clang ASan build of this tree (otp_release=30 erts=17.0.4 emu_type=asan).

A/B on the minimal reproducer. Two builds differing in exactly one file (erts/emulator/beam/erl_proc_sig_queue.c), same two-node script: node v creates alias([priority]), node u sends 1 MiB with [priority], v acknowledges receipt.

	fix reverted	fix applied
node v exit	134 (SIGABRT)	0
node u exit	1 (timeout, peer died)	0
1 MiB delivered + acked	no	yes (1048576 bytes)
sanitizer report	yes	none

The over-read on the unpatched build, showing the read 8 bytes past the 40-byte allocation made in handle_altact_msg():

==427==ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 8 at 0x50400069ed80 thread T6
    #0 erts_save_message_in_proc   erts/emulator/beam/erl_message.c:981:51
    #1 beam_jit_remove_message     erts/emulator/beam/jit/beam_jit_common.cpp:1170:5

0x50400069ed80 is located 8 bytes after 40-byte region [0x50400069ed50,0x50400069ed78)
allocated by thread T6 here:
    #1 erts_alloc_message          erts/emulator/beam/erl_message.h:578:22
    #2 handle_altact_msg           erts/emulator/beam/erl_proc_sig_queue.c:6163:18
    #3 erts_proc_sig_handle_incoming erts/emulator/beam/erl_proc_sig_queue.c:6874:20

With the fix applied the same script produces no sanitizer output and both nodes exit 0.

OTP regression under ASan. signal_SUITE:priority_messages_order via ct:run_test/1: {1,0,{0,0}} (1 ok, 0 failed, 0 skipped), no sanitizer files.

Affected versions

Confirmed on OTP 29.0.4 (ERTS 17.0.4) and reproduced on master (OTP 30 dev, ERTS 17.0.4); also seen on 28.4.1. Priority messages are an OTP 28 feature, so 28.x and 29.x are affected and ≤27 is not.

[github-actions]

CT Test Results

    3 files    136 suites   50m 36s ⏱️
1 683 tests 1 627 ✅ 56 💤 0 ❌
2 326 runs  2 252 ✅ 74 💤 0 ❌

Results for commit 013ecef.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot

[rickard-green]

Thanks for the fix!

Please rebase this branch on top of OTP-28.5 and force push it to your fork, so that we can use your commit for patching of OTP 28 and OTP 29 as well. I tried to rebase it and it rebased cleanly if executing the following when the branch is checked out:

 $ git rebase --onto OTP-28.5 HEAD~1 fix/11416-priority-message-fragment

[potatosalad]

@rickard-green Rebased onto OTP-28.5 and force-pushed. Thanks!

[rickard-green]

Thanks!