Skip to content

Introduce table coding system for program memory savings - #70

Open
tpwrules wants to merge 5 commits into
dronecan:masterfrom
tpwrules:pr/table-coding
Open

Introduce table coding system for program memory savings#70
tpwrules wants to merge 5 commits into
dronecan:masterfrom
tpwrules:pr/table-coding

Conversation

@tpwrules

@tpwrules tpwrules commented Jul 26, 2024

Copy link
Copy Markdown
Collaborator

If the message is compatible, the DSDL compiler generates a table which describes its structure. If enabled in libcanard, an interpreter is then used to follow the table description and encode or decode the message. This occupies substantially fewer bytes per message as the table is much smaller than the C function previously used. This translates to a large reduction in program memory usage, especially if many different message types need to be coded.

The table can describe any structure of message supported by DSDL, however the quantities and sizes of specific aspects are limited by the datatypes chosen; see the code for details. Requires dronecan/dronecan_dsdlc#30 to actually generate the tables.

On top of Ardupilot master (a560f89b3), this system saves over 9K flash on Cube Orange, over 5K flash on Pixhawk1-1M, and 250-2000 bytes across MatekL431-based peripherals. DroneCAN-enabled bootloaders do not seem to profit however, so this should be disabled for them. No performance testing has been done yet, suggestions about how to go about it would be appreciated if deemed necessary. Also not sure if any of this needs to be documented further than already done, it is mostly invisible to the user.

Tested using the dronecan_dsdlc test scripts and Ardupilot SITL. Testing on real hardware remains to be done.

@tpwrules

Copy link
Copy Markdown
Collaborator Author

Removed fallthroughs that clang was whining about. Tested and GCC can optimize them anyway.

@tpwrules

Copy link
Copy Markdown
Collaborator Author

Still have some minor cleanups and polishing to do but I figure that will come after the first round of reviews.

If the message structure is compatible, the DSDL compiler generates a
table which describes its structure. If enabled, an interpreter is then
used to follow the table description and encode or decode the message.
This occupies substantially fewer bytes per message as the table is much
smaller than the C function previously used. This translates to a large
reduction in program memory usage, especially if many different message
types need to be coded.

This commit sets up all the hooks to enable and disable the system and
defines the interpreter functions. However, they don't do anything as
the system currently doesn't support any types.
Each table has a four byte header, plus some number of entries (4
bytes). Each field consumes one entry.

The maximum encoded message length in bytes, size of the C message
struct in chars, and number of table entries all are limited to 65535
for table coding to be possible.

Also supports nested compound types containing these fields.
Static arrays have a 2-entry header (8 bytes), plus the entries
describing the contents of the array.

Array contents cannot consist of more than 256 entries (including header
entries and entries describing nested types). The array length cannot
exceed 65535.
Dynamic arrays have a 3-entry header (12 bytes), plus the entries
describing the contents of the array.

Array contents cannot consist of more than 256 entries (including header
entries and entries describing nested types). The maximum array length
cannot exceed 65535.
Unions have a 2-entry header (8 bytes), plus 1 entry for each field,
plus the entries describing each field.

The maximum number of fields is 255. Each field cannot consist of more
than 255 entries (including header entries and entries describing nested
types).

This completes the list of DSDL types.

@tridge tridge left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this review was AI-generated (Claude Code), then cross-checked by a
second model and by me. Both defects below were reproduced against the actual
code rather than inferred by reading it, and the repro steps are included so you
can verify them independently. Please treat the reasoning with the scepticism
that warrants.

Two defects in tableDecodeCore(). Together they mean the table decoder is not
currently equivalent to the inline decoder in either direction: it rejects
traffic the inline decoder accepts, and accepts traffic the inline decoder rejects.

Measured on the real DSDL set, same generated code, only CANARD_ENABLE_TABLE_DECODING
differing:

case inline decoder table decoder
empty TAO esc.RawCommand (0-byte payload) accepted, len=0 rejected
param.GetSet_req with string_value.len=255 rejected accepted, len=255

Defect 1 — union member decode discards the error return

canard.c:2048,
in case CANARD_TABLE_CODING_UNION:

if (num_entries && tag == union_tag) {
    tableDecodeCore(union_entry, union_entry+num_entries-1, transfer, bit_ofs, p, tao);
}

Every other recursive call site does if (tableDecodeCore(...)) return true;.
Here the result is dropped.

This matters because the dynamic-array path
(canard.c:1977
and :1986)
writes the decoded length into the message struct before validating it:

canardDecodeScalar(transfer, *bit_ofs, len_bitlen, false, len_p);
...
if (elems > max_len) {
    return true;            // too late: *len_p already holds the bad value
}

So a union member containing a dynamic array can leave an out-of-range length in
the struct, and the union handler then swallows the error and reports success.

libcanard itself does not read or write out of bounds here — it bails before
the element loop. The hazard is that it hands the caller a struct whose .len
exceeds the buffer, having said "no error". Any consumer that iterates on .len
without re-validating it then reads out of bounds.

Reachability — plain classic CAN is enough

I initially thought this needed CAN FD to force transfer->tao == false. It does not.
In param.GetSet request and response, Value is not the tail field, so its
string_value subtable is generated as explicitly non-TAO and the explicit-length
path runs even with tao == true.

A 3-byte classic-CAN GetSet_req — 13-bit index, 3-bit union tag = 4
(string_value), 8-bit length = 255, payload {0x00, 0x04, 0xFF} — decodes as:

err=0  value.union_tag=4  value.string_value.len=255   (buffer is 128 bytes)

The inline decoder rejects the identical input.

To be precise about severity: the packet itself is deliberately malformed — a
claimed length of 255 for a max-128 field — so this is not something routine
traffic produces. But it is network-reachable: param.GetSet is a routine
service that AP_Periph registers and dispatches (Tools/AP_Periph/can.cpp), and
the payload fits in a single classic-CAN transfer. No CAN FD, and no special
node state, is needed to deliver it.

Affected real messages: param.GetSet_req, param.GetSet_res (and param.Value
as the nested type). enumeration.Indication is not affected — its union is
NumericValue, which has no dynamic array.

And to avoid overstating it: I checked the ArduPilot consumers and could not find
a current caller that turns this into an out-of-bounds access
. AP_DroneCAN.cpp
copies the string with sizeof(val) rather than .len; Tools/AP_Periph/can.cpp
bounds-checks name.len against AP_MAX_NAME_SIZE and the crafted request has
name.len == 0 anyway; the Xacti mount callbacks clamp or require an exact length.
So this is a broken decoder invariant and a latent hazard for future or third-party
callers, not a demonstrated live OOB in ArduPilot today. Combined with the PR being
unmerged and table decoding off by default, I'd treat it as "fix before enabling",
not as an emergency.

Fix:

if (num_entries && tag == union_tag) {
    if (tableDecodeCore(union_entry, union_entry+num_entries-1, transfer, bit_ofs, p, tao)) {
        return true;
    }
}

Verified: the crafted GetSet_req is then correctly rejected.

Separately, and independent of the return-propagation fix, this would be safer if
the length never reached the struct while invalid. Note it isn't simply a matter
of moving the existing check — elems is currently read back out of *len_p
after canardDecodeScalar writes it. The fail-safe form is: decode into a local
uint8_t/uint16_t, validate against max_len, and only then assign to *len_p.

Defect 2 — unsigned underflow rejects valid empty TAO arrays

canard.c:2000:

uint32_t max_bits = (transfer->payload_len*8)-7;

With payload_len == 0 this underflows to 0xFFFFFFF9, so the loop is entered for
what should be an empty tail array. It terminates via the !max_len-- guard and
returns an error.

Consequence: a zero-length uavcan.equipment.esc.RawCommand — a legitimate
message — is accepted by the inline decoder (err=0, cmd.len=0) and rejected by
the table decoder (err=1).

Guard the length before subtracting, e.g. only enter the TAO loop when
transfer->payload_len * 8 > *bit_ofs + 7.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants