Propose Rust backend design by jasongraffius · Pull Request #268 · google/emboss

jasongraffius · 2026-07-07T17:08:55Z

Proposal and design for a Rust backend for Emboss

EricRahm

Love this, thanks for drafting it up! I didn't do a super deep review, but just added some conversation starter questions.

EricRahm · 2026-07-07T18:41:25Z

+    *   If the storage is too small to read the presence condition: returns `Err(Error::OutOfBounds)`.
+    *   If the condition is successfully read and evaluates to `false`: returns `Ok(None)` (even if the storage is too small to hold the field itself).
+    *   If the condition is `true` and storage is valid for the field: returns `Ok(Some(FieldView))`.
+    *   If the condition is `true` but storage is too small for the field: returns `Err(Error::OutOfBounds)`.


Do we want to distinguish this somehow from "too small to read condition"?

I don't know that there's much use in doing that, in both cases its "you need more storage or you haven't seen the whole packet yet" so the remedy should be the same.

jasongraffius

Thanks, added some comments, I'll update to cover some of the questions here

reventlov

Overall, I'm excited to see this, especially since I recently started work on a primarily-Rust project.

reventlov · 2026-07-07T20:10:45Z

+*   When a nested writer is accessed, the parameters are propagated implicitly because the `NestedStorage` holds the parent writer (which contains the parameters) by value.
+
+#### 8. Alignment and LLVM Optimizations
+Unlike C++ which tracks buffer alignment in the template types, the Rust backend delegates alignment optimization to the compiler.


First, I agree with the decision to skip over Aligned reader/writer types, at least for now: they add complication to end user code that is rarely worth it in practice: the only widely-used target with alignment requirements that I know of, in 2026, is Web Assembly, which has minimal overlap with Emboss's uses.

(There are still a few PowerPC, MicroBlaze, and MIPS processors in use, but they're much more niche. Plus, the vast majority of uses are totally fine with the slight performance hit of alignment-neutral code even on platforms where it matters.)

(OK, so technically there are a few x86 instructions that have alignment requirements, but I think no current compilers emit them, and their unaligned counterparts have nearly the same speed on the last few generations of x86 processors.)

The justification for having Aligned at all on the C++ side is to hit the marketing benchmark of "equal the performance of packed structs." (Which also requires using the Unsafe methods.)

Second, rustc cannot do any better than clang or gcc, for the same reasons: the compiler cannot always infer the strictest alignment of the base address of a view. In C++, the Aligned view types let the programmer force the compiler to assume a certain base alignment, even if that base alignment is not deducible. (Generally, the base alignment is not deducible when receiving a pointer from some external source, and that is true with &[u8] just as much as char *.)

Ah got it, that's a good point, I'll fix up the wording here, but yeah for a first pass we probably want to skip the Aligned stuff for now. FWIW I believe that an AlignedStorage would be implementable by a client as a Storage implementation and should work correctly after monomorphization.

reventlov · 2026-07-07T20:21:22Z

+A Rust backend for Emboss should satisfy the following requirements:
+
+1.  **Zero-copy**: Accessing fields should not require copying data into intermediate structures.
+2.  **Efficiency**: Generated code should compile to efficient machine code, equivalent to hand-written unsafe pointer manipulation but with safety checks.


I think I would not make this level of efficiency a first-version requirement. Real-world experience, including on microcontrollers, is that 99% of Emboss-using code does not need absolute maximum performance. Better to get something that covers most cases first, and only then start taking on the few ultra-performance-critical bits.

Also, in many cases, "absolute maximum performance" means that the end developer has to provide hints (and in Rust, probably go unsafe).

See my comments below about the proposed type states and about C++ Aligned view types.

Fair point, that's true, it's not quite the same level of efficiency. When experimenting, I did try to get to the point that properly used generated code compiled down in assembly to equivalent hand-written implementation, but yeah there's lots of edge cases this would get bogged down on if we needed to actually hit that level across the board. I'll update to clarify!

reventlov · 2026-07-07T20:23:39Z

+    .into_struct(); // Consumes writer, returns the base struct view
+```
+
+### Compile-Time State Tracking (Typestates)


My high-level comment about this section is that I would plan to launch with only (the equivalent of) UncheckedState, and then worry about adding more states (and more complication to end user code) later, and only in cases where the performance really matters. Especially since this all falls under the umbrella of "could be done automatically by a sufficiently smart compiler."

The main reason I came up with the typestate stuff here is because a common paradigm I've seen used for Emboss stuff is "check complete, then write stuff (unchecked or panicking, not error checking), then check okay" not realizing an earlier write could make a later write invalid (either silent UB for the unchecked or crash for panic-checked).

That said, it is technically an optimization, only offering the try_write and try_read fallible options would be a good starting point. I think I'll reconfigure this to just launch with the try_ ops without a typestate, and offer the stateful views as a convenience later.

Another reason I liked the typestate is that you can declare your requirements "This function requires a fully complete and validated XyzStruct" means a lot less boilerplate and missed/forgotten checks

Yeah, just to be clear, I'm not saying "this is a bad idea" or "don't do this;" I'm only suggesting that you defer it until you have something working at all.

Yep yep! In agreement here, I'm gonna reconfigure this doc to move this to a future work section.

reventlov · 2026-07-07T21:06:24Z

+
+A view or writer can be in one of several states:
+*   `UncheckedState`: Initial state. Fields cannot be accessed directly without runtime bounds checks. Read/write accessors in this state perform dynamic bounds checking on the fly.
+*   `MinimallyCompleteState`: The buffer is known to be at least large enough for the minimum size of the structure.


In terms of access patterns, this state seems mostly unnecessary? It would only allow unchecked access to a somewhat arbitrary set of fields (i.e., those that are static + those that are close enough to the start of a "union" that they're always covered).

That starts getting into "how much do you want to explode the state space?". You could theoretically get very, very fine-grained.

Yeah, I figured it was a happy medium between fine-grained explosion and fully coarse, like a common case of access to a known-good fixed header or similar. Honestly this is basically me trying to get Rust to work like a dependently typed system in a way... I'm inclined to keep it as it doesn't add much complexity itself, but you're right there does need to be a fairly arbitrary "where do you stop" line if you add something like this.

reventlov · 2026-07-07T21:13:38Z

+
+### Compile-Time State Tracking (Typestates)
+
+To avoid runtime overhead while ensuring safety, we propose using a typestate pattern to track the validation and completeness status of a view at compile time.


A DynamicSizeIsKnownState might be useful here, as it would let you elide bounds and existence checks when reading fields for size computations.

This was basically my intent with something like MinimallyComplete, but now that I think about it, the dynamic size may require more than just MinimallyComplete (I'm considering a jagged 2D array where each array embeds its length, which is dependent on an outer length)

reventlov · 2026-07-07T21:59:31Z

+2.  **Efficiency**: Generated code should compile to efficient machine code, equivalent to hand-written unsafe pointer manipulation but with safety checks.
+3.  **Safety**: The generated API must be memory-safe. Any use of `unsafe` must be internal to the runtime, minimized, and thoroughly verified.
+4.  **Idiomatic Rust**: The API should feel natural to Rust developers, utilizing features like `Result`, `Option`, and traits.
+5.  **`no_std` Support**: The runtime and generated code must support `no_std` environments for use in embedded systems or kernels.


I don't recall whether no_std implies "no heap memory allocation," but if not it seems like that would be important to add here.

Also, similar to C++, I would only support no_std for "core" functionality, and let things like text serialization/deserialization require std.

Yeah it's somewhat orthogonal, but no alloc is also my intent, I'll add that.

And yeah, agreed, I'll update this to clarify that "no_std support" doesn't mean we can't have things that require no_std just there must exist a configuration that supports no_std.

reventlov · 2026-07-07T22:19:04Z

+*   `SHOUTY_CASE` for enum values and constants maps to Rust `SHOUTY_CASE` constants.
+
+#### Naming Conflict Resolution
+Emboss structures can have fields with arbitrary names. To avoid naming conflicts between generated field accessors and built-in helper methods (like `size_in_bytes` or `storage`), helper methods are defined on traits (e.g., `emboss_runtime::Struct`).


Not quite arbitrary: most Rust keywords are banned. However, the keywords added in the 2018 and 2024 editions are not banned, so you will need a strategy to handle them.

(You'd have to look up the internal change history to figure out when I actually made that list, but I feel it might have been before Rust 1.0? It's missing async, await, and dyn, where were reserved but unused sometime before the 2018 edition, and it has alignof, which seems weird.)

Ah yes, I actually meant to include that, we want to carve out keywords that aren't in the banlist for r#keyword treatment, the only ones you can't use that for are covered by the banlist.

reventlov · 2026-07-07T22:42:38Z

+3.  **Safety**: The generated API must be memory-safe. Any use of `unsafe` must be internal to the runtime, minimized, and thoroughly verified.
+4.  **Idiomatic Rust**: The API should feel natural to Rust developers, utilizing features like `Result`, `Option`, and traits.
+5.  **`no_std` Support**: The runtime and generated code must support `no_std` environments for use in embedded systems or kernels.
+


These are more of policy questions than design questions, but they have some design implications:

What versions of Rust are you planning to support, and what is the plan for how that list changes over time? I.e., what will you support "at launch," and what is the Rust equivalent of "until 10 years after the release of the following standard" on the C++ side?

My gut feeling is that Rust users are a lot less likely to be tied to ancient toolchains than embedded C++ users, so a much shorter (say, 3-5 year) support window may be fine. Also, it's fine to only target, say, the Rust 2025 edition at launch, although I don't think I saw anything in here that needs anything that modern.

reventlov · 2026-07-07T22:51:52Z

+*   `UncheckedState`: Initial state. Fields cannot be accessed directly without runtime bounds checks. Read/write accessors in this state perform dynamic bounds checking on the fly.
+*   `MinimallyCompleteState`: The buffer is known to be at least large enough for the minimum size of the structure.
+*   `CompleteState`: The buffer is large enough for the current dynamic layout of the structure. Always-present fields can be read/written without runtime bounds checks.
+*   `AlwaysCompleteState`: The buffer is large enough for the maximum possible size of the structure.


Note that AlwaysCompleteState does not imply that sub-structures are actually complete, so any field access for a dynamically-sized field puts you back into UncheckedState. Lots of real-world messages have degenerate encodings where a "payload length" field can have a value that is too short for the actual payload.

I'm trying to think of the implications of this... like a struct containing a sub-struct in a payload, that sub-struct is sized dynamically by some other field, and therefore IsComplete is false. Am I getting that right?

Doesn't have to be that complicated. If you have:

struct Message: # ... fields ... 8 [+2] MessageType message_type (mt) 10 [+2] UInt message_length (len) if mt == MessageType.Foo: 12 [+len] Foo foo if mt == MessageType.Bar: 12 [+len] Bar bar # ... etc ...

When message_length is 0, you can have infinite bytes available in Message's storage, but foo/bar/etc. will never be complete.

Right right, that's the same scenario, no? Foo/Bar as the substruct, len as the dynamic size by some other field, the inner structs can be incomplete while the outer is complete.

But yeah, that's a good point this should be called out, in the typestate model, dynamically sized sub-structs need to have their state downgraded on access.

I may have misread what you said; I thought you were saying that one more level of wrapping was needed. (Like that Foo needed to have a dynamically-sized sub-structure.)

reventlov · 2026-07-07T23:03:05Z

+
+#### 6. Dynamic Writes (State-Stable Structs)
+
+For complex imperative updates where the developer wants to avoid typestate transitions entirely, they can use the writer in `UncheckedState` directly. A struct or writer in a state-stable typestate (such as `UncheckedState` or `MinimallyCompleteState`) is referred to as a **state-stable struct** (or state-stable writer).


To avoid confusion, I would not call the read-only types structs: they do not hold their data inline the way that an actual struct does. I would either stick with "view" (by analogy with relational database views) and disambiguate with "real-only view" and "read-write view," or consistently use "reader" and "writer."

By making the storage generic, I was hoping to actually permit the "carries storage with it" style structs. If you have Simple6ByteStruct::new([u8; 6]), for instance, that would give you a Simple6ByteStruct<[u8; 6]>. So the difference between a view and owning is whether the storage is a view or owning, and the struct does "own" its storage, just that that storage may itself be a borrow or other referential type.

I see. That's a good goal, but I would still avoid the name "struct" for the generic type; let someone make a type Simple6ByteStruct = Simple6ByteSomething<[u8; 6]>; alias for the inline storage version.

EricRahm reviewed Jul 7, 2026

View reviewed changes

jasongraffius commented Jul 7, 2026

View reviewed changes

Propose Rust backend design

22ff619

jasongraffius force-pushed the rust-design branch from 4fcf660 to 22ff619 Compare July 7, 2026 21:36

reventlov reviewed Jul 7, 2026

View reviewed changes


		### Compile-Time State Tracking (Typestates)

		To avoid runtime overhead while ensuring safety, we propose using a typestate pattern to track the validation and completeness status of a view at compile time.


		#### 6. Dynamic Writes (State-Stable Structs)

		For complex imperative updates where the developer wants to avoid typestate transitions entirely, they can use the writer in `UncheckedState` directly. A struct or writer in a state-stable typestate (such as `UncheckedState` or `MinimallyCompleteState`) is referred to as a state-stable struct (or state-stable writer).

Uh oh!

Conversation

jasongraffius commented Jul 7, 2026

Uh oh!

EricRahm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jasongraffius left a comment

Choose a reason for hiding this comment

Uh oh!

reventlov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reventlov Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

reventlov Jul 7, 2026 •

edited

Loading