feat: renderer self-describes prefix-stability for SFT / RL / inference consumers

[hallerite]

Motivation

A renderer's output for render(messages[:k]) vs render(messages[:k+1]) can differ in non-obvious ways depending on the template's history-handling semantics. Today this knowledge is implicit — each renderer's emit_* helpers encode it, but consumers (SFT dataloaders, RL trajectory builders, inference incremental rendering) have to either hard-code per-template assumptions or assume the worst case.

Two concrete consumer problems this causes:

1. SFT loses intermediate reasoning on non-prefix-stable templates. build_training_sample does a single full-conversation render. For Qwen3/GLM5 defaults, that single render strips reasoning from every assistant turn except the last (history-stripping). The model is trained to produce reasoning only on the final turn. Workarounds today: set preserve_all_thinking=True (changes inference semantics too) or use build_incremental_token_mask (errors out on non-prefix-stable templates — IncrementalTokenizationError). Neither is great.
2. Inference bridges can't short-circuit. RendererClient._get_incremental_prompt_ids always calls bridge_to_next_turn, even when the renderer is fully prefix-stable and a naive concat would be correct. The bridge does real work it doesn't need to.

The renderer is the source of truth for these semantics — it has to be, because it implements the emit logic. Today it doesn't expose that truth.

Proposed API

A declarative property describing which append-boundaries preserve the rendered prefix:

# renderers/base.py
from dataclasses import dataclass
from typing import Literal

Boundary = Literal["tool", "user", "system", "developer"]

@dataclass(frozen=True)
class RenderStability:
    """Which append-boundaries preserve the rendered prefix.

    ``boundary in preserves_through`` means: for any messages M and a single
    appended message m where ``m["role"] == boundary``, ``render(M).token_ids``
    is a prefix of ``render(M + [m]).token_ids``. The bridge for that boundary
    is then a trivial "append new tokens" — no history transformation.

    Boundaries *not* in the set may still be appendable (via ``bridge_to_next_turn``),
    but the renderer transforms earlier tokens (e.g. strips reasoning from a prior
    assistant when a new user turn arrives).
    """
    preserves_through: frozenset[Boundary]

    @property
    def fully_stable(self) -> bool:
        return self.preserves_through >= {"tool", "user", "system", "developer"}


class Renderer:
    @property
    def stability(self) -> RenderStability: ...

Per-renderer values

Renderer	preserves_through	Notes
Qwen3 default	{"tool"}	Stable within a tool cycle; user boundary strips assistant reasoning.
Qwen3 + preserve_thinking_between_tool_calls=True	{"tool"}	Same — that flag only preserves reasoning within the current cycle, which is the cycle's default for emit.
Qwen3 + preserve_all_thinking=True	{"tool", "user", "system", "developer"}	Reasoning preserved across every boundary.
GLM5 / variants	analogous to Qwen3
Kimi K2, K2.5	TBD by template emit logic
DeepSeek V3, Nemotron 3, Laguna XS.2	TBD
GPT-OSS	TBD (Harmony format)
DefaultRenderer (Jinja)	frozenset()	Opaque template, assume nothing.

Stability is dynamic w.r.t. construction-time flags (preserve_all_thinking, preserve_thinking_between_tool_calls): the same renderer class can advertise different stability depending on init args. Cache key into shared pools already includes these flags (verifiers/clients/renderer_client.py:486-494).

Consumer impact

SFT (in prime-rl)

build_training_sample queries renderer.stability:

• fully_stable → current single-render path is correct.
• "tool" in preserves_through but not "user" (Qwen3/GLM5 default) → split conversation at user boundaries; for each assistant-terminated segment, produce a separate training sample. Each captures that turn's reasoning under inference-correct history-stripping. Dataloader fans out 1 conversation → N samples.
• empty (DefaultRenderer) → fall back to per-assistant render. Slowest, always correct.

The fan-out shifts batch sizing semantics: "batch of 32 conversations" becomes "batch of sum(stages_per_conversation) samples." Worth deciding upfront whether to expand at the dataloader level or pack N segments into one sample with attention isolation.

RL (verifiers / prime-rl orchestrator)

bridge_to_next_turn stays the canonical correctness path. When renderer.stability.fully_stable, the caller can skip the bridge and do a pure concat — an optimization, not a correctness change. Existing code keeps working.

Inference

Same — RendererClient._get_incremental_prompt_ids can branch on stability and skip the bridge dispatch for fully-stable renderers.

Open questions

1. Include "developer"? GPT-OSS already handles a developer role (gpt_oss.py:442) for Harmony / Responses-API messages. Including it keeps GPT-OSS first-class without consumer-side special-casing. Templates that don't use developer messages just never see them — the flag is moot for them.
2. Per-boundary bridge declarations? Could extend to RenderStability.bridge_required: frozenset[Boundary] so consumers know which bridges do non-trivial work vs. trivial-append. Today every renderer has one bridge_to_next_turn; this would be a separate signal.
3. Tools change as a boundary? Today tools is passed alongside messages. If the tool list changes between renders (e.g. tool added mid-trajectory), the system-section content changes and prefix-stability breaks regardless of message role. Should the API model that too? (Probably yes, as a separate stable_under_tools_change: bool.)

Implementation sketch

• Renderers side: add stability property to each renderer class. Most are one-liners returning a fixed RenderStability. Flag-aware ones (Qwen3, GLM5) compute it from their _preserve_*_thinking attributes.
• New module: renderers.stability exporting Boundary, RenderStability, helper constants for common cases (e.g. FULLY_STABLE, STABLE_IN_TOOL_CYCLE).
• Tests: parametrize the existing fixtures from tests/test_sampled_mask.py to also assert that render(M + [m]) extends render(M) exactly when m["role"] in stability.preserves_through. Catches drift between declared and actual behavior.

Related

• feat(base): add RenderedTokens.sampled_mask for SFT/RL parity #33 — sampled_mask AND in build_training_sample (closes the narrower "trains on <|im_start|>assistant\n scaffolding" issue)
• feat(base): per-message / per-role analytics on RenderedTokens #38 — per-message / per-role analytics; also lands message_indices/sampled_mask/message_roles on bridge output
• feat(renderer): expose preserve_*_thinking flags in RendererConfig prime-rl#2436 — exposes preserve_*_thinking flags through prime-rl's RendererConfig
• SFT renderer path trains on <|im_start|>assistant\n scaffolding (use renderers sampled_mask) prime-rl#2492 — narrower SFT scaffolding bug, fixed by feat(base): add RenderedTokens.sampled_mask for SFT/RL parity #33

[mvanhorn]

Implemented in #44 - adds the RenderStability enum on the base renderer and populates values for every shipped renderer module.

[CanadaApollo6]

I'd like to pick this up. I hit this exact failure mode porting a GRPO pipeline onto NeMo-RL, so I've already debugged most of the taxonomy here.

A few notes from reading the current code before I write anything, plus takes on the open questions.

Most of the table can be derived now. The sketch predates the thinking_retention refactor. The old preserve_all_thinking and preserve_thinking_between_tool_calls flags are gone (the typed configs raise on them. Every renderer now resolves effective_thinking_retention at construction and checks should_rerender_for_thinking_retention inside its bridge. That check is already a stability signal, just scoped to thinking and buried in the bridge:

• "all": never re-render, so the user boundary preserves the prefix
• "tool_cycle": re-renders on user-query boundaries, so preserves_through = {"tool"}
• "template": always re-renders, so frozenset()

So I'd compute RenderStability from effective_thinking_retention plus per-renderer structural facts, and hand-declare only the exceptions where instability isn't thinking-driven. One source of truth instead of two that drift. Longer term, the bridge's re-render decision could consult stability instead of the retention check directly. I saw #44 sketched the declarative version before the refactor landed; deriving the values avoids maintaining that table by hand.

Retention alone doesn't prove token-level stability, so the tests have to. Two breakage sources sit outside history transformation:

1. Boundary tokenization. build_trajectory_step already works around generation prompts diverging from the full render at token boundaries (\n vs \n\n when thinking is empty on Qwen3.5), and the Qwen3 emit path special-cases BPE merges across spans (.\n collapsing to one token). A renderer can be message-stable and still fail the check that render(M).token_ids is a strict prefix of render(M + [m]).token_ids.
2. Template shape effects, the class of failures the renderers blog post documents.

So every declared boundary should be backed by a parametrized test asserting exact token-prefix extension over the existing conversation fixtures. Declared stability is a guarantee; anything unaudited stays OPAQUE, which claims nothing and is safe by construction. A declaration API without those tests would just be an unverified promise in a correctness-critical path.

On the open questions:

1. "developer": include it. gpt_oss.py already routes developer messages, templates that never see the role pay nothing, and excluding it forces GPT-OSS consumers back into the special-casing this API exists to remove.
2. bridge_required: skip it for now. Under the derivation above, a boundary missing from preserves_through already implies the bridge does real work there. A second frozenset adds surface with no consumer today. Easy to add later.
3. Tools-change: yes, as the separate flag you suggest. Tools alter the system section, so a mid-trajectory tools change breaks the prefix regardless of role. stable_under_tools_change: bool (probably False everywhere today) keeps role-boundary semantics clean and makes the hazard explicit for trajectory builders.

Plan, if this direction works for you:

1. renderers/stability.py: Boundary, RenderStability, constants, and a from_thinking_retention() helper.
2. Renderer.stability property on base, derived from effective_thinking_retention; DefaultRenderer pinned OPAQUE (unknown Jinja); explicit overrides where needed.
3. Parametrized prefix-extension tests over the existing bridge/roundtrip fixtures, asserting declared boundaries extend exactly.
4. Short README section on when consumers can skip the bridge.

I'd leave consumer wiring (the build_training_sample fan-out in prime-rl, client short-circuits) for follow-ups to keep this reviewable. Can open the PR this week if the shape looks right.