Summary
Gasoline workflow loops can keep one OpenTelemetry trace ID for the full lifetime of a long-running workflow. Each loop iteration is instrumented as a child of the long-lived LoopBuilder::run / workflow span, so traces grow without a bound.
The current implementation is here:
https://github.com/rivet-dev/actors/blob/caed5835f/engine/packages/gasoline/src/builder/workflow/lupe.rs#L284
Evidence
Observed with Rivet Engine 2.3.0-rc.5 in self-hosted staging and production:
- One block-only Tempo response contained 12,205 Rivet spans over 65 minutes and required 50,389,617 inspected bytes.
- The visible iteration counters were already 7,650–7,778, showing that the trace belonged to a much older workflow parent.
- The largest operation groups were
txn (2,046), txn_attempt (2,045), and four signal-listening operations (1,269 each).
- Over 36 hours, Tempo emitted 247 oversized-trace warnings from 3 trace IDs in staging and 151 warnings from 39 trace IDs in production.
- The most repeated production trace had the same shape: 12,313 spans dominated by transaction and signal-loop operations.
- These traces repeatedly hit Tempo's trace-size guard during compaction and contributed to ingester/compactor memory pressure.
The current public main still has the same inherited iteration span, and I could not find an existing issue or PR for this behavior.
Suggested fix
Start each workflow iteration as a new root trace and preserve causality by linking it to the workflow span:
let parent_span_ctx = tracing::Span::current()
.context()
.span()
.span_context()
.clone();
let iteration_span =
tracing::info_span!(parent: None, "iteration", iteration=%previous_iteration);
iteration_span.add_link(parent_span_ctx);
let res = async {
// existing iteration body
}
.instrument(iteration_span)
.await?;
This bounds each trace to one workflow iteration while retaining a causal link to the long-lived workflow. I have this change locally and rustfmt --check plus cargo check -p gasoline pass.
Summary
Gasoline workflow loops can keep one OpenTelemetry trace ID for the full lifetime of a long-running workflow. Each loop iteration is instrumented as a child of the long-lived
LoopBuilder::run/ workflow span, so traces grow without a bound.The current implementation is here:
https://github.com/rivet-dev/actors/blob/caed5835f/engine/packages/gasoline/src/builder/workflow/lupe.rs#L284
Evidence
Observed with Rivet Engine
2.3.0-rc.5in self-hosted staging and production:txn(2,046),txn_attempt(2,045), and four signal-listening operations (1,269 each).The current public
mainstill has the same inherited iteration span, and I could not find an existing issue or PR for this behavior.Suggested fix
Start each workflow iteration as a new root trace and preserve causality by linking it to the workflow span:
This bounds each trace to one workflow iteration while retaining a causal link to the long-lived workflow. I have this change locally and
rustfmt --checkpluscargo check -p gasolinepass.