Skip to content

fix(gasoline): bound traces from long-lived workflow loops #5671

Description

@the-company-company-staging

Summary

Gasoline workflow loops can keep one OpenTelemetry trace ID for the full lifetime of a long-running workflow. Each loop iteration is instrumented as a child of the long-lived LoopBuilder::run / workflow span, so traces grow without a bound.

The current implementation is here:

https://github.com/rivet-dev/actors/blob/caed5835f/engine/packages/gasoline/src/builder/workflow/lupe.rs#L284

Evidence

Observed with Rivet Engine 2.3.0-rc.5 in self-hosted staging and production:

  • One block-only Tempo response contained 12,205 Rivet spans over 65 minutes and required 50,389,617 inspected bytes.
  • The visible iteration counters were already 7,650–7,778, showing that the trace belonged to a much older workflow parent.
  • The largest operation groups were txn (2,046), txn_attempt (2,045), and four signal-listening operations (1,269 each).
  • Over 36 hours, Tempo emitted 247 oversized-trace warnings from 3 trace IDs in staging and 151 warnings from 39 trace IDs in production.
  • The most repeated production trace had the same shape: 12,313 spans dominated by transaction and signal-loop operations.
  • These traces repeatedly hit Tempo's trace-size guard during compaction and contributed to ingester/compactor memory pressure.

The current public main still has the same inherited iteration span, and I could not find an existing issue or PR for this behavior.

Suggested fix

Start each workflow iteration as a new root trace and preserve causality by linking it to the workflow span:

let parent_span_ctx = tracing::Span::current()
    .context()
    .span()
    .span_context()
    .clone();
let iteration_span =
    tracing::info_span!(parent: None, "iteration", iteration=%previous_iteration);
iteration_span.add_link(parent_span_ctx);

let res = async {
    // existing iteration body
}
.instrument(iteration_span)
.await?;

This bounds each trace to one workflow iteration while retaining a causal link to the long-lived workflow. I have this change locally and rustfmt --check plus cargo check -p gasoline pass.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions