feat(telemetry): span the transition pipeline end to end, and remove the redundancy it exposed

[yilmaztayfun]

What this is

Two threads that turned out to be the same thread.

The first was the ask: make every time-consuming mechanism visible in the trace tree. Before this, a transition showed as a transaction with a handful of spans under it, and everything that actually cost time — validation, context load, task input/invoke/output compilation, subflow mapping, locks, component reads — happened inside gaps. A reader could see that a hop took 400 ms and had no way to learn where it went.

The second thread was what the spans then showed. With the tree filled in, a single business request turned out to resolve the same workflow definition 3+ times per hop, load the same instance twice, validate the same payload schema twice, and build a fully populated TransitionExecutionContext only to throw it away. So the second half of this PR removes that redundancy — which was only findable because of the first half.

Spans added

Named so the tree reads without opening anything: the subject is in the span name, details are tags.

Span	Covers
Step.{Name}	every pipeline step, with vnext.step.order / vnext.step.outcome
Transition.LoadContext, Instance.Load	context construction and the instance read behind it
Transition.Validate, Transition.ValidatePolicy	schema validation and the execution policy separately
Task.Invoke, Invoke.{taskType}/{taskKey}, CacheAside.{Read,Write}/{key}	the Execution service, which previously emitted no spans of its own at all
Script.Compile, Script.Execute, Script.ResolveHelpers	Roslyn compile vs. execution, told apart
Cache.Get/{key}, Cache.GenerationGet/{key}	component body read and the generation-token round trip in front of it
Lock.Acquire/{lockKey}, Lock.Release/{lockKey}	with vnext.lock.kind = status | chain
Instance.AppendData, Uow.Commit, Events.PublishDeferred	the write side
Transition.Continuation/{mode}, Transition.Settle	the stretch after the pipeline that used to be invisible
FanOut.Item[i]	per item, with batch summary tags on the parent

Three refinements worth calling out because they change what a reader sees:

• A step that did no work emits no span. StepOutcome.ContinueNoWork() clears Recorded, so a profile-excluded or short-circuited step leaves no row. A 20-row tree of mostly no-ops is worse than a 6-row tree of real ones.
• The transaction is named after its transition (TransitionJob.Execute/{key}), which made the transition/{key} child node redundant — it is gone.
• Cache and lock spans carry their subject in the name, not only in tags.

All of this is always on, not gated behind Verbose. That works without touching Aether because BusinessSpanFilterProcessor suppresses only [-prefixed DisplayNames at export — verified, not assumed.

Behavior changes a reviewer should actually check

These are the risky parts. Everything above is additive; these are not.

1. IWorkflowContext is gone. It was added for a SchemaValidationAttribute aspect in d680b4a6; that aspect was deleted in 23a7acbf and the context has been orphaned since — 2 writers, 3 readers, ~15 test doubles. Its three readers now take the workflow from their caller: PostCommitParentMutationService from PostCommitParentSnapshot, InstanceDataWriteService from a Workflow? parameter, and InstanceCommandAppService's own memo is deleted outright — it compared only Key, so a pinned-version request could silently get a different version back.

2. A resolved workflow is carried instead of re-resolved. WorkflowExecutionContext.ResolvedWorkflow is [JsonIgnore] transport-only state. No version guard is needed and the reason is worth understanding before reviewing: the carried definition was resolved from the same context object's Domain/WorkflowKey/WorkflowVersion, and the consumer asks with exactly those fields. The deleted memo was wrong precisely because it lacked that property.

3. Intake admits from the projection, not the aggregate. The full GetActiveAsync in InstanceCommandAppService existed only to build a context for a validation that AsyncTransitionStrategy already performs before enqueue. Both are gone; admission now reads InstanceExecutionSnapshot. Two things guard this and both are pinned by tests: InstanceExecutionSnapshot.IsTerminal was added because Instance.IsCompleted counts Faulted and Passive too and the snapshot's IsCompleted did not (a faulted instance would have been admitted), and the InstanceNotFound error is reproduced verbatim — my first version returned Instance:100013 where the repository returns Instance:100017, and a test caught it.

4. Payload-schema validation has a single owner. The sync path gained the schema check it never had, conditioned on !context.IsPreReserved — the same "validated at accept" invariant already documented in TransitionPipeline.

5. GenerationMemoSeconds now defaults to 5, not 0. This one is a policy decision, not a tuning change, and it is the only item here that trades correctness for latency: the memo caches the token, so a bump written by another pod stays invisible on this pod for up to 5 s. L1 does not share this exposure — an L1 key embeds the token, so L1 cannot go stale. Set it to 0 where a publish must be cluster-visible immediately.

Measurements

From one business request traced end to end (2dc8b6b7, 39 transactions, warm runtime):

• Time inside SyncTransitionStrategy that no child span accounted for: 976 ms → 14 ms. Residual unattributed across all transaction roots is 266 ms (9.7%), most of it the HTTP entry and job dispatch edges.
• Component reads: 74 reads for 14 distinct components, 12 distinct generation tokens, all l1=false — 148 Redis round trips. This is what motivated item 5 and it is what the L1 warm-up phase will target next.
• Wall-clock share, parallelism accounted: external amorphie-contract 45.5%, component cache reads 21.1%, DB (commit + instance load) 9.4%, locks 0.9%.

The lock number is the one to notice: 0.9% of wall clock across 28 transitions is the busy-as-mutex design behaving as intended, and it is now measurable rather than argued.

Config

AdditionalSources extended across both hosts and both workers — without it the new ActivitySources produce spans that are never exported, which is a silent failure. The local etc/docker collector filter was renamed and broadened to filter/dapr-internals. The orchestration host's ComponentCache.GenerationMemoSeconds: 0 override is removed now that the default carries the intended value.

Testing

dotnet build vnext.sln clean. Full suite run and the failing-test name set compared against the master baseline (master carries 191 pre-existing failures, mostly AmbientServiceProvider leakage across parallel collections) — no branch-only failures. New tests cover the carried-workflow path, schema-validation ownership, snapshot admission including the not-found/Faulted/Passive/Busy matrix, and the continuation/settle/generation spans.

Not done

• End-to-end Jaeger/Elastic verification of hop-chain re-parenting is still open. The spans are verified individually and in aggregate from Elastic queries; nobody has sat down with the waterfall and confirmed every subflow hop nests where it should.
• L1 stays disabled deliberately — lazy warm-up cost per pod is too high right now, and it is being handled as its own phase.
• L1 has no eviction/capacity observability. Worth closing before that phase, or "we turned L1 on and the hit rate is low" will be undiagnosable.
• vnext-docs/docs/components/tasks/fan-out.md:530 still says FanOut.Item is verbose-only. Stale since 3255c457, different repo.

🤖 Generated with Claude Code

[sonarqubecloud]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 80%)
D Security Rating on New Code (required ≥ A)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE