Background-job span gating, deferred arm handle, and jittered poll pacing - #96
Merged
yilmaztayfun merged 3 commits intoAug 21, 2026
Merged
Conversation
…stic BackgroundJob.Schedule, .Schedule.OneShot, .Schedule.Delete and .Dispatch called Source.StartActivity directly, bypassing the Verbose gate that the rest of the infrastructure instrumentation goes through. They therefore exported in the default Business profile, adding depth to traces that are already deep with transition chains and subflows without telling the reader anything the job's own execution span does not. Route them through InfrastructureActivitySource.StartDiagnosticActivity, which is the existing mechanism for exactly this. BackgroundJob.Execute is deliberately left alone: it is a real service boundary and the ambient parent of the work the job performs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A caller that must persist a job inside a critical section had no way to keep the scheduler round-trip out of it. EnqueueAsync either armed inline or on the ambient unit of work's completion, and both land inside the caller's lock. Measured on a workflow accept path, that external call WAS the lock hold time under load (p50 214 ms of a 198 ms hold), serialising every other request on the same instance behind it. EnqueueWithDeferredArmAsync persists with the same semantics as directly: true — the row lands Scheduled, an arm failure rolls it back to Pending for the arming poller — and returns an IBackgroundJobArmHandle instead of calling the scheduler. The handle closes over the scheduler arguments captured at enqueue time, so arming later costs one scheduler call: no job-row read, no extra status write. EnqueueAsync's body moved to a shared private core; its behaviour is unchanged. This commit also gates the Enqueue/Update/Delete producer spans as diagnostic, for the same reason as the scheduler and dispatch spans: they live in this file and splitting them into their own commit is not worth the churn. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… fleet Two problems that only bite with several replicas, which is how these workers run (10 each in nonprod and prod). No jitter. The backoff was a deterministic doubling, so pods started together — what a rolling deployment produces — poll in lockstep. That costs twice: the fleet loses the natural staggering that makes N replicas pick work up roughly N times sooner than one, and every tick becomes a burst of simultaneous claim queries over the same rows. Every delay is now jittered by +/-25%, and each loop takes a random startup offset within its idle interval so the first pass is spread too. Errors jumped straight to MaxPollingInterval. One transient fault — a brief database hiccup, a single poison message — stalled every replica for a full maximum interval, right when it was least affordable. An error now backs off one step like an empty round, floored at the idle interval so a hard failure is not retried at the busy cadence, and repeated errors still escalate to the cap. The pacing rules move into PollingDelay, shared by the outbox, inbox and background-job arming loops. The arming loop has no adaptive backoff by design — an unarmed job must be claimed within a bounded time — which makes jitter the only thing keeping its replicas apart. The existing AdaptivePollingTests exercised a private copy of the delay arithmetic rather than the production code, so a change to the real rules could not fail it. They now drive PollingDelay itself, with coverage for jitter, the error floor and the startup offset. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reviewer's GuideIntroduces a deferred background job arm handle and API, gates background job producer/scheduler/dispatcher spans as diagnostic-only, and centralizes adaptive, jittered polling delay logic (including startup offsets and improved error handling) for background job arming, inbox, and outbox services with updated tests. Sequence diagram for deferred background job arm handlesequenceDiagram
actor Caller
participant BackgroundJobService
participant JobStore
participant Scheduler as DaprJobScheduler
participant DeferredArmHandle
Caller->>BackgroundJobService: EnqueueWithDeferredArmAsync(handlerName, jobName, payload, schedule, ...)
activate BackgroundJobService
BackgroundJobService->>BackgroundJobService: EnqueueCoreAsync(handlerName, jobName, payload, schedule, metadata, failurePolicyOptions, directly: true, jobId, kind, deferArm: true, cancellationToken)
BackgroundJobService->>JobStore: SaveAsync(jobInfo, cancellationToken)
JobStore-->>BackgroundJobService: job persisted with Status Scheduled
BackgroundJobService-->>Caller: DeferredArmHandle(JobId, ArmAsync)
deactivate BackgroundJobService
Note over Caller,DeferredArmHandle: later, outside critical section
Caller->>DeferredArmHandle: ArmAsync(cancellationToken)
activate DeferredArmHandle
DeferredArmHandle->>BackgroundJobService: ArmNowAsync(handlerName, jobName, schedule, payloadBytes, failurePolicyOptions, JobId, cancellationToken)
BackgroundJobService->>Scheduler: ScheduleJobAsync(...)
Scheduler-->>BackgroundJobService: result
BackgroundJobService-->>DeferredArmHandle: Task completed (or row rolled back to Pending)
deactivate DeferredArmHandle
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
Up to standards ✅🟢 Issues
|
| Metric | Results |
|---|---|
| Complexity | 27 |
| Duplication | 0 |
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three related changes in the background-job and polling area, all driven by measurements taken while investigating why workflow transitions were slow and occasionally stranded.
vNext needs the
IBackgroundJobArmHandleAPI from this PR, so a release is required before the vNext side can be merged.1. Gate
BackgroundJob.*producer spans as diagnosticEnqueue,Update,Delete,Schedule,Schedule.OneShot,Schedule.DeleteandDispatchcalledSource.StartActivitydirectly, bypassing the Verbose gate the rest of the infrastructure instrumentation uses. They exported in the default Business profile, adding depth to traces that are already deep with transition chains and subflows.BackgroundJob.Executeis deliberately left visible — it is a real service boundary and the ambient parent of the job's work.2. Deferred arm handle
A caller that must persist a job inside a critical section had no way to keep the scheduler round-trip out of it:
EnqueueAsynceither armed inline or on the ambient UoW's completion, and both land inside the caller's lock.Measured on a vNext workflow accept path under load, that external call was the lock hold time — arming p50 214 ms of a 198 ms median hold — serialising every other request on the same instance behind an external round-trip.
EnqueueWithDeferredArmAsyncpersists with the same semantics asdirectly: true(row landsScheduled; an arm failure rolls it back toPendingfor the arming poller) and returns a handle instead of calling the scheduler. The handle closes over the arguments captured at enqueue time, so arming later costs one scheduler call — no job-row read, no extra status write.EnqueueAsync's behaviour is unchanged; its body moved to a shared private core.3. Poll pacing: jitter, and errors no longer stall the fleet
Both problems only bite with several replicas — which is how these workers run (10 each in nonprod and prod).
No jitter. The backoff was a deterministic doubling, so pods started together — what a rolling deployment produces — polled in lockstep. That costs twice: the fleet loses the staggering that makes N replicas pick work up ~N times sooner than one, and every tick becomes a burst of simultaneous claim queries over the same rows. Every delay is now jittered ±25%, plus a random startup offset within the idle interval.
Errors jumped straight to
MaxPollingInterval. One transient fault — a brief database hiccup, a single poison message — stalled every replica for a full maximum interval, exactly when it was least affordable. An error now backs off one step like an empty round, floored at the idle interval so a hard failure is not retried at the busy cadence; repeated errors still escalate to the cap.The rules moved into
PollingDelay, shared by the outbox, inbox and background-job arming loops. The arming loop has no adaptive backoff by design — an unarmed job must be claimed within a bounded time — which makes jitter the only thing keeping its replicas apart.Testing
BBT.Aether.Infrastructure.Tests: 170 passed / 0 failed (was 161; +9).AdaptivePollingTestsexercised a private copy of the delay arithmetic, so a change to the real rules could not fail it. They now drivePollingDelayitself, with coverage for jitter, the error floor and the startup offset.background service errorentries in either worker.Completed, none stranded inPending.Reviewer notes
Enqueue/Update/Deletespan gating rides in commit 2 rather than commit 1 because it lives in the same file as the arm handle; splitting it would not have been worth the churn.IBackgroundJobServicegains one method. Existing implementors outside this repo would need it — worth confirming there are none before merge.BackgroundJob.Scheduleas diagnostic removes the span that was used to verify "is arming inside the lock" from Business-profile traces. That check now needs Verbose, or the job-row status as a proxy. Worth deciding whether that span should stay visible.🤖 Generated with Claude Code
Summary by Sourcery
Decouple background-job persistence from scheduler arming and make worker polling more resilient and evenly distributed.
New Features:
Bug Fixes:
Enhancements:
Tests: