std.resilience: retry, backoff, rate limiting and circuit breaking, shared - #635
Merged
Conversation
a result field on a value whose payload type is a generic parameter — the is_ok/ok/err of a T! inside fn f[T] — fell through the struct lookup to the bare no-offset field form, which the ir contract rejects. so a generic function could not look inside a result it held, which is exactly the shape a generic retry helper is: fn attempt[T](policy, op: fn() -> T!). a result's layout does not depend on what it carries: the flag is at 0, ok at 8, err at 16. the fallback now emits those fixed offsets, and is_err — which is not stored, only derived — reads the flag and negates it rather than loading offset 0 under the wrong name.
retry and backoff logic existed as private implementations inside otlp and tcp, and nothing anywhere offered a rate limiter or a circuit breaker. this is the shared layer: a Retry policy whose deterministic doubling curve is pure math the protocols can consume without adopting the runner, a generic attempt/attempt_if for string-error operations, a token-bucket Limiter, and a Breaker with the closed/open/half-open cycle and a single-probe half-open. the limiter and breaker are shared across tasks on purpose — one bucket for however many tasks are serving — so their state lives in handle-keyed registries behind one module mutex, the same shape the tls config registry settled on. the breaker's open state is an explicit flag rather than a timestamp sentinel, because the mono clock starts at zero with the process and 'opened at 0' is a perfectly real moment.
otlp's export retry and tcp's accept backoff each carried a private copy of the same doubling curve; both now configure a resilience.Retry and sleep policy.delay_ms. their domain logic stays where it was — otlp still classifies http statuses and honors Retry-After, tcp still gives up after a run of consecutive failures — only the arithmetic moved. grpc gains what it never had: Conn.unary_retry, retrying a unary call under a policy when the status is one that describes the service's moment rather than the request's content — UNAVAILABLE, RESOURCE_EXHAUSTED, ABORTED. everything else propagates on the first attempt, deadlines included.
the middleware story was App plus access_log. these are the first two parameterized middleware: rate_limit spends a token from a shared bucket per request and answers 429 dry, and circuit runs requests through a breaker — a 5xx from the handler counts as a failure, an open circuit answers 503 without reaching the handler, and the half-open probe is a real request whose outcome decides. both compose with group(), so a limit or a breaker can guard one prefix instead of the whole app.
docs/resilience.md covers the retry curve, the limiter, the breaker, and what is deliberately absent — no jitter, no waiting allow, no timeout combinator, each with the reasoning. the web docs point at the new middleware from the middleware section, and examples/resilience.pith runs the whole surface deterministically: recovery under retry, a classifier refusing a permanent error, a burst draining the bucket, and a breaker tripping, probing and closing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
retry and backoff logic lived as private implementations inside the protocol modules, and nothing anywhere offered a rate limiter or a circuit breaker — the middleware story was App plus access_log. this factors the resilience layer out as
std.resilienceand composes both web and grpc against it.the module. a
Retrypolicy —resilience.retries(5).base(100).cap(2_000)— whose deterministic doubling curve is pure math, a genericattempt/attempt_ifrunner for string-error operations, a token-bucketLimiter, and aBreakerwith the full closed/open/half-open cycle and a single-probe half-open. the limiter and breaker are shared across tasks on purpose (one bucket for however many tasks are serving), so their state lives in handle-keyed registries behind one mutex — the shape the tls config registry settled on after its race.the adopters. otlp's export retry and tcp's accept backoff now configure a policy and sleep its curve; their domain logic — status classification, Retry-After, give-up-after-N — stays where it was, only the arithmetic moved. grpc gains
Conn.unary_retry, retrying on the codes that describe the service's moment rather than the request's content (UNAVAILABLE, RESOURCE_EXHAUSTED, ABORTED) and propagating everything else — deadlines included — on the first attempt. std.web gains its first parameterized middleware:web.rate_limit(limiter)answers 429 dry,web.circuit(breaker)answers 503 open and counts handler 5xx as failures; both compose withgroup()so a guard can cover one prefix.the compiler fix that unblocked it. a generic function could not look inside a result it held —
result.is_okon aT!insidefn f[T]emitted the no-offset field form the ir contract rejects, which is exactly the shape a generic retry helper is. a result's layout does not depend on its payload, so the fallback now emits the fixed offsets, withis_errderived by negating the stored flag rather than misreading offset 0.honest inventory against the feedback: the five private implementations turned out to be two real ones (otlp, tcp) — sse's retry_ms is a protocol field and tls's hello retry is handshake state, neither retry logic. the missing half of the feedback (limiter, breaker, parameterized middleware) is the larger share of this change.
what was tested
make test,make bootstrap-verify(seed regenerated for the emitter fix),make run-examples,make fuzz-checkandmake leak-check, all clean. the module's own tests pin the delay schedule exactly, retry counts through success/exhaustion/permanent-error paths, bucket drain and refill, and the breaker's trip → refuse → single-probe → recover cycle including a failed probe re-opening. web tests drive both middleware through fake handlers; grpc tests pin the retryable-code classification; otlp and tcp keep their curve tests against the shared math.notes
deliberately absent, with reasons in docs/resilience.md: jitter (determinism is worth more at this scale; the limiter is the spreading tool), a waiting
allow(backpressure decisions belong to the caller), and a timeout combinator (deadlines stay per-protocol — a timeout that cannot cancel the underlying work is a lie). the breaker's open state is an explicit flag rather than a timestamp sentinel: the mono clock starts at zero with the process, and 'opened at 0' is a real moment — a bug the module's own tests caught.