Decide and implement retry and backoff

[iderex]

A retry policy invented per call site is eleven policies, and the aggressive one
turns a server having a bad minute into a server being hammered by every client
in the house.

What this produces

One policy, written down with its numbers and the reason for each. Which error
kinds are retried, which are never retried, how many attempts, and the interval
between them.

Randomised spacing, so that a wall of tiles failing together does not retry
together.

An overall bound on how long a caller can be kept waiting by retries, connected
to the thresholds in the slow-server record in #7, so that retrying never quietly
exceeds the point at which the core should have reported that an answer is not
coming.

Done when

The policy is recorded with its numbers and reasons, a test proves a
never-retried kind is attempted exactly once, a test proves the total time
stays within the bound from #7, and a test proves two callers failing at the
same moment do not retry at the same moment. Blocked on #4, #7, and #102 for
the clock a backoff interval is measured against, since an interval taken
across a clock correction is either instant or a month.

[iderex]

The clock this issue was waiting on is decided in docs/decisions/0102-the-clocks-every-deadline-is-measured-against.md.

A backoff interval is on the steady clock: an interval between two events inside one run, measured on a clock that only moves forward and that nothing corrects. That is what removes the failure named in this issue's own condition, since an interval taken across a wall-clock correction is either instant or a month.

The steady clock rather than the elapsed one, for the same reason a request timeout is on it. A device that suspended between two attempts has no attempt in flight, and counting the sleep against the backoff would turn every wake into an immediate retry storm.

The jitter that separates two callers failing at the same moment is a duration and is on the same clock.

[iderex]

The policy landed as docs/decisions/0038-retry-and-backoff.md, on 090a604, with its numbers and the reason for each.

Three kinds are retried inside a call, timed-out, server-busy and server-failed, and no other kind is. A call gets at most three attempts, and every attempt and every wait sits inside the same five seconds 0007 already abandons a request after, which is the overall bound this issue asks for. The wait before each retry is drawn uniformly at random from zero to a doubling value starting at 250 ms, which is the randomised spacing, and the draw is per attempt per caller so that two callers failing together are separated rather than moved together.

Two things that a retry loop written later would have settled by accident are settled there. The deadline and the 400 ms late report belong to the call and are not restarted by a retry, so three attempts cannot turn five seconds into fifteen while each number still looks right on its own. And a call that spends every attempt is one abandonment rather than three, which matters because 0007 declares a server unreachable after two consecutive ones.

The record also resolves a disagreement between two landed records rather than leaving it to be discovered. 0004 marks server-unreachable retryable after a delay; 0007 reports the server absent at once and attempts nothing further until the bounded recovery in #45 says otherwise. The delay is that schedule, and no retry of the request happens inside the call.

Every number is chosen and none is measured, in the same terms 0007 uses for the thresholds these are fitted around, and #65 is the harness that would replace a choice with a number.

Three of the four conditions here are untouched. A never-retried kind attempted exactly once, the total time staying inside 0007's bound, and two callers not retrying at the same moment are all statements about running code, and there is no language chosen, no build and no test command in this tree to assert them against. This stays open on those three.

[iderex]

Clearing the label: nothing this issue names as a blocker is still open

The label says this waits on a decision nobody has taken. Its body's Blocked on
sentence names the issues below, and every one of them is closed as completed:

for n in 4 7 102; do gh api repos/Flowfin/core/issues/$n --jq '"\(.number) \(.state|ascii_upcase) \(.state_reason|ascii_upcase)"'; done
4 CLOSED COMPLETED
7 CLOSED COMPLETED
102 CLOSED COMPLETED

The decision this issue owed is on the default branch and names this issue:

git show origin/main:docs/decisions/0038-retry-and-backoff.md | sed -n '5,7p'
Status: accepted. Supersedes nothing. Superseded by nothing.

Issue: #38

So what is left here is the implementing half rather than a decision, and the
label sends every session that meets it past an issue that can be worked.

Removing blocked-on-decision. No condition and no scope on this issue changes, and nothing
here says the work is small or that anything else is unblocked. It says this one is
waiting on nothing that is still open.