Follow-up to #1399, which added the run ceiling. The ceiling is a cooperative stop: the run checks it between months and at batch boundaries. A run blocked inside a call reaches no checkpoint, so the ceiling cannot end it.
What that costs
The range recovers — the claim stops being renewed at the ceiling, the lease lapses, and another process claims it. What does not recover is this process. worker_main runs a single poller loop, so a wedged run means this container indexes nothing more until it is restarted. Recovering rows while losing a worker slot per wedge is not fleet recovery.
The Java worker solves this by interrupting the run thread (RunHandle(Thread.currentThread()) / runHandles in IndexWorker), which is also why StatementTimeouts deliberately exempts the worker's writes: it has cancellation instead.
What #1399 does about it
Bounds the two ways a call can hang rather than adding cancellation:
- chess.com was already bounded —
request_timeout_ms = 60'000 per attempt, three attempts.
- the database was not.
one_d4_worker/db_options.h now applies statement_timeout=120s and tcp_user_timeout=150s in the conninfo, keeping DataSourceFactory's ordering (the transport bound must outlast the statement bound, or a healthy slow statement is severed before the server can cancel it cleanly).
So a run blocked on a lock wait or a black-holed connection now unwedges itself within a couple of minutes and hits the ceiling check normally. What remains uncovered is anything that blocks outside SQL and HTTP — nothing in the run does today, which is why bounding was judged proportionate.
What would close it
A cancellation path: the poller runs the job on its own thread and stops waiting on it once the ceiling passes, reclaiming the slot. The complication is that an abandoned thread keeps its pg::Client and may still attempt a write — safe, since every write is fenced on the request row and would be refused, but it needs to be deliberate rather than incidental, and the connection has to be accounted for.
Worth doing when a wedge is observed in practice, or when the worker grows past one poller loop per process.
Follow-up to #1399, which added the run ceiling. The ceiling is a cooperative stop: the run checks it between months and at batch boundaries. A run blocked inside a call reaches no checkpoint, so the ceiling cannot end it.
What that costs
The range recovers — the claim stops being renewed at the ceiling, the lease lapses, and another process claims it. What does not recover is this process.
worker_mainruns a single poller loop, so a wedged run means this container indexes nothing more until it is restarted. Recovering rows while losing a worker slot per wedge is not fleet recovery.The Java worker solves this by interrupting the run thread (
RunHandle(Thread.currentThread())/runHandlesinIndexWorker), which is also whyStatementTimeoutsdeliberately exempts the worker's writes: it has cancellation instead.What #1399 does about it
Bounds the two ways a call can hang rather than adding cancellation:
request_timeout_ms = 60'000per attempt, three attempts.one_d4_worker/db_options.hnow appliesstatement_timeout=120sandtcp_user_timeout=150sin the conninfo, keepingDataSourceFactory's ordering (the transport bound must outlast the statement bound, or a healthy slow statement is severed before the server can cancel it cleanly).So a run blocked on a lock wait or a black-holed connection now unwedges itself within a couple of minutes and hits the ceiling check normally. What remains uncovered is anything that blocks outside SQL and HTTP — nothing in the run does today, which is why bounding was judged proportionate.
What would close it
A cancellation path: the poller runs the job on its own thread and stops waiting on it once the ceiling passes, reclaiming the slot. The complication is that an abandoned thread keeps its
pg::Clientand may still attempt a write — safe, since every write is fenced on the request row and would be refused, but it needs to be deliberate rather than incidental, and the connection has to be accounted for.Worth doing when a wedge is observed in practice, or when the worker grows past one poller loop per process.