fix(renderer): give chrome_proxy recovery arm its own budget and load-shed it#321
Merged
Conversation
…-shed it
The residential-egress recovery arm never fired on real /v1/scrape
hard-blocks: the HTTP + LightPanda + Chrome ladder burns a ~15s scrape
deadline down to ~2s before the arm, far below the 8s floor, so the gate
`deadline.remaining() >= floor` was never satisfied (prod-measured ~2s).
- dispatch the arm with a fresh `Deadline::now_plus(CHROME_PROXY_ARM_BUDGET_MS
= 12s)` instead of the exhausted shared deadline; effective render budget is
`min(this, chrome_nav_budget_ms)`. Only hard-blocked scrapes (which would
otherwise fail) pay it, and the SaaS->engine fetch tolerates 120s.
- drop the `deadline.remaining()` floor (a no-op against a fresh deadline) and
the CHROME_PROXY_ARM_FLOOR_MS const.
- load-shed: a non-blocking `chrome_proxy_arm_sem` (sized to the chrome_proxy
pool) gates firing so a burst of datacenter-blocked URLs can never queue on
the residential pool for up to the SaaS timeout and collapse co-tenant
throughput; no permit -> return the block (best-effort recovery).
- count sheds as render_route_decision_total{armShed} so an undersized
residential pool is observable under sustained hard-block load.
Also removes the temporary arm budget diagnostic.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The residential-egress recovery arm never fired on real /v1/scrape hard-blocks. Prod-measured with a temporary diagnostic: the HTTP + LightPanda + Chrome ladder burns a ~15s scrape deadline down to ~2s before the arm (
ARM_DIAG gate ... remaining_ms=1984 arm_floor_ms=8000), so thedeadline.remaining() >= 8sfloor was never met and chrome_proxy never fetched. Wikipedia's ~12/13 recovery was the HTTP-tier residential proxy retry, not the arm.Fix
Deadline::now_plus(CHROME_PROXY_ARM_BUDGET_MS = 12s)instead of the exhausted shared deadline (effective render budgetmin(this, chrome_nav_budget_ms), both 12s). Only hard-blocked scrapes pay it; the SaaS->engine fetch tolerates 120s (crw-client.ts TIMEOUT_MS), and the fetch overall-timeout still clamps to the fresh 12s so a hung page can't hang it.chrome_proxy_arm_semsized to the chrome_proxy pool (config.pool_size, == itsconn_semaphore). No permit -> return the block. Prevents a datacenter-blocked batch/crawl from queueing every page on the ~pool residential slots for up to the SaaS timeout and collapsing co-tenant throughput (the 500-1000-concurrent invariant). On the managed prod path chrome_proxy is arm-exclusive (!proxy_active), so a permit implies a free pool slot; a fired arm never queues.render_route_decision_total{armShed}.Removes the temporary arm diagnostic.
Validation
Two review rounds (correctness + p90/concurrency plan review; correctness + adversarial code review) reached consensus. Tests:
chrome_proxy_arm_fires_below_old_floor(2s deadline -> fires),chrome_proxy_arm_load_shed_when_pool_saturated(drained permits -> shed), full workspace green (1292), clippy clean lean +--features cloak.Post-deploy: a real datacenter-blocked hard scrape the HTTP-tier retry didn't clear now fires chrome_proxy (chain includes it) and recovers; watch
armShedand the hard-block p90.