barrage v1: rate-controlled load generator over the shared core - #18
Merged
Conversation
Barrage is a load generator that fires provider-shaped payloads at an endpoint the operator controls, at a high but controlled rate, and reports behaviour under sustained load. It is a load tester against your own infrastructure. It is NOT an email sender, NOT a flooding tool, and NOT for endpoints you do not own. runner.py: open-loop (fixed arrival rate) and closed-loop (fixed concurrency) modes, a warmup ramp plus steady-state hold, and a hard rate-and-duration ceiling (50 req/s, 300s) that requires an explicit allow_high_rate to raise. The ceiling is a safety control: a typo in --rate must not become a self-inflicted denial of service. Pacing reuses core.ratelimit.TokenBucket with injected clock and sleep. payloads.py: reuses blast.generate.generate_corpus for realistic-but-valid seeded bodies. Clean load only; blast/corrupt.py is deliberately untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ramp path could block forever under an injected clock. Root cause is not the ramp loop's exit condition and not a zero rate at ramp start: TokenBucket.acquire() itself cannot be driven to completion by a purely additive injected clock when the rate's reciprocal is not exactly representable in binary. It computes wait_for = deficit / rate, sleeps exactly that, then refills by wait_for * rate, which rounds to just under deficit. The residual deficit is ~1e-16, so the next wait_for is ~1e-17, and adding 1e-17 to a clock reading ~0.67 is a no-op at float precision: elapsed becomes 0, no refill happens, and the loop spins forever. Rates 2 and 4 are exactly representable and never trip it, which is why the steady-state tests passed. A ramp to 10 in 5 steps yields rates 2, 4, 6, 8, 10, and rate 6 hung the run. A real wall clock ticks forward on its own and masks this, so this is latent in core/ratelimit.py rather than Barrage-specific. core is not this lane's to edit and its own tests only exercise rates 1 and 2, both exactly representable. Reported to the director rather than worked around silently. Barrage now paces on an absolute per-stage schedule and uses the bucket as the rate-authority gate, arriving only once a token is already earned, so the bucket never enters its internal wait loop. Stage dispatch counts are bounded integers, so a stage always terminates. Regression tests pin it: parametrized over rates whose reciprocal is not binary exact, and a BoundedSleeper that fails loudly on a degenerate sub-nanosecond sleep or an implausible sleep count rather than spinning. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
tests/test_lane_hygiene.py documents that adding __init__.py under tests/ was tried and reverted because it breaks the integration lane's sibling import of fake_sink. This was the only one in the repo. Barrage's test basenames are unique, so it was never needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Where Blast's core/report.py asks whether the parser correctly rejected garbage, Barrage asks whether the pipeline held. Mirrors core/report.py's build_artifact/format_summary shape without importing its expectation rules, which are about parser correctness and do not apply to load. Reports throughput achieved versus targeted, latency p50/p90/p99, error rate over time, and the knee where the endpoint starts shedding (errors climb) or slowing (median latency exceeds 2x the healthy baseline). JSON artifact plus a human summary that reads visibly differently for a run that held and a run that fell over. Reporting math is pure: no I/O, no network, no clock reads. Latency and wall-clock enter only as caller-supplied data, so output is a pure function of input and is tested entirely on synthetic samples. Judgement calls worth review: - Percentiles use nearest-rank, so a reported p99 is always a real request's latency rather than an interpolated number no request saw. - Timeouts are excluded from the latency distribution and reported via the error rate instead: a timeout's latency reflects the timeout setting, not the endpoint's speed, and mixing it in skews every percentile. - 4xx is not an error: that is the endpoint correctly rejecting something under load, not the pipeline failing to hold. 5xx and no-response are. - Empty middle buckets are emitted, not skipped: a stretch where the endpoint accepted nothing is the shedding this report exists to show. - Empty inputs report None, never 0.0, which would read as "extremely fast" when it means "no data". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
testinghq barrage fire --target ... --rate ... --duration ... --concurrency ... --send, plus replay. Dry-run is the DEFAULT; --send is required to put anything on the wire. Orchestration lives in barrage/fire.py; cli.py only gains the barrage subcommand and its parser, so nothing safety-relevant is decided in the argparse layer. The blast subcommands are untouched and a test pins that. Guardrails, imported and never reimplemented, called positionally: - evaluate_send decides dry-run vs send. - require_configured_target gates BOTH the target name (allow-list) and the resolved URL (public-host check). Checking only the name is the trap: a bare single-label name like "local" has no dot, so the public-host hardening classifies it internal and passes it unconditionally, leaving that hardening inert while looking correct. Follows testinghq/cli.py's existing blast fire path. - require_synthetic_content runs over every address in the whole pool before any network call, so a bad payload aborts the run before the first request rather than after some prefix already fired. The ceiling is checked in the CLI as well as inside run(), so a dry run reports an over-limit plan as refused rather than previewing a run that would never be allowed. Dry-run's zero-network property is PROVEN, not asserted: the socket module is patched to raise. Verified by deliberately sabotaging the dry-run path to make a real call; the three socket-patch tests failed as they should, and the sabotage was reverted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…example The freshness guard from #17 fired on its first live run: landing testinghq/barrage/runner.py turned tests/test_backlog_freshness.py red because the backlog still claimed Barrage was in progress. That is the guard working as designed, and this commit is what "keep docs current in the same PR as the code" looks like when it is enforced rather than requested. The build would not go green without it. Two fixes. 1. Barrage v1 marked done, item by item, against what is actually in this branch. 2. The documentation example in the staleness-markers section named testinghq/barrage/runner.py as its sample path. It was a live marker, not an illustration, so it armed itself the moment Barrage landed and the guard began reporting its own instructions as a stale claim. Rewritten to name a path that does not exist and to spell the marker with the word split so the example cannot arm itself again. An example of a tripwire must not be a tripwire. Also records the core/ratelimit.py TokenBucket precision bug that this lane found and correctly worked around rather than reaching into another lane to fix. The root cause is still on main and the item is filed for Lane A, including the part that matters most: tests/security/test_rate_limit_gate_contract.py passes today because it chose a binary-representable rate, not because the code is right. 448 passed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Caught by actually running barrage against a local sink rather than by a hermetic test: the first second of a 10 req/s run reported 15 req/s achieved, and an 8 second run finished in 6.2 seconds. Two defects, both in stage sequencing: 1. A stage ended the instant its last request was dispatched, up to one full interval early. With a 3s warmup over 10 steps the stages are 0.3s each, so round(rate * 0.3) is often 1: the stage fired its single request at stage_start and returned immediately, taking no time at all. The ramp collapsed into a burst. 2. A stage whose budget rounded to 0 (an early low-rate step too short to earn a whole request) was skipped outright via `continue`, so its window never elapsed and the ramp was silently shortened further. Both are safety-relevant, not cosmetic. The warmup ramp exists to ease into load rather than slam a cold endpoint, and it was doing the exact opposite: overshooting the configured rate by 50% in the first second. A ramp that fires everything at once is as much a defect as one that blocks forever. Stages now hold to their nominal end, and a zero-budget stage still occupies its window. Verified against the live sink: the ramp reads 4/s of 5/s, 7/s of 8/s, 9/s of 10/s, then 10/s steady, no bucket overshoots its target, and an 8s run now takes 8.19s. Tests pin it under an injected clock: the ramp occupies its full warmup, a single-request stage still occupies its stage, steady-state dispatches are never spaced tighter than the target interval, and the ramp carries less load early than late. README documents barrage now that it ships a usable command, including the three controls that keep it a load tester rather than a weapon. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Barrage v1 per assignments 05 and 06. 448 passed under bare
pytest -q. Built in four committed steps, plus a bug fix and the backlog reconciliation the freshness guard forced.Where Blast proves the parser is correct under messy input, Barrage proves the pipeline holds under load. Same suite, same installable, same core. Clean load only, never
blast/corrupt.py.barrage/runner.py— closed-loop fixed-concurrency and open-loop fixed-arrival-rate, warmup ramp, steady-state hold, and a hard rate-and-duration ceiling requiring an explicit flag.barrage/payloads.py— reusesblast.generate.generate_corpus, seeded.barrage/report.py— throughput achieved versus target, latency p50/p90/p99, error rate over time, the knee. JSON artifact plus human summary.barrage/fire.py—testinghq barrage fire, dry-run default, plus replay.core/untouched, verified by diff.Safety controls, verified end to end in a container, not asserted
That last one is the important one. The target is in the config and is still refused because the resolved URL points at a real public host. It proves the CLI passes both the name (allow-list) and the resolved URL (public-host check) through
require_configured_target. Passing only the name leaves the hardening inert, because a single-label name has no dot and classifies as internal. That bug was found in my own instructions earlier today; it went into this brief and landed right first time.The rate ceiling, the configured-target rule, and the dry-run default are what keep this a load tester against infrastructure you own rather than a flooding tool.
A real bug in
core/ratelimit.py, found here, deliberately not fixed hereTokenBucket.acquire()spins forever under a purely additive injected clock when the rate's reciprocal is not exactly representable in binary. It computeswait_for = deficit / rate, advances by exactly that, refills bywait_for * rate, which rounds to just underdeficit. Residual ~1e-16, nextwait_for~1e-17, and adding 1e-17 to a clock reading ~0.67 is a no-op at float precision: elapsed becomes 0, no refill, infinite loop.Rates 2 and 4 are binary-exact and never trip it, which is why every existing test passes. Rate 10 hangs. Under a real monotonic clock it self-heals, so it is invisible in production and fatal under the injected clocks this repo mandates for hermetic tests.
The implication worth reading twice:
tests/security/test_rate_limit_gate_contract.pypasses today because it happened to pick a representable rate, not because the code is correct. A guardrail test passing by coincidence of a parameter choice.This lane worked around it in
runner.pyand correctly did not reach intocore/. Root cause is still on main and is filed for Lane A in the backlog, including parameterising both regressions over non-representable rates so neither can pass by luck again.The freshness guard fired on its first live run
Landing
testinghq/barrage/runner.pyturnedtests/test_backlog_freshness.pyred, because the backlog still claimed Barrage was in progress. 447 passed, 1 failed — exactly as designed. This PR could not go green until the docs matched the code. That is #17 doing its job on contact and "keep docs current in the same PR" being enforced rather than requested.It also exposed a flaw in the guard: the documentation example used
testinghq/barrage/runner.pyas its sample path, so it was a live marker rather than an illustration and armed itself the moment Barrage landed. Rewritten to name a non-existent path with the marker word split. An example of a tripwire must not be a tripwire.Also fixed by the agent unprompted: it had added a
tests/unit/barrage/__init__.pythat no other test directory carries, and removed it to match convention.