Tail Latency Lab

A duplicate request can rescue your p99, or become the load that breaks it.

Tail Latency Lab is a deterministic Rust simulator for hedged requests. It races duplicate attempts against long-tail responses, places every attempt behind a finite worker pool, and shows the point where latency insurance turns into queue collapse.

The interesting part

A hedge sent after 100 ms looks cheap when only 3% of requests are slow. It is not cheap when the worker pool is nearly saturated: duplicates wait behind originals, uncancelled losers keep consuming service time, and more requests live long enough to launch another attempt.

On the default workload, hedging has a clear win:

hedge             p99  timeout   traffic      queue
disabled         727ms    0.00%     1.00x      0.0ms
50ms              89ms    0.00%     1.03x      0.0ms
100ms            139ms    0.00%     1.03x      0.0ms
200ms            239ms    0.00%     1.03x      0.0ms

Move the same policy closer to the capacity boundary and the feedback loop reverses. The included Near saturation preset demonstrates that transition.

Run the interactive lab

Rust 1.85 or newer is required.

cargo run --release

Open http://127.0.0.1:3000. If that port is occupied:

PORT=3001 cargo run --release

The browser interface is served by Axum and calls the same simulation engine as the CLI. It has no frontend build step or CDN dependency.

Use the CLI

cargo run --release -- compare --preset "Quiet heavy tail"
cargo run --release -- compare --preset "Near saturation" --json
cargo run --release -- compare --preset "Pathological tail" --hedge-ms 150

Available presets:

• Quiet heavy tail
• Near saturation
• Pathological tail

Use the engine

use tail_latency_lab::{simulate, SimulationConfig};

let result = simulate(SimulationConfig {
    arrival_rps: 1_000,
    workers: 64,
    hedge_delay_ms: 75,
    ..SimulationConfig::default()
})?;

println!("p99={}ms traffic={:.2}x", result.p99_latency_ms, result.amplification_factor);
# Ok::<(), String>(())

Model boundaries

The engine is a seeded discrete-event simulation, not a production capacity planner.

• Logical requests arrive at a uniform rate.
• Service time comes from a jittered fast/slow mixture.
• Attempts queue against a fixed-size worker pool.
• A hedge is sent only if the logical request is still pending.
• The first successful attempt wins, but losing work is not cancelled.
• Queued attempts that cannot start before their deadline are discarded.
• Autoscaling, connection pools, network loss, retries beyond one hedge, and correlated failures are intentionally omitted.

Stable per-request samples make policy comparisons fair: changing the hedge delay does not silently redraw the original workload.

Optional aggregate telemetry

The lab runs offline by default. To record aggregate usage bands with the Telemetry Rust SDK:

export TELEMETRY_API_KEY=your_project_api_key
cargo run --release --features telemetry

Only the hedge-delay band, timeout band, and whether traffic exceeded 1.1x are sent. Request values, simulation timelines, and IP-derived data are not logged by the integration.

Development

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features

License

MIT