UPDATE 4: Request-#2 EOF — status after CUB/JIT fix (correlation, not yet proven root cause)
The previously observed 9 ms EOF is reproducible only under the earlier configuration and has not reproduced under the current constrained configuration. Historical evidence rules out TCP connection reuse as the cause (the failing run used fresh connections; 3/3 requests on one keep-alive connection succeed under the current config).
Historical runs for reference:
- 22:48 run (SmolLM2, --num-tokens 4096 only, cuda-graph-max-bs default, attention auto→fi): req 1 = 200, req 2 = empty/9 ms on a fresh TCP connection; server process stayed alive (no worker exit visible).
- 23:08 run (same model/flags): req 1 = 200, req 2 = 200, backend worker exited ~9 s later.
- 00:17 run (35B, --attn triton): first request triggered the flashinfer JIT compile of sampling.cu → 100 CUB errors → worker died mid-generation.
Current run (35B, --num-tokens 8192, --cuda-graph-max-bs 1, --attn triton, after CUB fix): 15/15 requests 200 (2 fresh connections, 3 on one keep-alive connection, 10 sequential), full dispatch trace for every request, zero errors, no worker exit.
The remaining suspect is CUDA-graph / request-state behavior under the default batch-size (cuda-graph-max-bs=auto) configuration — but this is NOT yet proven. A controlled A/B test (only cuda-graph-max-bs varied, everything else identical) is planned; results will be posted here.
UPDATE 5: A/B test result — cuda-graph-max-bs is NOT the trigger
Controlled A/B experiment on the Qwen3.6-35B-A3B (everything identical except one flag):
- A: --cuda-graph-max-bs 1 -> 5/5 requests 200 (fresh TCP connection per request)
- B: cuda-graph-max-bs default/auto -> 5/5 requests 200 (fresh TCP connection per request)
Both runs: full dispatch trace for every request, zero errors, no worker exit.
The default batch-size (CUDA-graph) configuration alone does not reproduce the 9 ms EOF.
The earlier EOF remains tied to the pre-fix first-request JIT/rebuild instability
(flashinfer JIT compile of sampling.cu failing with the bundled CUB headers).
If it reappears under any configuration, a trace-based reproducer will be posted here.
UPDATE 6: historical EOF configuration reproduced exactly — 5/5 OK
Run C: exact historical failing configuration (SmolLM2-1.7B, --num-tokens 4096,
attention backend auto -> fi, cuda-graph-max-bs default, port 11501, fresh TCP
connection per request):
- Req 1: 200 (15.9 s — first-request phase incl. CUDA-graph capture)
- Req 2-5: 200 (0.17 s, KV-cache hits)
- 5/5 OK, full dispatch trace, zero errors, worker alive
Systematic PRE-FIX vs. CURRENT diff (only code changes between the 22:48 EOF run
and now):
- flashinfer/jit/env.py: CCCL_INCLUDE_DIRS switched from bundled CCCL 3.0.3 to
system CUB/Thrust 2.0.1 + vendor libcudacxx (the only behavioral change)
- freetoken api_server.py/generation.py: trace prints only, no logic changes
- no toolchain/version changes in between
Summary across all configurations after the env.py fix: 30/30 requests 200
(15 current-config + 5 cuda-graph-max-bs=1 + 5 default + 5 historical-config).
The 9 ms EOF is not reproducible with the fixed JIT/toolchain state.
UPDATE 7: Status — fixed / no longer reproducible (watch item, open)
The 9 ms EOF issue is no longer reproducible after correcting the FlashInfer JIT
include environment. The exact causal mechanism of the historical EOF remains
unproven.
Evidence: 30/30 requests 200 after the env.py fix (current config 15, cuda-graph-max-bs=1
5, default 5, historical EOF config 5); the only behavioral change is
flashinfer/jit/env.py (CCCL_INCLUDE_DIRS -> system CUB/Thrust 2.0.1 + vendor
libcudacxx); connection reuse and cuda-graph-max-bs ruled out.
Keeping this issue open as a watch item: if the 9 ms EOF ever reappears with a
live READY worker, a trace-based reproducer will be posted here. A PR with the
minimal env.py fix is being prepared against the flashinfer repository.
UPDATE 4: Request-#2 EOF — status after CUB/JIT fix (correlation, not yet proven root cause)
The previously observed 9 ms EOF is reproducible only under the earlier configuration and has not reproduced under the current constrained configuration. Historical evidence rules out TCP connection reuse as the cause (the failing run used fresh connections; 3/3 requests on one keep-alive connection succeed under the current config).
Historical runs for reference:
Current run (35B, --num-tokens 8192, --cuda-graph-max-bs 1, --attn triton, after CUB fix): 15/15 requests 200 (2 fresh connections, 3 on one keep-alive connection, 10 sequential), full dispatch trace for every request, zero errors, no worker exit.
The remaining suspect is CUDA-graph / request-state behavior under the default batch-size (cuda-graph-max-bs=auto) configuration — but this is NOT yet proven. A controlled A/B test (only cuda-graph-max-bs varied, everything else identical) is planned; results will be posted here.
UPDATE 5: A/B test result — cuda-graph-max-bs is NOT the trigger
Controlled A/B experiment on the Qwen3.6-35B-A3B (everything identical except one flag):
Both runs: full dispatch trace for every request, zero errors, no worker exit.
The default batch-size (CUDA-graph) configuration alone does not reproduce the 9 ms EOF.
The earlier EOF remains tied to the pre-fix first-request JIT/rebuild instability
(flashinfer JIT compile of sampling.cu failing with the bundled CUB headers).
If it reappears under any configuration, a trace-based reproducer will be posted here.
UPDATE 6: historical EOF configuration reproduced exactly — 5/5 OK
Run C: exact historical failing configuration (SmolLM2-1.7B, --num-tokens 4096,
attention backend auto -> fi, cuda-graph-max-bs default, port 11501, fresh TCP
connection per request):
Systematic PRE-FIX vs. CURRENT diff (only code changes between the 22:48 EOF run
and now):
system CUB/Thrust 2.0.1 + vendor libcudacxx (the only behavioral change)
Summary across all configurations after the env.py fix: 30/30 requests 200
(15 current-config + 5 cuda-graph-max-bs=1 + 5 default + 5 historical-config).
The 9 ms EOF is not reproducible with the fixed JIT/toolchain state.
UPDATE 7: Status — fixed / no longer reproducible (watch item, open)
The 9 ms EOF issue is no longer reproducible after correcting the FlashInfer JIT
include environment. The exact causal mechanism of the historical EOF remains
unproven.
Evidence: 30/30 requests 200 after the env.py fix (current config 15, cuda-graph-max-bs=1
5, default 5, historical EOF config 5); the only behavioral change is
flashinfer/jit/env.py (CCCL_INCLUDE_DIRS -> system CUB/Thrust 2.0.1 + vendor
libcudacxx); connection reuse and cuda-graph-max-bs ruled out.
Keeping this issue open as a watch item: if the 9 ms EOF ever reappears with a
live READY worker, a trace-based reproducer will be posted here. A PR with the
minimal env.py fix is being prepared against the flashinfer repository.