Skip to content

feat(bench): pass ft serve options through '--' in bench_decode_moe - #254

Open
yuyi2439 wants to merge 1 commit into
FlashML-org:mainfrom
yuyi2439:feat/bench/passthrough
Open

feat(bench): pass ft serve options through '--' in bench_decode_moe#254
yuyi2439 wants to merge 1 commit into
FlashML-org:mainfrom
yuyi2439:feat/bench/passthrough

Conversation

@yuyi2439

@yuyi2439 yuyi2439 commented Aug 28, 2026

Copy link
Copy Markdown

Replace the script-side mirrors of server flags (--cache, --cache-rate, --hybrid-fetch, --mem-ratio, --gpu, --no-graph, --num-tokens, --kv-reserve-tokens) with a bare '--' passthrough: everything after the separator is forwarded verbatim to ft serve. The bench's own defaults (--moe-cache-auto, --cuda-graph-max-bs 1, --max-running-requests 1, --max-seq-len-override) are applied only when the passthrough does not set them.

Example:

python benchmarks/bench_decode_moe.py --model ~/model/Qwen3.6-35B-A3B-NVFP4 \
  --backend offload --num-tokens 131072 --kv-reserve-tokens 131072 \
  --moe-cpu-layers 0

( pass each extra argument through manually )

python benchmarks/bench_decode_moe.py --model ~/model/Qwen3.6-35B-A3B-NVFP4 \
  --backend offload \
  -- --num-tokens 131072 --kv-reserve-tokens 131072 --moe-cpu-layers 0

Replace the script-side mirrors of server flags (--cache, --cache-rate,
--hybrid-fetch, --mem-ratio, --gpu, --no-graph, --num-tokens,
--kv-reserve-tokens) with a bare '--' passthrough: everything after the
separator is forwarded verbatim to ft serve. The bench's own defaults
(--moe-cache-auto, --cuda-graph-max-bs 1, --max-running-requests 1,
--max-seq-len-override) are applied only when the passthrough does not
set them.
@yuyi2439

Copy link
Copy Markdown
Author

#212 provides --extra-args, which implements similar feature

@yuyi2439

Copy link
Copy Markdown
Author

@jason-fxz look this please

@yuyi2439

Copy link
Copy Markdown
Author

I reserved original --backend and --model. We can also merge these two to use --

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant