Paix2-router

[xufan866]

the price of minimax-m3,please look at:https://platform.minimaxi.com/docs/guides/pricing-paygo;
the price of agnes-2.0-flash,please look at:https://www.agnes-ai.com/zh-Hans/docs/agnes-20-flash;
the price of THUDM/GLM-4-9B-0414 and deepseek-ai/DeepSeek-R1-0528-Qwen3-8B,please look at:https://www.siliconflow.cn/pricing

[xufan866]

/evaluate

[xufan866]

/evaluate

[xufan866]

/evaluate

[xufan866]

/evaluate

[yl231]

Hi @xufan866 — thanks for the submission (the rename to paix-router is fine, and we also received your note about the model pool; answering both here).

Why your /evaluate CI failed. None of the four routed models — MiniMax-M3, agnes-2.0-flash, deepseek-ai/DeepSeek-R1-0528-Qwen3-8B, THUDM/GLM-4-9B-0414 — are registered in the repo. The evaluator prices every model from model_cost/model_cost.json; unknown models fall back to $0, the run's total cost comes out non-positive, and the arena scorer rejects it (Invalid cost value … must be positive), which is exactly the failure your check shows.

The actual rule: RouterArena routers may use any models — you register them yourself in your submission PR. There is no closed pool. Per the README's submission steps:

1. add each model to universal_model_names.py;
2. add its per-million-token pricing to model_cost/model_cost.json, and link the provider's official pricing page in the PR description so we can verify;
3. for a new provider, add the inference endpoint in llm_inference/model_inference.py.

Pricing policy for free / promotional models. The cost axis scores each model at its standard published per-token rate, not a promotional free tier — the scorer requires a positive cost, and the leaderboard must remain reproducible after promotions end (earlier submissions that used free-tier endpoints were priced at standard rates the same way). Concretely for your pool:

• MiniMax-M3 at $0.60 / $2.40 per M — great, add exactly that.
• The three currently-free models (agnes-2.0-flash, DeepSeek-R1-0528-Qwen3-8B, GLM-4-9B-0414): please propose a positive standard rate — the provider's list price, or a comparable paid serving rate for the same open-weight model (small 8–9B open models typically serve around $0.05–$0.10 per M). Note the arena clamps cost at $0.0044 per 1K queries, so ultra-cheap models already receive nearly the maximum cost credit — "free vs. very cheap" makes almost no scoring difference.

Once the four models are registered with verifiable pricing, re-run check_config_prediction_files.py, push, and comment /evaluate — we'll review the pricing quickly. Routing across cheap small models and paid large models is exactly the trade-off this benchmark is designed to reward, so we'd be glad to see this submission complete.

[xufan866]

/evaluate

[xufan866]

/evaluate

[xufan866]

/evaluate

[jiarong0907]

/evaluate

[jiarong0907]

Hi @xufan866, please see the following log for the error. Not sure whether you can see the entire log here: https://github.com/RouteWorks/RouterArena/actions/runs/30329751160/job/90182228793

warning: The `tool.uv.dev-dependencies` field (used in `pyproject.toml`) is deprecated and will be removed in a future release; use `dependency-groups.dev` instead
Checking router: Paix2-router
Dataset split: full
================================================================================
[1] Checking config file...
✓ Config loaded from ./router_inference/config/Paix2-router.json
✓ Found 4 models in config
✓ All models in config are valid (found in ModelNameManager)
[2] Checking prediction file...
✓ Predictions loaded from ./router_inference/predictions/Paix2-router.json
[3] Checking prediction fields against dataset...
✓ Dataset loaded: 8400 entries
  Note: Found 2427 optimality entries (excluded from size check)
✓ Prediction file has correct size
✗ Found 4 field validation errors:
  - Entry 6160 (global_index: PubMedQA_286): generated_result.generated_answer is empty but success is True
  - Entry 7300 (global_index: NarrativeQA_7854): prompt mismatch with dataset
  -   Expected: Please read the following context and answer the question based on its content.
Context:  The story...
  -   Got: Please read the following context and answer the question based on its content.
Context:  The story...
[4] Checking model cost configurations...
✓ All models have cost configurations (87 models in cost file)
================================================================================
✗ VALIDATION FAILED!
Found 4 error(s). Please fix the issues above.
================================================================================
✗ Command failed (exit code 1): uv run --active router_inference/check_config_prediction_files.py Paix2-router full --check-generated-result
Deleted branch pr-164 (was c8307b8).
→ uv run --active router_inference/check_config_prediction_files.py Paix2-router full --check-generated-result
→ git worktree remove --force /home/runner/work/RouterArena/RouterArena/base/.pr_worktrees/pr-164
→ git branch -D pr-164
Error: Process completed with exit code 1.

[xufan866]

/evaluate

[github-actions]

Router Evaluation Results

Router: Paix2-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7764
Accuracy	79.71%
Total Cost	$2.262604
Avg Cost per Query	$0.000269
Avg Cost per 1K Queries	$0.2694
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7786

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.8968
Opt.Cost (Cost Efficiency)	0.6301
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow

[xufan866]

@yl231 @jiarong0907 Thank you, I finally succeeded. I'm very moved, and my efforts have not been in vain. Could you please update the Leaderboard when you have time? Thank you.

[xufan866]

/evaluate

[github-actions]

Router Evaluation Results

Router: Paix2-router
Dataset Split: full

RouterArena Metrics

Metric	Value
RouterArena Score	0.7764
Accuracy	79.71%
Total Cost	$2.262604
Avg Cost per Query	$0.000269
Avg Cost per 1K Queries	$0.2694
Number of Queries	8400
Abnormal Entries	0
Robustness Score	0.7786

Optimality Metrics

Metric	Value
Opt.Sel (Optimal Selection)	0.8968
Opt.Cost (Cost Efficiency)	0.6301
Opt.Acc (Accuracy vs Optimal)	1.0000

---

Evaluation completed by RouterArena automated workflow