LLM Benchmark — B2B Chatbot Stack (DBB Software, May 2026)

Open data + scripts behind our article Claude vs ChatGPT vs Gemini: Best LLM for B2B Chatbots 2026. We benchmarked 10 LLMs through our production chatbot's request path on the Vercel AI Gateway. Published so anyone can verify or re-run.

What's here

• benchmark-ai-models.mjs — single-turn inference latency (TTFT + total), 6 probes, raw fetch.
• benchmark-ai-models-with-tools.mjs — tool-call end-to-end pass/fail (Vercel AI SDK + stub tools).
• llm-benchmark-2026-05-20.json — the full results dataset (CC BY 4.0).

Methodology (honest scope)

• Harness: Vercel AI Gateway, one sequential call per (model × probe), no concurrency.
• Trials: N = 1 per pair. This is a directional snapshot, not a statistically rigorous benchmark.
• System prompt: a short, representative brand-voice prompt (well under 100 tokens) — deliberately minimal to isolate inference/tool-call latency from prompt size. It is not our full production prompt.
• Tools: deterministic in-process stubs that mimic our MCP tool shapes (no network round-trips).
• Region: Frankfurt (eu-central-1).
• Not measured: multi-turn drift, concurrent load, rate limits, context-window pressure. See the article for the full write-up and caveats.

Models (10)

Mistral Medium 3.5 · Claude 3 Haiku · Claude 3.5 Haiku · GPT-3.5 Turbo · GPT-4o-mini · Gemini 2.5 Flash · Gemini 2.5 Flash-Lite · Gemini 3 Flash · Gemini 3.5 Flash · Z.AI GLM 4.7 Flash

Re-run it

npm install
export AI_GATEWAY_API_KEY=your_vercel_ai_gateway_key   # or put it in .env
npm run benchmark          # single-turn latency
npm run benchmark:tools    # tool-call end-to-end

Requires Node.js >= 24. Your numbers will vary with region, time of day, and Gateway routing.

License

• Data (llm-benchmark-2026-05-20.json): CC BY 4.0 — cite DBB Software when reusing.
• Code (*.mjs): MIT.

Citation

DBB Software (2026). LLM Benchmark — B2B Chatbot Stack. https://github.com/DBB-Software/llm-benchmark-2026