AI Charts

AI Charts is an open-source home for sourced, interactive AI benchmark charts. It compares models and agents across performance, cost, speed, and token use without collapsing those trade-offs into one rank.

The current chart focuses on coding agents. That is the first published comparison in a broader product for AI model and agent benchmarks, not the limit of the AI Charts brand.

Current chart: coding agents

The site is a static-data Next.js and TypeScript application. Its coding-agent snapshot is committed to the repository, validated at build time, and refreshed automatically every day. Production never depends on the upstream data source being available during a request.

• Compare Artificial Analysis's AA Index, DeepSWE, Terminal-Bench v2.1, and SWE-Atlas-QnA results.
• Plot each result against cost, duration, or total token use.
• Pin a model to see its nearby performance cohort, or pin a provider to inspect its range.
• Explore the cost/performance Pareto frontier and per-provider score ranges.
• Follow a checked timeline of newly detected models, settings, and material benchmark changes.
• Share the current axes and selection as a link or export a full-resolution PNG.
• Open a profile-specific benchmark card for each model, then share its branded image through the native share sheet, download it, or post its URL from /models. Cataloged identities and settings use stable canonical routes; newly observed identities or profile settings use deterministic provisional routes until reviewed. Every curated model carries a checked first-party release date and source shared by all of its profiles.
• Read sourced analysis at /blog, including the current AA Index versus cost snapshot analysis, open models on coding-agent benchmarks, why a closing benchmark gap does not settle product choice, how cheaper AI models can make everyday products viable, why a high score still needs a holdout, and why coding-agent scores still need expertise.
• Inspect the current snapshot, methodology, provenance, full configuration table, and machine-readable distribution at /data.

The checked snapshot records its source URL, retrieval time, and material update history. The /data page exposes the same checked facts and links to a JSON distribution so people, search engines, and answer engines can verify what the chart shows. AI Charts is an independent visualization and is not affiliated with Artificial Analysis or the model providers represented in the data.

GPT subsidy history

/gpt-subsidy tracks the measured API-retail-equivalent value of one user’s available local Codex logs. Each daily point covers seven complete UTC days. The collector globally deduplicates parent, child-agent, active, and archived session events, values each recorded model with the checked gpt-subsidy-pricing.json rate manifest, and publishes only aggregate token and dollar totals. The checked gpt-subsidy-measurement.json manifest pins the parser, adapter, updater, and rolling-window constants so a methodology change cannot silently mix unlike historical points.

This is a personal usage trace, not a platform-wide estimate or a representative sample of ChatGPT Pro usage. Historical logs span account switches without durable account attribution. The public history therefore leaves the subscription-adjusted multiple null instead of dividing all usage by one $200 subscription. It does not publish a monthly projection or one-plan normalization. API-key or otherwise API-billed usage, purchased credits, free resets, and temporary promotions still cannot be separated. The page keeps those limits beside the chart and publishes the calculation and source links in static HTML.

Local development

AI Charts uses Bun 1.3.14 and Node.js 24.

bun install --frozen-lockfile
bun run dev

Open http://localhost:3000. Analytics are disabled outside production on aicharts.io, so local development does not send PostHog events.

Run the complete local gate before opening a pull request:

bun run check

That validates generated files and the checked data contract, runs strict TypeScript and ESLint, executes example and property tests, and creates a production build.

Data refresh

The data-refresh.yml workflow checks OpenRouter discovery and direct DeepSWE evidence every four hours, adds the heavier Artificial Analysis refresh daily at 10:43 UTC, and can run either mode manually. Poll-metadata-only checks remain visible in Actions without creating a data pull request. It treats model discovery, comparable benchmark ingestion, and early benchmark evidence as separate failure domains:

1. OpenRouter's public models API runs first. It supplies a bounded 90-day release radar for the providers already represented by the site and the exact model-ID catalog used to identify direct benchmark observations. The radar keeps text-output models with tool support, filters aliases and hosted variants, and records OpenRouter's listing timestamp as discovery metadata—not as a claimed release date. Its checked snapshot retains a durable observedListings ledger for identity reconciliation only.
2. Artificial Analysis supplies the comparable AA Index, DeepSWE, Terminal-Bench v2.1, and SWE-Atlas-QnA observations used by the chart and cards. The importer parses the public Next.js Flight payload, reconciles upstream slug changes against stable model semantics, and rejects duplicates, suspicious row loss, and metric-coverage regressions. Release status is reconciled after both source attempts using the valid checked snapshots that remain on disk. A release leaves the incomplete radar only after one observation contains the complete four-benchmark index.
3. DataCurve's official DeepSWE v1.1 live artifact supplies earlier harness-specific pass@1 evidence. OpenRouter remains identity-only here because its benchmark catalog does not expose this direct coding-agent harness dataset. The importer requires one exact semantic match against the full OpenRouter model-ID tail and preserves that checked identity across partial responses or outages. A unique Artificial Analysis model identity is the fallback for a previously unseen model OpenRouter does not resolve. Ambiguous matches fail closed, unmatched models remain explicit, and every observation retains its harness, effort, run count, attempts, and source provenance in data/deep-swe-evidence.json.
4. The radar is discovery-only, and direct DeepSWE observations are early-evidence-only. Direct evidence alone cannot create an Artificial Analysis score, chart point, model card, or synthesized AA Index. Partial Artificial Analysis observations can appear on the chart and cards with missing metrics left explicit while their release remains on the incomplete radar. The homepage now opens on the DeepSWE axis, using only the comparable DeepSWE values in the Artificial Analysis snapshot.
5. The workflow atomically updates only data/model-release-radar.json, data/coding-agents.json, and data/deep-swe-evidence.json, runs the full project check, opens a dedicated automation pull request, explicitly dispatches required CI on its head commit, waits for that exact run, and verifies the protected squash auto-merge completed. Data-only pull requests are excluded from the redundant pull_request CI event because GitHub holds workflows opened by GITHUB_TOKEN for manual approval. After exact-head CI succeeds, the trusted publisher records the required GitHub Actions commit status linked to that run because workflow_dispatch check runs are not included in the pull request's merge rollup. The publisher never bypasses the repository's protected-branch rules.
6. Official card dates live in the manually reviewed data/model-release-dates.json ledger, keyed by stable canonical model ID. Each verified record cites a provider-owned announcement, release note, or official model index. OpenRouter timestamps never populate this ledger, never drive the New sort, and never appear as release dates. Missing catalog coverage, malformed dates, or non-provider evidence fail the network-free release-date check.

The repository keeps default workflow-token permissions read-only and grants write capabilities only inside this workflow. GitHub's repository-level “Allow GitHub Actions to create and approve pull requests” setting must remain enabled so that the scoped token can open its data PR; the workflow never submits reviews. The sources refresh independently, so one outage does not discard a valid update from the other. Dependency installation is retried, and any unhealthy run creates or updates one durable GitHub issue that closes automatically after the next healthy full refresh. Source-shape changes, suspicious data loss, failed required CI, and unmerged update PRs all fail closed, leaving the last-known-good snapshot in production.

To refresh locally:

bun run releases:refresh
bun run data:refresh
bun run releases:reconcile
bun run deepswe:refresh
bun run check

Review the resulting data diff before committing it. bun run data:check, bun run releases:check, bun run release-dates:check, and bun run deepswe:check are network-free validations of the committed snapshots. bun run releases:reconcile updates only radar benchmark statuses from the checked Artificial Analysis snapshot and remains safe when either upstream source is temporarily unavailable. Official-date corrections remain reviewed source edits; scheduled discovery never overwrites them.

PostHog

PostHog is initialized only in production on the canonical AI Charts domains. The browser configuration is cookieless and privacy constrained:

• no person profiles, persistent identifiers, autocapture, session replay, surveys, heatmaps, or feature flags;
• memory-only persistence, Do Not Track support, masked text and element attributes;
• page-view, page-leave, Core Web Vitals, and four explicit product event families only.

The product events are chart metric selected, chart selection pinned, chart shared, and content chart opened. They contain controlled enum-like properties, never chart URLs, query strings, free-form text, or model-level user data. Every event also receives a bounded page classification so acquisition and engagement can be compared without storing article slugs or query strings.

The durable positioning, search-intent map, technical invariants, event schema, baseline, and review cadence live in docs/seo-strategy.md. Search Console measures impressions, queries, clicks, click-through rate, and search position. PostHog measures acquisition and qualified engagement after a visitor arrives.

Copy .env.example to .env.local to exercise configuration. The public project token and ingest host are safe browser variables. POSTHOG_API_KEY is a private build credential used only to upload production source maps; never expose it through a NEXT_PUBLIC_ variable.

Deployment

The production site is deployed from main with Vercel. The repository-level vercel.json pins the Bun install and build commands. Configure these environment variables in Vercel:

Variable	Scope	Purpose
NEXT_PUBLIC_POSTHOG_KEY	Production, Preview	Public PostHog project token
NEXT_PUBLIC_POSTHOG_HOST	Production, Preview	Regional PostHog ingest host
POSTHOG_API_KEY	Production	Private key for source-map upload
POSTHOG_PROJECT_ID	Production	Numeric PostHog project ID
POSTHOG_UI_HOST	Production	https://us.posthog.com or https://eu.posthog.com

Production source maps are uploaded only when all private build settings and Vercel's commit SHA are present, then removed from the deployment output.

Repository map

• app/ contains the App Router chart, sourced benchmark notes, metadata, error states, and product styling.
• components/ contains the interactive chart, model cards, update timeline, linked summaries, sharing, export, and local UI primitives.
• lib/ contains strict data and model-card boundaries, chart math, deterministic layout, analytics events, and property tests.
• data/ contains the checked benchmark, discovery radar, official model-release ledger, model-card catalog, subsidy-history, and pricing snapshots.
• scripts/ contains the guarded benchmark and release-radar refreshes, local subsidy collector and publisher, and deterministic color generator.
• styles/ contains the portable plain-publication styles used by the benchmark notes.
• docs/ contains the current search, measurement, and engineering strategy.
• .github/workflows/ contains CI and daily refresh automation.

License and data notice

The application code is available under the MIT License. The repository also contains normalized public facts sourced from Artificial Analysis, the OpenRouter Models API, the DataCurve DeepSWE leaderboard, and the first-party model-provider pages cited in the official release ledger. The MIT license does not grant rights to third-party data, names, logos, or trademarks; see NOTICE.md.

Citation

GitHub can generate a citation from CITATION.cff. Cite AI Charts when referring to this software or its visualization method, Artificial Analysis or DataCurve DeepSWE for their respective measurements, OpenRouter for discovery and model-identity metadata, and the linked first-party provider source for an official release date. Include the source URL, retrieval date, selected metrics, harness, and configuration when a claim depends on a particular snapshot.

Contributions are welcome. Start with CONTRIBUTING.md, and report security issues through the process in SECURITY.md.