feat(copilot): read per-request input/cache from session-store.db

CHANGELOG.md

@@ -2,6 +2,9 @@
 
 ## Unreleased
 
+### Added (CLI)
+- **Copilot input/cache tokens are read per request from `~/.copilot/session-store.db`.** Previously, codeburn relied on `session.shutdown` rollups from the Copilot CLI and GitHub Copilot desktop app. Those rollups are written only after a clean shutdown, stamp all usage on the shutdown day, and reset their counters at in-session compaction — so a crash could lose an entire session's input/cache usage, and even cleanly-closed long sessions were silently truncated. On one machine with long history, reading the per-request rows recovered about 35% of actual Copilot spend. Covered sessions now use per-request tokens with their real timestamps, counted exactly once against existing rollups and never added as extra calls or turns. Pre-store CLI sessions continue using the unchanged rollup path, and a locked or unreadable store defers only its own re-read instead of prematurely sealing daily history. Copilot reasoning tokens are also no longer double-billed: they are a subset of output already priced through the per-turn calls. This triggers a one-time re-parse, with the daily cache bumped from v17 to v19 to re-derive finalized days. (#946)
+
 ## 0.9.20 - 2026-08-10
 
 ### Added

docs/providers/copilot.md

@@ -8,15 +8,17 @@ GitHub Copilot Chat (CLI, VS Code core chat sessions, VS Code extension transcri
 
 ## Where it reads from
 
-Three JSONL locations plus an optional OpenTelemetry SQLite source (see below). OTel is
-preferred when present; chatSessions are only discovered when no OTel source is found.
-Other discovered sources are walked on every run; results merge and dedupe.
+Three JSONL locations plus two optional SQLite sources (see the OTel and session-store
+sections). OTel is preferred when present; chatSessions are only discovered when no OTel
+source is found. Other discovered sources are walked on every run; results merge and
+dedupe.
 
 1. **Legacy CLI sessions:** `~/.copilot/session-state/`
 2. **VS Code core chat sessions:** `~/Library/Application Support/Code/User/workspaceStorage/<hash>/chatSessions/*.jsonl` plus `~/Library/Application Support/Code/User/globalStorage/emptyWindowChatSessions/*.jsonl` and equivalents on Windows / Linux
 3. **VS Code transcripts:** `~/Library/Application Support/Code/User/workspaceStorage/<hash>/GitHub.copilot-chat/transcripts/` and equivalents on Windows / Linux
 4. **OTel SQLite store:** VS Code Copilot Chat's `agent-traces.db` (see the OTel section). Preferred when present because it carries full input / output / cache token counts; legacy JSONL sources only record output tokens.
-5. **JetBrains IDE sessions:** `~/.config/github-copilot/<ide>/<kind>/<storeId>/copilot-*-nitrite.db` (see the JetBrains section). Covers IntelliJ IDEA, PyCharm, RubyMine, etc.
+5. **CLI session store:** `~/.copilot/session-store.db` (see the session-store section). One `assistant_usage_events` row per API request — the authoritative input/cache source for CLI and GitHub desktop-app sessions.
+6. **JetBrains IDE sessions:** `~/.config/github-copilot/<ide>/<kind>/<storeId>/copilot-*-nitrite.db` (see the JetBrains section). Covers IntelliJ IDEA, PyCharm, RubyMine, etc.
 
 ## Storage format
 
@@ -45,6 +47,52 @@ instead of trying to dedupe across stores.
   before the upgrade cannot be recovered, so monotonicity starts from the upgrade point,
   not retroactively.
 
+## Session store (CLI / GitHub desktop app)
+
+The Copilot CLI and the GitHub Copilot desktop app both write
+`~/.copilot/session-store.db` (SQLite/WAL) unconditionally. Its
+`assistant_usage_events` table records one row per API request as it happens —
+where the `session.shutdown` rollup in `events.jsonl` is written only on clean
+shutdown (a crash loses the leg's input/cache accounting), lumps each leg into
+one per-model total, and resets its counters at in-session compaction. Rows are
+therefore authoritative for input / cache-read / cache-write / reasoning
+tokens, with real per-request timestamps; per-turn output stays owned by the
+`events.jsonl` `assistant.message` calls. `input_tokens` is cache-INCLUSIVE
+(input + cache_read + cache_write), the same convention as the rollups; the
+parser emits the uncached remainder. Override the path with
+`CODEBURN_COPILOT_SESSION_STORE_DB` (deliberately NOT in the env fingerprint —
+see the #927 ruling in `src/session-cache.ts`).
+
+- **Rollup reconciliation happens at serve time, per (session, model), in
+  `parseProviderSources`** — never in the parser. Both representations always
+  parse and cache; wherever store rows exist for a pair, the rollup calls are
+  dropped and each rollup leg's usage beyond the rows in its own interval
+  serves once as a synthesized residual call at that leg's timestamp. Sessions
+  with no rows (pre-store CLI builds) keep the rollup path unchanged.
+- **Behavioral weight.** Rollups and residuals are aggregate accounting: tokens
+  and cost count, but never api-call / model-call / turn weight. A store row
+  pairs with its per-turn call by timestamp adjacency (2-minute window,
+  computed over the full serve set); only unpaired rows — crash-recovered,
+  store-only requests — count as calls.
+- **Failure semantics.** True absence (ENOENT, no sqlite driver, `no such
+  table/column` from pre-store CLI builds) reads as absent — no source, rollups
+  rule. Every other failure (locked, EACCES, corrupt, mid-replace) emits the
+  source anyway: its parse defers on the busy shape, previously cached rows
+  keep serving, and the pass reports incomplete hydration so the daily
+  backfill holds its watermark.
+- **Durable cache.** Rides the same `durableSources` union as OTel, with one
+  scoped difference: the store declares `retainWhilePresent`, so its rows are
+  never evicted while the DB exists (crash-only rows have no rollup to fall
+  back to); every other copilot source keeps the standard durable schedule,
+  aging out at 90 days whether or not the file remains. A deleted store's rows
+  serve as orphans until the 90-day age-out. Reconciliation reads only cached
+  contents, so deleting or resetting the store never changes served totals.
+- **Billing metadata.** Each row's `total_nano_aiu` and `request_multiplier`
+  are captured onto the cached call when the store's schema has them (older
+  stores parse identically without). Nothing prices or displays them yet —
+  billing-grade cost is upstream #890.
+- **Requires Node 22+** (`node:sqlite`), same as the OTel source.
+
 ## JetBrains IDEs (IntelliJ, PyCharm, …)
 
 The JetBrains Copilot plugin does **not** write to any of the VS Code or CLI
@@ -160,11 +208,11 @@ surfaces, add a reader with a captured fixture.)
 
 ## Caching
 
-None for the JSONL sources. The OTel source uses a durable cache (see above).
+None for the JSONL sources. The OTel and session-store sources use the durable cache (see above).
 
 ## Deduplication
 
-Legacy JSONL and transcript sessions dedupe per `messageId`. Core chat sessions dedupe per `copilot-chatsession:<sessionId>:<requestId>`, and are not discovered when an OTel source is present. JetBrains `.db` turns dedupe per `copilot:jb:<conversationId>:<turnIndex>` (a per-conversation index, plus reply-content dedup within each conversation). These sources otherwise touch disjoint locations from the VS Code / CLI sources.
+Legacy JSONL and transcript sessions dedupe per `messageId`. Core chat sessions dedupe per `copilot-chatsession:<sessionId>:<requestId>`, and are not discovered when an OTel source is present. Session-store rows dedupe per `copilot-store:<sessionId>:<rowId>:<hash>` (the hash covers `created_at`, token counts, and model, so a same-path DB reset reusing AUTOINCREMENT ids cannot alias a different request onto a cached key); shutdown rollups per `copilot:<sessionId>:shutdown:<model>:<n>`, with serve-time residuals synthesized (never cached) under `copilot:<sessionId>:shutdown-residual:<model>:<leg>`. JetBrains `.db` turns dedupe per `copilot:jb:<conversationId>:<turnIndex>` (a per-conversation index, plus reply-content dedup within each conversation). These sources otherwise touch disjoint locations from the VS Code / CLI sources.
 
 If a workspace hash contains at least one `chatSessions/*.jsonl` file, the provider skips that hash's legacy `GitHub.copilot-chat/transcripts/` directory. The core chat session journal is the modern token-bearing source for the same conversations, so reading both would inflate call counts.
 

src/audit-report.ts

@@ -1,3 +1,4 @@
+import { isBehavioralCall } from './behavioral-weight.js'
 import { getModelCosts, type ModelCosts } from './models.js'
 import { getProvider } from './providers/index.js'
 import { formatCost, formatTokens } from './format.js'
@@ -98,7 +99,9 @@ export async function aggregateAudit(projects: ProjectSummary[]): Promise<AuditR
           // cache-read vocabularies, so the audit's displayed total matches.
           bucket.cacheReadDisplayed += Math.max(u.cacheReadInputTokens, u.cachedInputTokens)
           bucket.attributedCostUSD += call.costUSD
-          bucket.calls += 1
+          // Supplementary accounting calls keep their tokens and cost above but are not
+          // distinct requests, so they add no call weight (see behavioral-weight.ts).
+          if (isBehavioralCall(call)) bucket.calls += 1
         }
       }
     }
@@ -120,9 +123,14 @@ export async function aggregateAudit(projects: ProjectSummary[]): Promise<AuditR
   const rows: AuditRow[] = []
   for (const bucket of buckets.values()) {
     const meta = await resolveProvider(bucket.provider)
+    // Buckets are keyed by (provider, model), so this provider test covers every call in one.
+    // Copilot reasoning tokens are already INSIDE outputTokens (same rule as cachedCallToApiCall
+    // in parser.ts), so folding them in would display phantom output for its store rows/rollups.
     const displayed = {
       inputTokens: bucket.raw.inputTokens,
-      outputTokens: bucket.raw.outputTokens + bucket.raw.reasoningTokens,
+      outputTokens: bucket.provider === 'copilot'
+        ? bucket.raw.outputTokens
+        : bucket.raw.outputTokens + bucket.raw.reasoningTokens,
       cacheWriteTokens: bucket.raw.cacheCreationInputTokens,
       cacheReadTokens: bucket.cacheReadDisplayed,
     }
@@ -207,7 +215,7 @@ export function renderAuditTable(rows: AuditRow[]): string {
   const legend = [
     '',
     'Columns are the raw token fields each provider records. codeburn then normalizes for pricing:',
-    '  - Reason folds into Output (priced output = output + reasoning)',
+    '  - Reason folds into Output (priced output = output + reasoning), except copilot, whose reasoning is already inside its output',
     '  - Cache rd = max(Anthropic cacheReadInput, OpenAI cached), since providers fill one or both',
     '  - Cache wr is priced at 1.25x the input rate, Cache rd at 0.1x, when a model omits explicit cache rates',
     'Use --format json for per-component cost, the rates applied, and both raw cache-read fields.',

src/behavioral-weight.ts

@@ -0,0 +1,35 @@
+// Behavioral weight — the single definition of which served calls count as
+// requests. A copilot shutdown rollup, synthesized residual, or store row
+// paired with its per-turn call (`supplementaryAccounting`, assigned at serve
+// time) carries real tokens and cost but is not a distinct behavioral
+// request. Every user-visible calls/turns counter weighs calls through these
+// helpers so no surface can disagree with the session summaries or the
+// sealed daily history. Token and cost sums intentionally keep every call —
+// supplementary accounting must never be filtered out, only weightless.
+
+type WeightedCall = { supplementaryAccounting?: boolean }
+type WeightedTurn = { assistantCalls: readonly WeightedCall[] }
+
+/** True when the call is a real request (weight 1); false for supplementary accounting (weight 0). */
+export function isBehavioralCall(call: WeightedCall): boolean {
+  return !call.supplementaryAccounting
+}
+
+/** Number of real requests among a turn's calls. */
+export function behavioralCallCount(calls: readonly WeightedCall[]): number {
+  let n = 0
+  for (const call of calls) if (!call.supplementaryAccounting) n++
+  return n
+}
+
+/** True when the turn holds at least one behavioral call — supplementary-only turns add no turn/edit weight. */
+export function isBehavioralTurn(turn: WeightedTurn): boolean {
+  return turn.assistantCalls.some(call => !call.supplementaryAccounting)
+}
+
+/** Number of behavioral turns (weightless accounting containers excluded). */
+export function behavioralTurnCount(turns: readonly WeightedTurn[]): number {
+  let n = 0
+  for (const turn of turns) if (isBehavioralTurn(turn)) n++
+  return n
+}

src/compare-stats.ts

@@ -1,11 +1,20 @@
 import { readdir, readFile } from 'fs/promises'
 import { join } from 'path'
 
-import type { ProjectSummary } from './types.js'
+import type { ClassifiedTurn, ProjectSummary } from './types.js'
+import { isBehavioralCall } from './behavioral-weight.js'
 import { getShortModelName } from './models.js'
 
 const PLANNING_TOOLS = new Set(['TaskCreate', 'TaskUpdate', 'TodoWrite', 'EnterPlanMode', 'ExitPlanMode'])
 
+/// The turn's primary model: the model of its FIRST BEHAVIORAL call. Undefined
+/// when the turn has none — accounting-only turns carry no behavioral evidence,
+/// so they are excluded from efficiency comparisons (their spend still appears
+/// in every spend report).
+function primaryTurnModel(turn: ClassifiedTurn): string | undefined {
+  return turn.assistantCalls.find(isBehavioralCall)?.model
+}
+
 export type ModelStats = {
   model: string
   calls: number
@@ -39,9 +48,8 @@ export function aggregateModelStats(projects: ProjectSummary[]): ModelStats[] {
   for (const project of projects) {
     for (const session of project.sessions) {
       for (const turn of session.turns) {
-        if (turn.assistantCalls.length === 0) continue
-        const primaryModel = turn.assistantCalls[0]!.model
-        if (primaryModel === '<synthetic>') continue
+        const primaryModel = primaryTurnModel(turn)
+        if (primaryModel === undefined || primaryModel === '<synthetic>') continue
 
         const ms = ensure(primaryModel)
         ms.totalTurns++
@@ -57,7 +65,7 @@ export function aggregateModelStats(projects: ProjectSummary[]): ModelStats[] {
         for (const call of turn.assistantCalls) {
           if (call.model === '<synthetic>') continue
           const cs = call.model === primaryModel ? ms : ensure(call.model)
-          cs.calls++
+          if (isBehavioralCall(call)) cs.calls++
           cs.cost += call.costUSD
           cs.outputTokens += call.usage.outputTokens
           cs.inputTokens += call.usage.inputTokens
@@ -224,8 +232,7 @@ export function computeCategoryComparison(projects: ProjectSummary[], modelA: st
   for (const project of projects) {
     for (const session of project.sessions) {
       for (const turn of session.turns) {
-        if (turn.assistantCalls.length === 0) continue
-        const primary = turn.assistantCalls[0]!.model
+        const primary = primaryTurnModel(turn)
         if (primary !== modelA && primary !== modelB) continue
 
         const acc = ensure(primary === modelA ? mapA : mapB, turn.category)
@@ -272,17 +279,17 @@ export function computeWorkingStyle(projects: ProjectSummary[], modelA: string,
   for (const project of projects) {
     for (const session of project.sessions) {
       for (const turn of session.turns) {
-        if (turn.assistantCalls.length === 0) continue
-        const primary = turn.assistantCalls[0]!.model
+        const primary = primaryTurnModel(turn)
         if (primary !== modelA && primary !== modelB) continue
 
         const s = primary === modelA ? sA : sB
         s.totalTurns++
-        const turnTools = turn.assistantCalls.flatMap(c => c.tools)
-        if (turnTools.some(t => PLANNING_TOOLS.has(t)) || turn.assistantCalls.some(c => c.hasPlanMode)) {
+        const behavioralCalls = turn.assistantCalls.filter(isBehavioralCall)
+        const turnTools = behavioralCalls.flatMap(c => c.tools)
+        if (turnTools.some(t => PLANNING_TOOLS.has(t)) || behavioralCalls.some(c => c.hasPlanMode)) {
           s.planModeUses++
         }
-        for (const call of turn.assistantCalls) {
+        for (const call of behavioralCalls) {
           s.totalToolCalls += call.tools.length
           if (call.hasAgentSpawn) s.agentSpawns++
           if (call.speed === 'fast') s.fastModeCalls++

src/daily-cache.ts

@@ -5,7 +5,24 @@ import { homedir } from 'os'
 import { join } from 'path'
 import type { DateRange, ProjectSummary } from './types.js'
 
-// Bumped to 17: copilot CLI sessions were misclassified as VS Code transcripts
+// Bumped to 19, deliberately skipping 18: an earlier head of this same change
+// (#946) was pushed publicly claiming v18 under DIFFERENT accounting — no
+// supplementary-call weighting, whole-session rollup suppression — and
+// adoptOlderDailyCaches/isMigratableCache would take those days forward as
+// finalized without re-deriving them. A distinct version is the only thing
+// that stops one version number meaning two accountings.
+//
+// v18 (never released): copilot input/cache tokens for sessions covered by the CLI's
+// session-store.db moved from one shutdown-rollup lump (stamped at session
+// end) to per-request DB rows with real timestamps, supplementary accounting
+// calls (rollups, covered rows) stopped counting as api/model calls, and
+// reasoning tokens left the copilot cost recompute (they are inside the
+// output the per-turn calls already bill). Per-day attribution, call counts
+// and costs all move, so days finalized at v17 would disagree with the live
+// parse. Re-derivation rides the v14 carry-forward semantics; sourceless
+// days carry forward as-is.
+//
+// v17: copilot CLI sessions were misclassified as VS Code transcripts
 // (#944), so days finalized at v16 or earlier carry output-only copilot costs —
 // the session.shutdown rollup's input/cache tokens were dropped. Raising
 // MIN_SUPPORTED_VERSION forces the one-time re-derivation under the
@@ -73,8 +90,8 @@ import type { DateRange, ProjectSummary } from './types.js'
 // that older binaries skipped. v8 added local-model savings to the daily
 // rollup; the `savingsConfigHash` field is invalidated separately when the
 // user changes their `localModelSavings` mapping.
-export const DAILY_CACHE_VERSION = 17
-const MIN_SUPPORTED_VERSION = 17
+export const DAILY_CACHE_VERSION = 19
+const MIN_SUPPORTED_VERSION = 19
 // Version-suffixed so different binaries each own a distinct file and never
 // clobber an incompatible schema. Bumping the version mints a fresh filename;
 // adoptOlderDailyCaches then unions days out of every previous file (including

src/dashboard.tsx

@@ -14,6 +14,7 @@ import { scanAndDetect, type WasteFinding, type WasteAction, type OptimizeResult
 import { aggregateFileChurn, buildCoachingNotes, computePricingCoverage, medianTimeToFirstEditMs, scanUserCorrections, worstOneShotCategory, type ReworkedFile } from './workflow-insights.js'
 import { estimateContextBudget, type ContextBudget } from './context-budget.js'
 import { dateKey } from './day-aggregator.js'
+import { behavioralCallCount } from './behavioral-weight.js'
 import { CompareView } from './compare.js'
 import { getPlanUsages, type PlanUsage } from './plan-usage.js'
 import { planDisplayName } from './plans.js'
@@ -431,7 +432,7 @@ export function getDailyActivityRows(projects: ProjectSummary[]): DailyActivityR
         if (!turn.timestamp) continue
         const day = dateKey(turn.timestamp)
         dailyCosts[day] = (dailyCosts[day] ?? 0) + turn.assistantCalls.reduce((s, c) => s + c.costUSD, 0)
-        dailyCalls[day] = (dailyCalls[day] ?? 0) + turn.assistantCalls.length
+        dailyCalls[day] = (dailyCalls[day] ?? 0) + behavioralCallCount(turn.assistantCalls)
       }
     }
   }