Skip to content

refactor(storage): replace replay Artifact paths with a SQLite context-offload store #4071

Description

@likun666661

Problem

Discussion #4030 separated three decisions that the current ArtifactStore conflates:

  1. the immediate causal fix for fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027;
  2. logical ownership and product surfaces;
  3. the physical byte-storage substrate.

This issue owns one cohesive domain only: Agent context offload. A model-visible value is too large to keep inline, durable replay bytes are stored elsewhere, Session/Event history retains an authorized reference, and a later replay can resolve the complete value.

Today the two load-bearing producers use the generic Artifact authority:

  • Read image snapshots, capped at 5 MiB;
  • Tool Result archives, rejected above 4 MiB.

Both consumers currently hydrate and verify the complete value. ArchiveRead applies offset/limit only after whole-object hydration; it is response pagination, not physical range I/O.

The current authority stores metadata in SQLite and payloads in the filesystem, then implements cross-domain publication, purge intent, stable-path recovery, global metadata reload/rewrite, and payload-tree inspection. On the motivating workspace, approximately 11.7k records / 6 GB are enough to block Host readiness for minutes because the access pattern is O(all records/files), and Session retirement can become O(M×N). That is a control-plane design failure, not a scale that should require a custom object store.

Decision

Use a dedicated SQLite database as the first physical substrate for capped, whole-object context offload.

Runtime consumers
  ├── ReadImageSnapshotStore
  └── ToolResultArchiveStore
            │ typed Session-owned references
            ▼
     ContextOffloadStore
            │ one SQLite transaction
            ▼
     context-offload.sqlite
       ├── context_blobs
       ├── context_refs
       └── usage counters

This database is separate from runtime.sqlite so multi-gigabyte replay payloads do not inflate operational-state backup, migration, or ordinary query paths. Blob bytes and context references live in the same database and commit atomically.

The current storage contract remains capped whole-object. This issue does not add streaming writes or claim true range reads. node:sqlite does not expose sqlite3_blob_open/read, and SQL substr(BLOB, ...) limits the returned value but does not guarantee bounded SQLite-side materialization.

If a future requirement needs genuinely large paged archives, evaluate fixed-size SQLite chunk rows with independently verifiable chunk integrity first. Add an external filesystem/S3-like CAS only after measured large-object or streaming workloads prove that chunked SQLite is insufficient.

Goals

  • Move new Read image snapshots and Tool Result archives off ArtifactStore.
  • Make blob + typed reference creation one SQLite transaction.
  • Preserve whole-object integrity verification and Session authorization.
  • Make Session copy copy references, not payload bytes.
  • Make Session retirement proportional to references owned by that Session.
  • Enforce explicit per-blob, per-Session logical, and workspace physical quotas.
  • Keep payload enumeration, reconciliation, GC, checkpoint, vacuum, and compaction off Host Ready.
  • Migrate still-referenced legacy replay payloads without rewriting immutable RuntimeEvents.
  • Stop producing unconsumed Write/Edit/Bash derivation Artifacts independently.

Non-goals

  • Redesigning Deep Research, upload, subagent write-back, or Desktop Artifact surfaces.
  • Moving Session recap effect state; it belongs in operational state and can be handled separately.
  • A generic replacement for every ArtifactSource.
  • Large-object streaming or physical range-read semantics.
  • Long-lived dual writes between ArtifactStore and the new context store.
  • Running full VACUUM, GC, or legacy cleanup before Host Ready.

Durable contracts

Core reference

Add a durable reference that names logical Session-owned context, not a filesystem path:

export interface SessionContextRef {
  readonly kind: 'session_context';
  readonly sessionId: string;
  readonly refId: string;
}

StorageRef accepts this variant for new Read image snapshots. Persisted session_file values remain a legacy input during the bounded cutover.

Tool Result placeholders move to a new rewrite version with contextRefId. The decoder continues to accept v1 artifactId placeholders. Migration preserves ref_id = legacy artifactId where possible, so old immutable RuntimeEvents resolve through the new store without event rewriting.

Storage authority

export type ContextOffloadOwner =
  | {
      readonly kind: 'read_image_snapshot';
      readonly ownerId: string;
    }
  | {
      readonly kind: 'tool_result_archive';
      readonly ownerId: string;
    };

export interface ContextOffloadRecord {
  readonly refId: string;
  readonly sessionId: string;
  readonly owner: ContextOffloadOwner;
  readonly blobId: string;       // canonical lowercase SHA-256
  readonly sizeBytes: number;
  readonly mediaType: string;
  readonly createdAt: number;
}

export type ContextOffloadPutResult =
  | { readonly ok: true; readonly record: ContextOffloadRecord }
  | {
      readonly ok: false;
      readonly reason:
        | 'too_large'
        | 'session_quota_exceeded'
        | 'workspace_quota_exceeded'
        | 'identity_conflict'
        | 'unavailable';
    };

export type ContextOffloadReadResult =
  | {
      readonly ok: true;
      readonly record: ContextOffloadRecord;
      readonly bytes: Uint8Array;
    }
  | {
      readonly ok: false;
      readonly reason:
        | 'not_found'
        | 'session_mismatch'
        | 'too_large'
        | 'corrupt'
        | 'unavailable';
    };

export interface ContextOffloadStore {
  put(input: {
    readonly sessionId: string;
    readonly owner: ContextOffloadOwner;
    readonly bytes: Uint8Array;
    readonly mediaType: string;
    readonly expectedSha256?: string;
  }): Promise<ContextOffloadPutResult>;

  read(input: {
    readonly sessionId: string;
    readonly refId: string;
    readonly maxBytes: number;
  }): Promise<ContextOffloadReadResult>;

  copyReferences(input: {
    readonly sourceSessionId: string;
    readonly targetSessionId: string;
    readonly references: readonly {
      readonly sourceRefId: string;
      readonly targetOwner: ContextOffloadOwner;
    }[];
  }): Promise<{
    readonly copied: readonly {
      readonly sourceRefId: string;
      readonly targetRefId: string;
    }[];
  }>;

  releaseReference(input: {
    readonly sessionId: string;
    readonly refId: string;
  }): Promise<void>;

  retireSession(sessionId: string): Promise<{
    readonly releasedReferences: number;
    readonly releasedLogicalBytes: number;
  }>;

  collectGarbage(input: {
    readonly olderThan: number;
    readonly maxBlobs: number;
    readonly maxBytes: number;
  }): Promise<{
    readonly deletedBlobs: number;
    readonly deletedBytes: number;
    readonly hasMore: boolean;
  }>;

  usage(sessionId?: string): Promise<{
    readonly references: number;
    readonly logicalBytes: number;
    readonly physicalBytes: number;
  }>;

  close(): void;
}

The public interface remains asynchronous even if the first implementation uses DatabaseSync, preserving the option to move storage work to a dedicated worker without changing consumers.

ownerId is stable within a Session. A retry with the same (sessionId, owner.kind, ownerId) and identical bytes returns the existing reference. Different bytes for the same identity fail with identity_conflict.

Typed consumer facades

The physical store does not expose ArtifactSource, product visibility, filenames, preview listing, or arbitrary retention classes.

export interface ReadImageSnapshotStore {
  snapshot(input: {
    readonly sessionId: string;
    readonly ownerId: string;
    readonly bytes: Uint8Array;
    readonly mimeType: string;
  }): Promise<SessionContextRef>;

  read(input: SessionContextRef): Promise<ContextOffloadReadResult>;
}

export interface ToolResultArchiveStore {
  archive(input: ToolResultArchiveRecorderInput): Promise<{
    readonly contextRefId: string;
  }>;

  readForReplay(input: ToolResultArchiveReaderInput): Promise<ToolResultArchiveReadResult>;

  readResource(input: ToolResultArchiveResourceReadInput): Promise<ToolResultArchiveReadResult>;
}

The Host derives both facades from the same authenticated, root-lease-bound ContextOffloadStore. The existing ToolResultArchiveCapability remains indivisible: writer, replay reader, ref reader, and ArchiveRead decoder are still wired together.

SQLite schema

PRAGMA journal_mode = WAL;
PRAGMA synchronous = FULL;
PRAGMA foreign_keys = ON;
-- Set before creating tables in a new database.
PRAGMA auto_vacuum = INCREMENTAL;

CREATE TABLE context_blobs (
  blob_id BLOB PRIMARY KEY CHECK(length(blob_id) = 32),
  payload BLOB NOT NULL,
  size_bytes INTEGER NOT NULL CHECK(size_bytes >= 0),
  created_at INTEGER NOT NULL CHECK(created_at >= 0)
);

CREATE TABLE context_refs (
  ref_id TEXT PRIMARY KEY,
  session_id TEXT NOT NULL,
  owner_kind TEXT NOT NULL CHECK(
    owner_kind IN ('read_image_snapshot', 'tool_result_archive')
  ),
  owner_id TEXT NOT NULL,
  blob_id BLOB NOT NULL REFERENCES context_blobs(blob_id) ON DELETE RESTRICT,
  media_type TEXT NOT NULL,
  created_at INTEGER NOT NULL CHECK(created_at >= 0),
  UNIQUE(session_id, owner_kind, owner_id)
);

CREATE INDEX context_refs_session
  ON context_refs(session_id, created_at, ref_id);

CREATE INDEX context_refs_blob
  ON context_refs(blob_id);

CREATE TABLE context_session_usage (
  session_id TEXT PRIMARY KEY,
  reference_count INTEGER NOT NULL CHECK(reference_count >= 0),
  logical_bytes INTEGER NOT NULL CHECK(logical_bytes >= 0)
);

CREATE TABLE context_store_usage (
  singleton INTEGER PRIMARY KEY CHECK(singleton = 1),
  blob_count INTEGER NOT NULL CHECK(blob_count >= 0),
  physical_bytes INTEGER NOT NULL CHECK(physical_bytes >= 0)
);

Usage counters update in the same transaction as blob/reference mutations. A Session quota counts logical referenced bytes; the workspace quota counts unique physical bytes, so Session copy cannot bypass logical quota and dedup does not double-count physical storage.

The per-kind hard limits remain 5 MiB for Read images and 4 MiB for Tool Result archives. Session/workspace quota values must be selected from retained-producer measurements and committed as explicit product constants before implementation is considered complete.

Write, read, copy, and delete semantics

Put

One BEGIN IMMEDIATE transaction:

  1. Validate the owner-specific byte limit and expected SHA-256.
  2. Resolve the stable ref identity and reject identity drift.
  3. Check Session logical and workspace physical quota counters.
  4. INSERT ... ON CONFLICT DO NOTHING for the content-addressed blob.
  5. Insert the typed reference.
  6. Update usage counters.
  7. Commit.

The store verifies size/hash on an existing blob identity before accepting the reference and fails closed on inconsistency.

Read

  1. Query by (session_id, ref_id); knowing a blob hash never grants access.
  2. Reject values beyond the caller and owner-kind cap before returning bytes.
  3. Materialize the complete capped BLOB.
  4. Verify byte length and SHA-256.
  5. Return the whole object.

ArchiveRead may continue slicing its model response, but storage does not advertise range-read semantics.

Session copy

Copy selected context_refs in one transaction and point them at the same immutable blob rows. Return an old-ref to new-ref map for RuntimeEvent/message rewriting. Physical byte count must not increase.

Release, retirement, and GC

Reference deletion is indexed and Session-scoped. It does not synchronously delete payload rows. Background GC deletes only blobs with no references, in bounded (maxBlobs, maxBytes) batches older than a watermark. Incremental vacuum and WAL checkpointing are maintenance work, never readiness work.

Replay-critical references have no independent TTL. Known failed writes perform best-effort releaseReference; a hard crash between context commit and RuntimeEvent commit may leave a safe orphan. The orphan counts against explicit quota and is removed on Session retirement. A later bounded reconciler may remove proven-unreferenced owner identities, but is not required for the first cutover and must never infer that old replay-critical data is disposable from age alone.

Failure and degraded behavior

Failure point Required behavior
Crash before context transaction commit No blob or reference becomes visible.
Retry same owner and same bytes Return the existing reference.
Retry same owner with different bytes Fail identity_conflict.
Context commit succeeds, RuntimeEvent commit fails Safe orphan; best-effort release, quota-bound until Session retirement.
RuntimeEvent references missing/corrupt context Scoped not_found/corrupt; never scan the payload set or silently substitute another Session.
Archive put fails Do not replace the original Tool Result with an archive placeholder.
Image snapshot put fails Return an explicit Read/storage failure; do not emit a dangling SessionContextRef.
Session mismatch Fail closed before returning bytes.
Quota/database full Typed failure; Host remains available and pruning stays inline where possible.
Context database cannot open Context capability is unavailable with diagnostics; unrelated Host readiness must not perform payload recovery scans.
GC/vacuum/checkpoint failure Report and retry out of band; do not poison Host Ready or future bounded maintenance.

Legacy transition

Do not rewrite immutable RuntimeEvents.

  1. New writes switch directly to SessionContextRef / Tool Result placeholder v2; no dual write.
  2. A post-Ready, versioned, resumable migration discovers durable legacy replay references from RuntimeEvents/messages, rather than migrating every record solely by ArtifactSource.
  3. For each referenced v1 Tool Result archive or Read image snapshot, copy and verify the payload in a bounded transaction. Preserve the legacy Artifact id as ref_id where possible.
  4. The compatibility reader resolves v1 archive placeholders and legacy snapshot session_file refs through the new context database, with legacy Artifact fallback only until that reference is migrated.
  5. Missing or malformed legacy payloads produce a recorded scoped degradation; they do not block unrelated Host readiness.
  6. Obsolete Write/Edit/Bash derivations are purged without migration.
  7. Reclaim old payloads in bounded post-Ready batches. Do not call the existing per-Session purge loop repeatedly.
  8. Remove the legacy context fallback immediately after migration evidence proves all load-bearing references are resolved. Other product-owned Artifact classes remain outside this issue.

The migration must expose progress and be restartable. Cleanup is never allowed to turn a one-time cutover into another O(M×N) startup path.

Implementation sequence

  1. Evidence and contract: measure count/total/p50/p95/p99/max for retained producers; freeze Session/workspace quotas and whole-object performance budgets.
  2. SQLite authority: add the dedicated owner, schema, atomic put/read/release, counters, bounded GC, backup, and corruption/error mapping.
  3. Typed consumers: route Read image snapshots and Tool Result archives through their facades; add SessionContextRef and placeholder v2 while retaining v1 decoding.
  4. Lifecycle: make conversation copy clone references only; make Session retirement release one Session's refs; run GC/checkpoint/incremental vacuum outside readiness.
  5. Legacy cutover: migrate durable referenced payloads post-Ready, preserve legacy ids, stop unused producers, and reclaim obsolete payloads in bounded batches.
  6. Retirement follow-through: remove legacy context paths from ArtifactStore; reassess perf(storage): artifact metadata rewrites the full record table on every mutation (O(M×N) during startup) #4037/perf(storage): batch multi-Session artifact purge during session retirement (O(M×N) guard scans) #4038 as fallback-only stopgaps and close fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027 only when a 10–12k legacy workspace no longer performs scale-dependent Artifact work before Ready.

Each slice must be independently revertible. No slice introduces long-lived dual writes or a second new physical authority.

Verification and acceptance invariants

Contract and correctness

  • Same owner/same bytes retry is idempotent; owner drift conflicts.
  • Blob and ref are never partially committed inside the context database.
  • Cross-Session reads fail even when the caller knows ref/blob identity.
  • Read image and archive hard byte caps are enforced before commit and before return.
  • Whole-object SHA-256 and byte length are verified.
  • Archive put failure leaves the original Tool Result unpruned.
  • Session copy creates new refs without copying payload bytes.
  • Session retirement touches only that Session's indexed refs.
  • GC cannot delete a referenced blob and obeys both batch limits.
  • Quota counters remain transactionally consistent across dedup, copy, release, retirement, and GC.

Crash and migration

  • Child-process crash tests cover before commit, after blob insertion, after ref insertion, after context commit/before RuntimeEvent commit, and during bounded GC.
  • Reopening relies on SQLite recovery; no payload enumeration is performed.
  • Legacy migration is idempotent and resumes from every batch boundary.
  • Unmigratable legacy content degrades explicitly without blocking unrelated Sessions.
  • No immutable RuntimeEvent is rewritten.

Performance

  • Benchmark monolithic SQLite BLOB put/get and peak RSS at retained-producer p50/p99/max on Linux, macOS, and Windows.
  • Use EXPLAIN QUERY PLAN evidence for read, copy, retire, quota, and GC queries.
  • A workspace with approximately 10–12k legacy records / 6 GB reaches Host Ready without enumerating, decoding, hashing, reconciling, vacuuming, or compacting the payload population.
  • Host Ready and unrelated Session retirement do not scale with total context blob/ref count.
  • WAL checkpoint, GC, legacy cleanup, and incremental vacuum execute only after Ready or in an explicitly surfaced bounded cutover step.

Alternatives considered

Patch current ArtifactStore only

Indexed delta writes (#4037) and batched purge (#4038) are valid stopgaps but preserve the cross-SQLite/filesystem authority and its mixed product taxonomy. They do not establish the context-offload boundary.

Filesystem immutable CAS + SQLite references

Valid for streaming/large objects, but reintroduces two durability domains, orphan policy, filesystem cleanup, and range integrity. Current retained consumers are capped whole-object values, so this complexity is not justified yet.

Chunked SQLite now

Provides bounded materialization for true paging, but requires chunk protocol, ordering, independently verifiable integrity, and more rows. Current consumers already hydrate at most 4–5 MiB whole objects. Defer until a new large-object contract requires it.

Put payloads in runtime.sqlite

Would permit wider single-database transactions but couples multi-gigabyte replay bytes to operational-state backup, schema migration, and ordinary runtime queries. A dedicated database gives the payload domain an independent capacity and maintenance boundary.

Relationships

AI use

Generative tooling made a substantive contribution to repository inspection, interface/schema design, failure analysis, migration planning, and this issue text. All claims and proposed contracts were reviewed against current main and the linked discussion.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions