You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue owns one cohesive domain only: Agent context offload. A model-visible value is too large to keep inline, durable replay bytes are stored elsewhere, Session/Event history retains an authorized reference, and a later replay can resolve the complete value.
Today the two load-bearing producers use the generic Artifact authority:
Read image snapshots, capped at 5 MiB;
Tool Result archives, rejected above 4 MiB.
Both consumers currently hydrate and verify the complete value. ArchiveRead applies offset/limit only after whole-object hydration; it is response pagination, not physical range I/O.
The current authority stores metadata in SQLite and payloads in the filesystem, then implements cross-domain publication, purge intent, stable-path recovery, global metadata reload/rewrite, and payload-tree inspection. On the motivating workspace, approximately 11.7k records / 6 GB are enough to block Host readiness for minutes because the access pattern is O(all records/files), and Session retirement can become O(M×N). That is a control-plane design failure, not a scale that should require a custom object store.
Decision
Use a dedicated SQLite database as the first physical substrate for capped, whole-object context offload.
This database is separate from runtime.sqlite so multi-gigabyte replay payloads do not inflate operational-state backup, migration, or ordinary query paths. Blob bytes and context references live in the same database and commit atomically.
The current storage contract remains capped whole-object. This issue does not add streaming writes or claim true range reads. node:sqlite does not expose sqlite3_blob_open/read, and SQL substr(BLOB, ...) limits the returned value but does not guarantee bounded SQLite-side materialization.
If a future requirement needs genuinely large paged archives, evaluate fixed-size SQLite chunk rows with independently verifiable chunk integrity first. Add an external filesystem/S3-like CAS only after measured large-object or streaming workloads prove that chunked SQLite is insufficient.
Goals
Move new Read image snapshots and Tool Result archives off ArtifactStore.
Make blob + typed reference creation one SQLite transaction.
Preserve whole-object integrity verification and Session authorization.
Make Session copy copy references, not payload bytes.
Make Session retirement proportional to references owned by that Session.
Enforce explicit per-blob, per-Session logical, and workspace physical quotas.
Keep payload enumeration, reconciliation, GC, checkpoint, vacuum, and compaction off Host Ready.
Migrate still-referenced legacy replay payloads without rewriting immutable RuntimeEvents.
StorageRef accepts this variant for new Read image snapshots. Persisted session_file values remain a legacy input during the bounded cutover.
Tool Result placeholders move to a new rewrite version with contextRefId. The decoder continues to accept v1 artifactId placeholders. Migration preserves ref_id = legacy artifactId where possible, so old immutable RuntimeEvents resolve through the new store without event rewriting.
The public interface remains asynchronous even if the first implementation uses DatabaseSync, preserving the option to move storage work to a dedicated worker without changing consumers.
ownerId is stable within a Session. A retry with the same (sessionId, owner.kind, ownerId) and identical bytes returns the existing reference. Different bytes for the same identity fail with identity_conflict.
Typed consumer facades
The physical store does not expose ArtifactSource, product visibility, filenames, preview listing, or arbitrary retention classes.
The Host derives both facades from the same authenticated, root-lease-bound ContextOffloadStore. The existing ToolResultArchiveCapability remains indivisible: writer, replay reader, ref reader, and ArchiveRead decoder are still wired together.
Usage counters update in the same transaction as blob/reference mutations. A Session quota counts logical referenced bytes; the workspace quota counts unique physical bytes, so Session copy cannot bypass logical quota and dedup does not double-count physical storage.
The per-kind hard limits remain 5 MiB for Read images and 4 MiB for Tool Result archives. Session/workspace quota values must be selected from retained-producer measurements and committed as explicit product constants before implementation is considered complete.
Write, read, copy, and delete semantics
Put
One BEGIN IMMEDIATE transaction:
Validate the owner-specific byte limit and expected SHA-256.
Resolve the stable ref identity and reject identity drift.
Check Session logical and workspace physical quota counters.
INSERT ... ON CONFLICT DO NOTHING for the content-addressed blob.
Insert the typed reference.
Update usage counters.
Commit.
The store verifies size/hash on an existing blob identity before accepting the reference and fails closed on inconsistency.
Read
Query by (session_id, ref_id); knowing a blob hash never grants access.
Reject values beyond the caller and owner-kind cap before returning bytes.
Materialize the complete capped BLOB.
Verify byte length and SHA-256.
Return the whole object.
ArchiveRead may continue slicing its model response, but storage does not advertise range-read semantics.
Session copy
Copy selected context_refs in one transaction and point them at the same immutable blob rows. Return an old-ref to new-ref map for RuntimeEvent/message rewriting. Physical byte count must not increase.
Release, retirement, and GC
Reference deletion is indexed and Session-scoped. It does not synchronously delete payload rows. Background GC deletes only blobs with no references, in bounded (maxBlobs, maxBytes) batches older than a watermark. Incremental vacuum and WAL checkpointing are maintenance work, never readiness work.
Replay-critical references have no independent TTL. Known failed writes perform best-effort releaseReference; a hard crash between context commit and RuntimeEvent commit may leave a safe orphan. The orphan counts against explicit quota and is removed on Session retirement. A later bounded reconciler may remove proven-unreferenced owner identities, but is not required for the first cutover and must never infer that old replay-critical data is disposable from age alone.
Safe orphan; best-effort release, quota-bound until Session retirement.
RuntimeEvent references missing/corrupt context
Scoped not_found/corrupt; never scan the payload set or silently substitute another Session.
Archive put fails
Do not replace the original Tool Result with an archive placeholder.
Image snapshot put fails
Return an explicit Read/storage failure; do not emit a dangling SessionContextRef.
Session mismatch
Fail closed before returning bytes.
Quota/database full
Typed failure; Host remains available and pruning stays inline where possible.
Context database cannot open
Context capability is unavailable with diagnostics; unrelated Host readiness must not perform payload recovery scans.
GC/vacuum/checkpoint failure
Report and retry out of band; do not poison Host Ready or future bounded maintenance.
Legacy transition
Do not rewrite immutable RuntimeEvents.
New writes switch directly to SessionContextRef / Tool Result placeholder v2; no dual write.
A post-Ready, versioned, resumable migration discovers durable legacy replay references from RuntimeEvents/messages, rather than migrating every record solely by ArtifactSource.
For each referenced v1 Tool Result archive or Read image snapshot, copy and verify the payload in a bounded transaction. Preserve the legacy Artifact id as ref_id where possible.
The compatibility reader resolves v1 archive placeholders and legacy snapshot session_file refs through the new context database, with legacy Artifact fallback only until that reference is migrated.
Missing or malformed legacy payloads produce a recorded scoped degradation; they do not block unrelated Host readiness.
Obsolete Write/Edit/Bash derivations are purged without migration.
Reclaim old payloads in bounded post-Ready batches. Do not call the existing per-Session purge loop repeatedly.
Remove the legacy context fallback immediately after migration evidence proves all load-bearing references are resolved. Other product-owned Artifact classes remain outside this issue.
The migration must expose progress and be restartable. Cleanup is never allowed to turn a one-time cutover into another O(M×N) startup path.
Implementation sequence
Evidence and contract: measure count/total/p50/p95/p99/max for retained producers; freeze Session/workspace quotas and whole-object performance budgets.
SQLite authority: add the dedicated owner, schema, atomic put/read/release, counters, bounded GC, backup, and corruption/error mapping.
Typed consumers: route Read image snapshots and Tool Result archives through their facades; add SessionContextRef and placeholder v2 while retaining v1 decoding.
Lifecycle: make conversation copy clone references only; make Session retirement release one Session's refs; run GC/checkpoint/incremental vacuum outside readiness.
Legacy cutover: migrate durable referenced payloads post-Ready, preserve legacy ids, stop unused producers, and reclaim obsolete payloads in bounded batches.
Each slice must be independently revertible. No slice introduces long-lived dual writes or a second new physical authority.
Verification and acceptance invariants
Contract and correctness
Same owner/same bytes retry is idempotent; owner drift conflicts.
Blob and ref are never partially committed inside the context database.
Cross-Session reads fail even when the caller knows ref/blob identity.
Read image and archive hard byte caps are enforced before commit and before return.
Whole-object SHA-256 and byte length are verified.
Archive put failure leaves the original Tool Result unpruned.
Session copy creates new refs without copying payload bytes.
Session retirement touches only that Session's indexed refs.
GC cannot delete a referenced blob and obeys both batch limits.
Quota counters remain transactionally consistent across dedup, copy, release, retirement, and GC.
Crash and migration
Child-process crash tests cover before commit, after blob insertion, after ref insertion, after context commit/before RuntimeEvent commit, and during bounded GC.
Reopening relies on SQLite recovery; no payload enumeration is performed.
Legacy migration is idempotent and resumes from every batch boundary.
Unmigratable legacy content degrades explicitly without blocking unrelated Sessions.
No immutable RuntimeEvent is rewritten.
Performance
Benchmark monolithic SQLite BLOB put/get and peak RSS at retained-producer p50/p99/max on Linux, macOS, and Windows.
Use EXPLAIN QUERY PLAN evidence for read, copy, retire, quota, and GC queries.
A workspace with approximately 10–12k legacy records / 6 GB reaches Host Ready without enumerating, decoding, hashing, reconciling, vacuuming, or compacting the payload population.
Host Ready and unrelated Session retirement do not scale with total context blob/ref count.
WAL checkpoint, GC, legacy cleanup, and incremental vacuum execute only after Ready or in an explicitly surfaced bounded cutover step.
Alternatives considered
Patch current ArtifactStore only
Indexed delta writes (#4037) and batched purge (#4038) are valid stopgaps but preserve the cross-SQLite/filesystem authority and its mixed product taxonomy. They do not establish the context-offload boundary.
Filesystem immutable CAS + SQLite references
Valid for streaming/large objects, but reintroduces two durability domains, orphan policy, filesystem cleanup, and range integrity. Current retained consumers are capped whole-object values, so this complexity is not justified yet.
Chunked SQLite now
Provides bounded materialization for true paging, but requires chunk protocol, ordering, independently verifiable integrity, and more rows. Current consumers already hydrate at most 4–5 MiB whole objects. Defer until a new large-object contract requires it.
Put payloads in runtime.sqlite
Would permit wider single-database transactions but couples multi-gigabyte replay bytes to operational-state backup, schema migration, and ordinary runtime queries. A dedicated database gives the payload domain an independent capacity and maintenance boundary.
Generative tooling made a substantive contribution to repository inspection, interface/schema design, failure analysis, migration planning, and this issue text. All claims and proposed contracts were reviewed against current main and the linked discussion.
Problem
Discussion #4030 separated three decisions that the current
ArtifactStoreconflates:This issue owns one cohesive domain only: Agent context offload. A model-visible value is too large to keep inline, durable replay bytes are stored elsewhere, Session/Event history retains an authorized reference, and a later replay can resolve the complete value.
Today the two load-bearing producers use the generic Artifact authority:
Both consumers currently hydrate and verify the complete value.
ArchiveReadapplies offset/limit only after whole-object hydration; it is response pagination, not physical range I/O.The current authority stores metadata in SQLite and payloads in the filesystem, then implements cross-domain publication, purge intent, stable-path recovery, global metadata reload/rewrite, and payload-tree inspection. On the motivating workspace, approximately 11.7k records / 6 GB are enough to block Host readiness for minutes because the access pattern is O(all records/files), and Session retirement can become O(M×N). That is a control-plane design failure, not a scale that should require a custom object store.
Decision
Use a dedicated SQLite database as the first physical substrate for capped, whole-object context offload.
This database is separate from
runtime.sqliteso multi-gigabyte replay payloads do not inflate operational-state backup, migration, or ordinary query paths. Blob bytes and context references live in the same database and commit atomically.The current storage contract remains capped whole-object. This issue does not add streaming writes or claim true range reads.
node:sqlitedoes not exposesqlite3_blob_open/read, and SQLsubstr(BLOB, ...)limits the returned value but does not guarantee bounded SQLite-side materialization.If a future requirement needs genuinely large paged archives, evaluate fixed-size SQLite chunk rows with independently verifiable chunk integrity first. Add an external filesystem/S3-like CAS only after measured large-object or streaming workloads prove that chunked SQLite is insufficient.
Goals
ArtifactStore.Non-goals
ArtifactSource.VACUUM, GC, or legacy cleanup before Host Ready.Durable contracts
Core reference
Add a durable reference that names logical Session-owned context, not a filesystem path:
StorageRefaccepts this variant for new Read image snapshots. Persistedsession_filevalues remain a legacy input during the bounded cutover.Tool Result placeholders move to a new rewrite version with
contextRefId. The decoder continues to accept v1artifactIdplaceholders. Migration preservesref_id = legacy artifactIdwhere possible, so old immutable RuntimeEvents resolve through the new store without event rewriting.Storage authority
The public interface remains asynchronous even if the first implementation uses
DatabaseSync, preserving the option to move storage work to a dedicated worker without changing consumers.ownerIdis stable within a Session. A retry with the same(sessionId, owner.kind, ownerId)and identical bytes returns the existing reference. Different bytes for the same identity fail withidentity_conflict.Typed consumer facades
The physical store does not expose
ArtifactSource, product visibility, filenames, preview listing, or arbitrary retention classes.The Host derives both facades from the same authenticated, root-lease-bound
ContextOffloadStore. The existingToolResultArchiveCapabilityremains indivisible: writer, replay reader, ref reader, andArchiveReaddecoder are still wired together.SQLite schema
Usage counters update in the same transaction as blob/reference mutations. A Session quota counts logical referenced bytes; the workspace quota counts unique physical bytes, so Session copy cannot bypass logical quota and dedup does not double-count physical storage.
The per-kind hard limits remain 5 MiB for Read images and 4 MiB for Tool Result archives. Session/workspace quota values must be selected from retained-producer measurements and committed as explicit product constants before implementation is considered complete.
Write, read, copy, and delete semantics
Put
One
BEGIN IMMEDIATEtransaction:INSERT ... ON CONFLICT DO NOTHINGfor the content-addressed blob.The store verifies size/hash on an existing blob identity before accepting the reference and fails closed on inconsistency.
Read
(session_id, ref_id); knowing a blob hash never grants access.ArchiveReadmay continue slicing its model response, but storage does not advertise range-read semantics.Session copy
Copy selected
context_refsin one transaction and point them at the same immutable blob rows. Return an old-ref to new-ref map for RuntimeEvent/message rewriting. Physical byte count must not increase.Release, retirement, and GC
Reference deletion is indexed and Session-scoped. It does not synchronously delete payload rows. Background GC deletes only blobs with no references, in bounded
(maxBlobs, maxBytes)batches older than a watermark. Incremental vacuum and WAL checkpointing are maintenance work, never readiness work.Replay-critical references have no independent TTL. Known failed writes perform best-effort
releaseReference; a hard crash between context commit and RuntimeEvent commit may leave a safe orphan. The orphan counts against explicit quota and is removed on Session retirement. A later bounded reconciler may remove proven-unreferenced owner identities, but is not required for the first cutover and must never infer that old replay-critical data is disposable from age alone.Failure and degraded behavior
identity_conflict.not_found/corrupt; never scan the payload set or silently substitute another Session.SessionContextRef.Legacy transition
Do not rewrite immutable RuntimeEvents.
SessionContextRef/ Tool Result placeholder v2; no dual write.ArtifactSource.ref_idwhere possible.session_filerefs through the new context database, with legacy Artifact fallback only until that reference is migrated.The migration must expose progress and be restartable. Cleanup is never allowed to turn a one-time cutover into another O(M×N) startup path.
Implementation sequence
SessionContextRefand placeholder v2 while retaining v1 decoding.ArtifactStore; reassess perf(storage): artifact metadata rewrites the full record table on every mutation (O(M×N) during startup) #4037/perf(storage): batch multi-Session artifact purge during session retirement (O(M×N) guard scans) #4038 as fallback-only stopgaps and close fix(runtime-host): cold-start artifact recovery sweeps O(all files) realpath/lstat, blocking Host readiness for minutes #4027 only when a 10–12k legacy workspace no longer performs scale-dependent Artifact work before Ready.Each slice must be independently revertible. No slice introduces long-lived dual writes or a second new physical authority.
Verification and acceptance invariants
Contract and correctness
Crash and migration
Performance
EXPLAIN QUERY PLANevidence for read, copy, retire, quota, and GC queries.Alternatives considered
Patch current
ArtifactStoreonlyIndexed delta writes (#4037) and batched purge (#4038) are valid stopgaps but preserve the cross-SQLite/filesystem authority and its mixed product taxonomy. They do not establish the context-offload boundary.
Filesystem immutable CAS + SQLite references
Valid for streaming/large objects, but reintroduces two durability domains, orphan policy, filesystem cleanup, and range integrity. Current retained consumers are capped whole-object values, so this complexity is not justified yet.
Chunked SQLite now
Provides bounded materialization for true paging, but requires chunk protocol, ordering, independently verifiable integrity, and more rows. Current consumers already hydrate at most 4–5 MiB whole objects. Defer until a new large-object contract requires it.
Put payloads in
runtime.sqliteWould permit wider single-database transactions but couples multi-gigabyte replay bytes to operational-state backup, schema migration, and ordinary runtime queries. A dedicated database gives the payload domain an independent capacity and maintenance boundary.
Relationships
AI use
Generative tooling made a substantive contribution to repository inspection, interface/schema design, failure analysis, migration planning, and this issue text. All claims and proposed contracts were reviewed against current
mainand the linked discussion.