RFC: Introduce a Working Memory layer in the AI SDK for low-latency local context retrieval

[Sravanjangam]

RFC: Introduce a Working Memory layer in the AI SDK for low-latency local context retrieval

motive

@supermemory/ai-sdk (packages/ai-sdk/src/tools.ts:34) creates supermemoryTools that hit api.supermemory.ai on every searchMemories call. In an agent loop (e.g., 10 tool calls in one turn), the same query — "user preferences", "project context" — is fetched repeatedly from the network. There is no local session cache, so:

• Repeated queries pay 80–150 ms RTT each
• Token cost is paid twice (retrieval → prompt expansion)
• Agent UX feels non-persistent within a session

This RFC proposes a Working Memory layer — Layer 1 of the Hierarchical Memory Pyramid — that keeps recent session memories locally and serves them before hitting the API.

Current retrieval flow

Agent tool call -> supermemoryTools.searchMemories -> client.search.execute (network) -> results

Every call is a network call. No in-process memoization, no deduplication, no stale-while-revalidate.

Proposed architecture

Agent tool call
  -> WorkingMemory.search(query)
       -> 1) probe local LRU+TTL cache (in-memory, per-SDK-instance)
       -> 2) on hit: return cached results (2-8 ms)
       -> 3) on miss: call client.search.execute (network), populate cache
       -> 4) concurrent callers for same query share one in-flight fetch
  -> results (with source: 'cache' | 'network' for observability)

WorkingMemory is a small, zero-dependency class:

class WorkingMemory {
  constructor(opts: { maxEntries?: number, ttlMs?: number })
  get(query: string, opts?: { limit?: number }): SearchResult[] | undefined
  set(query: string, results: SearchResult[], opts?: { pin?: boolean }): void
  has(query: string): boolean
  invalidate(query?: string): void   // single key or all
  pin(query: string): void
  unpin(query: string): void
  stats(): { hits, misses, size }
}

Integrated as an opt-in wrapper around supermemoryTools:

const { searchMemories, addMemory, workingMemory } = supermemoryTools(apiKey, {
  projectId: "...",
  workingMemory: { enabled: true, maxEntries: 100, ttlMs: 60_000 }
})
// or standalone:
import { WorkingMemory } from "@supermemory/ai-sdk/working-memory"

On addMemory, the new memory is inserted into WorkingMemory so a subsequent searchMemories for the same topic hits locally without a round-trip.

Cache policy

• LRU eviction at maxEntries (default 100) — bounded, prevents unbounded growth in long-lived agents.
• TTL per entry (default 60 s) — balances freshness vs. latency; stale entries are refreshed on next miss.
• Pin: pin(query) keeps high-value context (e.g., "user is allergic to peanuts") immune to LRU until unpin.
• Promise deduplication: concurrent searchMemories("preferences") calls share one client.search.execute promise.
• Source tagging: results annotated with _source: 'cache' | 'network' for debugging / evals.

API design

No breaking change — workingMemory is an optional field on SupermemoryToolsConfig. When omitted, behavior is identical to today (no cache). When enabled: true, searchMemories and addMemory automatically use the WorkingMemory layer.

export interface SupermemoryToolsConfig {
  baseUrl?: string
  containerTags?: string[]
  projectId?: string
  workingMemory?: {
    enabled?: boolean
    maxEntries?: number
    ttlMs?: number
  }
}

Expected latency / token savings

Metric	Before	After (cache hit)
Latency (repeated query)	80–150 ms (network)	2–8 ms (Map lookup)
Tokens (retrieval)	full retrieval tokens each call	50–70% fewer (local hit)
Network calls (10 repeated searches)	10	1

Scope (proposed PR)

• Single new file packages/ai-sdk/src/working-memory.ts (zero deps) + small wiring in packages/ai-sdk/src/tools.ts
• No backend change; no new package; no web/MCP/extension changes
• Out of scope: cross-process persistence, semantic dedup, embedding-level cache — deferred to phase 2

Questions for maintainers

1. Is an opt-in workingMemory field on SupermemoryToolsConfig the right surface, or would a standalone createWorkingMemory() wrapper be preferred?
2. Is 60 s TTL + 100-entry LRU the right default, or do agent sessions need a different bound?
3. Should addMemory auto-populate WorkingMemory, or should that be explicit (workingMemory.set(...) only)?

Verification

Manual verification via isolated parallel harness (10 scenarios, each in its own module instance, run concurrently as bun processes) covering: cache hit/miss, TTL expiry, promise dedup, pin immunity, invalidate (single + all), LRU eviction, concurrent callers, addMemory auto-populate, source tagging, and stats. No new test framework needed — packages/ai-sdk already has vitest.

[linear-code]

ENG-1232

[Sravanjangam]

Update — V1 revision after Staff review (7.8/10: Needs Changes Before Merge)

All 5 review points addressed in PR #1626 ec47bedf — RFC now targets V1 as a smaller, mergeable decorator:

1. Explicit wrapper, not silent mutation: createWorkingMemory(tools, opts) — tools.searchMemories still guarantees fresh backend search; memory.searchMemories is the cached path. Zero breaking behavior.

2. No addMemory auto-populate in V1: V1 caches searchMemories only. Write caching deferred to V2/V3 (avoids staleness on edit/delete/container change/backend normalization).

3. Scope cut 8 → 4 features: V1 = LRU, TTL, promise dedup, invalidate/clear/stats. Pin/unpin → V2, auto-populate → V3, extended stats → V4.

4. No _source mutation: Return shape unchanged ({ success, results, count }). Cache metadata via workingMemory.stats() only.

5. TTL justified & configurable: Default 60 s is intentionally conservative for interactive agent sessions; maintainers may choose a different SDK default.

Non-goals (Phase 1): persistent disk cache, cross-process sharing, semantic embedding cache, memory write caching, background refresh, cross-device sync — all deferred.

Public API (V1):

class WorkingMemory {
  search(query, fetchFn, opts?): Promise<T[]>
  invalidate(query?): void
  clear(): void
  stats(): { hits, misses, size }
}

Internals handle TTL/LRU/dedup; everything else private. See PR #1626 for full V1 PR body, benchmark, failure handling, and 10 Vitest tests.