Tracking Issue: support distributed multi-replica Server and background workers

[Teingi]

Feature description

Support production deployment of PowerContext as multiple Server replicas with separately coordinated Scheduler and background Worker roles.

The target is horizontal API/MCP availability plus safe distributed processing of Memory extraction, Experience incubation, and future connector, retention, or evaluation jobs against a shared durable backend.

Problem and proposed solution

The current Builtin Runtime is intentionally single-owner:

• APScheduler persists jobs in a local SQLite sidecar;
• a process-local registry maps scheduler activations to the live Runtime;
• in-process locks serialize scoped operations;
• the current Runtime RFC explicitly provides no cross-process claims, leases, leader election, distributed scheduling, or exactly-once guarantee;
• the host must not open concurrent Runtime owners for one database.

Starting multiple identical Server processes would therefore risk duplicate background execution, concurrent cursor advancement, repeated model calls, duplicate Candidates/Revisions, unsafe shutdown behavior, and MCP/API state inconsistencies. A load balancer alone does not make the current architecture multi-replica safe.

Track a design and implementation that separates three roles:

API role

• horizontally scalable HTTP, MCP, Dashboard, and Client request serving;
• no implicit scheduler ownership;
• no process-local session or lock may be required for correctness across requests;
• shared authentication, authorization, rate-limit, and version behavior across replicas.

Scheduler role

• discovers due work and creates durable work items;
• uses database-backed leader election, advisory/fenced ownership, or another reviewed coordination mechanism;
• duplicate scheduling attempts remain idempotent;
• losing leadership cannot permit a stale scheduler to keep issuing work.

Worker role

• atomically claims bounded work with lease owner, expiry, attempt, and fencing information;
• renews or releases leases and safely recovers abandoned work;
• provides at-least-once execution with domain-level idempotency/CAS rather than claiming exactly-once delivery;
• preserves required per-scope ordering while allowing independent scopes to run concurrently;
• records bounded status, retry/backoff, cancellation, terminal failure, and operator recovery.

The RFC/design phase should decide whether the shared database is sufficient for the first implementation or whether a replaceable queue/coordinator SPI is required. OceanBase should be the first multi-replica backend. SQLite remains a single-node profile and must reject or clearly report unsupported multi-replica configuration.

The work should also define:

• durable job identity and idempotency keys for every background job family;
• transaction boundaries among Source cursor advancement, Artifact/Candidate persistence, and work completion;
• manual flush versus scheduled/queued work coordination;
• graceful drain, shutdown, rolling upgrade, version skew, and schema migration behavior;
• API, Scheduler, and Worker liveness/readiness/degraded states;
• bounded metrics and traces for queue depth, claim latency, lease expiry, attempts, outcomes, and per-family throughput;
• authorization and secret boundaries for internal Worker/Scheduler calls;
• deployment examples and failure-injection acceptance.

Acceptance criteria

• Two API replicas behind round-robin load balancing serve HTTP and MCP requests with equivalent behavior and no sticky in-process session requirement.
• Two Workers can process independent scopes concurrently without producing duplicate immutable Revisions or Candidates.
• If a Worker exits before, during, or after its durable commit, lease recovery and idempotency produce one valid logical outcome and an auditable attempt history.
• Concurrent manual and scheduled processing of the same Source window cannot double-advance a cursor or double-commit derived content.
• Scheduler failover prevents a stale leader from continuing to enqueue work after ownership changes.
• Rolling restart and one-version compatibility-window tests preserve in-flight work or fail it visibly for safe retry.
• Readiness distinguishes API availability from Scheduler/Worker health and from generation, embedding, database, or authorization dependency failures.
• In-process locks and process-local registries are optimizations only; distributed correctness is enforced by durable coordination.
• SQLite starts only in the documented single-node profile, while OceanBase passes multi-replica correctness, recovery, and load tests.
• Logs, traces, work records, and errors contain bounded identifiers and outcomes, never Source/Memory content, prompts, model responses, credentials, or configured secret URLs.

Alternatives considered

• Run the existing all-in-one Server behind a load balancer: duplicates Scheduler ownership and relies on process-local coordination.
• Use a shared APScheduler job store alone: shared schedules do not define domain work claims, cursor transactions, fencing, retries, or idempotent commits.
• Promise exactly-once execution: not realistic across process failure and network/database ambiguity; at-least-once plus exact domain idempotency is the safer contract.
• Require Redis/Kafka in the first release: may be appropriate later, but a new mandatory dependency should follow evidence that database-backed coordination is insufficient.
• Allow multi-replica SQLite: contradicts the current local single-owner storage and scheduler model.

Additional context

Related: #1214 for observability, #1321 for bounded storage/latency, #1395 for authorization, #1423 for future connector jobs, and #1425 for lifecycle jobs. Historical #1056 covered a PowerMem-specific multi-worker MCP defect; this issue defines the current PowerContext distributed Server and background-processing contract instead.

• I am willing to contribute code, docs, or design feedback.

[thunguo]

I'd like to give this a try; I'll create a new proposal to align on the plan first.