Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
292 changes: 190 additions & 102 deletions src/content/docs/Immediate.Jobs/api-reference.md

Large diffs are not rendered by default.

57 changes: 28 additions & 29 deletions src/content/docs/Immediate.Jobs/batches-and-continuations.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Batches and continuations
description: Build atomic job graphs, chains, fan-out/fan-in and dynamically expanded workflows.
description: Create batches, continuations, parallel branches and workflows that add jobs while running.
order: 7
group: Guides
---
Expand All @@ -9,19 +9,17 @@ group: Guides
import { Callout } from '$lib/components/docs';
</script>

Batches persist jobs and their dependency edges atomically. They require a graph-capable
provider; Redis exposes queue and recurring capabilities only. Resolve the scoped
`IJobBatchScheduler` from DI, normally through constructor injection alongside the generated job
schedulers.
Batches save jobs and their dependencies in one operation. They require storage with graph
support, which Redis does not provide. Inject the scoped `IBatchScheduler` alongside the generated
job schedulers.

## Atomic workflow graph
## Create a batch

The constructor below makes every receiver explicit: `batches` is the runtime batch scheduler;
the other parameters are nested scheduler types generated for their corresponding job classes.
Inject `IBatchScheduler` and the generated scheduler for each job in the workflow:

```csharp
public sealed class ImportWorkflow(
IJobBatchScheduler batches,
IBatchScheduler batches,
ImportData.Scheduler import,
BuildIndex.Scheduler index,
NotifyOwner.Scheduler notify,
Expand Down Expand Up @@ -65,9 +63,9 @@ public sealed class ImportWorkflow(
}
```

Within an open batch, `AddToBatch` and `AddToBatchAt` only buffer records. Continuations built from
their handles remain in the same buffer. `CommitAsync` performs one atomic graph write and returns
a `BatchHandle`; nothing becomes visible before commit.
Within an open batch, `AddToBatch` and `AddToBatchAt` keep jobs in memory. Continuations created
from their handles stay in the same batch. `CommitAsync` saves the entire batch in one operation
and returns a `BatchHandle`. Nothing is visible before the commit.

`Begin()` returns the in-memory buffer shown above. Always dispose it: disposal without commit
abandons the buffer. A batch can commit only once and cannot be modified after commit. As an
Expand All @@ -81,16 +79,15 @@ BatchHandle handle = await workflow.StartAsync(importId, cancellationToken);
await batches.CancelAsync(handle, cancellationToken);
```

This includes scheduled, active and continuation-waiting members, and the aggregate batch becomes
`Cancelled` after its members settle. Cancelling an active member records cancellation durably but
does not forcibly stop handler code already running in process; stale worker completion is fenced
from changing the terminal result.
This includes scheduled, active and continuation-waiting jobs. The batch becomes `Cancelled` after
every job reaches a final state. Cancelling an active job saves the cancellation but does not
forcibly stop handler code that is already running. If that code finishes later, it cannot
overwrite the cancelled result.

Failures before `CommitAsync` begins write nothing. Once commit begins, however, the batch is
closed even when the call throws, and a transport failure can leave the durable outcome unknown:
storage may have committed the graph before the caller lost the response. Do not retry the same
`JobBatch`; an operation that rebuilds and commits another batch needs application-level
idempotency or duplicate tracking.
A failure before `CommitAsync` begins saves nothing. Once the commit begins, the batch closes even
if the call throws. If the storage connection fails during the commit, the caller may not know
whether the batch was saved. Do not reuse the same `Batch`. If you create another batch, guard
against running the work twice.

Batch members can carry the same fair-queue group IDs as ordinary scheduled work:

Expand All @@ -102,9 +99,9 @@ var grouped = import.AddToBatchInGroup(batch, new(importId), tenantId);
var groupedAt = import.AddToBatchAt(batch, new(importId), runAt, tenantId);
```

`AddToBatchInGroup` also accepts an optional delay. Whitespace group IDs are normalized to no
group, the 128-character limit still applies, and the configured provider must support fair
acquisition for the group to affect dispatch order.
`AddToBatchInGroup` also accepts an optional delay. A blank group ID means no group, and group IDs
cannot exceed 128 characters. The group changes scheduling order only when the storage provider
supports fair queues.

## Chains, fan-out and fan-in

Expand Down Expand Up @@ -163,8 +160,8 @@ work relates to the current job's existing continuations:
| `BesideContinuations` | Current batch | Unchanged; the new job forms a parallel branch. |
| `BeforeContinuations` (default) | Current batch | They also wait for the new job, creating an additive dependency. |

The `BeforeContinuations` splice keeps each existing dependency on the current job and adds a
dependency on the new job. Existing continuations therefore wait for both jobs.
With `BeforeContinuations`, each existing follow-up job waits for both the current job and the new
job.

<Callout type="warning">

Expand All @@ -174,7 +171,9 @@ except for detached scheduling, the current job must belong to a batch. `IJOB001

</Callout>

Monitor a graph through `IJobBatchMonitor.GetStatusAsync`, `QueryMembersAsync` and `GetGraphAsync`.
`BatchStatus` counts succeeded, failed, cancelled and skipped members separately; a batch can
Use the scoped `JobMonitor` to read a graph. Call `GetBatchAsync`, `QueryBatchMembersAsync`, or
`GetBatchGraphAsync`. These methods return `null` when storage does not support graphs.
`BatchStatus` counts succeeded, failed, cancelled and skipped members separately. A batch can
succeed when every executed member succeeded even if conditional branches were skipped. The
dashboard exposes the same progress and workflow states alongside batch cancel/delete operations.
concrete monitor also provides `CancelBatchAsync` for jobs that have not finished and
`DeleteBatchAsync` for a completed batch. The dashboard offers the same actions.
69 changes: 39 additions & 30 deletions src/content/docs/Immediate.Jobs/choosing-storage.md
Original file line number Diff line number Diff line change
@@ -1,53 +1,62 @@
---
title: Choosing storage
description: Choose an Immediate.Jobs topology and provider by durability, scale and capability.
description: Choose storage by durability, worker count and supported job features.
order: 10
group: Guides
---

Storage choice has two dimensions: the provider holds records; the topology decides whether memory
or that provider is authoritative.
Choose both a storage provider and a mode. The provider stores job data. The mode controls whether
workers coordinate through memory or through the provider.

| Topology | Authority | Processes | Durability | Use for |
| -------------- | --------------------------------------- | ----------: | ------------------------ | ----------------------------------------- |
| `InMemory` | Process memory | One | None | Unit tests, local demos, disposable work. |
| `SingleServer` | Memory with synchronous durable replica | Exactly one | Durable restart recovery | Low-latency single-instance services. |
| `Distributed` | Durable provider | One or more | Durable coordination | Scale-out and high availability. |
| Mode | Where jobs are coordinated | Worker processes | Survives restart | Use for |
| -------------- | -------------------------------- | ---------------- | ---------------- | ----------------------------------------- |
| `InMemory` | Current process | One | No | Unit tests, local demos, disposable work. |
| `SingleServer` | Memory backed by durable storage | Exactly one | Yes | Low-latency single-instance services. |
| `Distributed` | Storage provider | One or more | Yes | Scale-out and high availability. |

Calling a durable provider selects single-server mode unless you explicitly call
`UseDistributed()`. `UseRedis` always selects distributed mode. Never point two processes at the
same single-server replica: each believes its private memory is authoritative and drift detection
will fail.
A durable SQL provider uses single-server mode unless you call `UseDistributed()`. Redis always
uses distributed mode. Do not connect two scheduler processes to the same single-server storage;
the mode expects exactly one process and fails when it detects another.

## Capability matrix
## Supported features

| Provider | Queue | Recurring | Graph | Fair groups | Topologies |
| Provider | Queue | Recurring | Graph | Fair groups | Modes |
| ------------ | :---: | :-------: | :---: | :---------: | -------------------------- |
| In-memory | ✓ | ✓ | ✓ | ✓ | In-memory only |
| EF Core SQL | ✓ | ✓ | ✓ | ✓ | Single-server, distributed |
| LinqToDB SQL | ✓ | ✓ | ✓ | ✓ | Single-server, distributed |
| Redis | ✓ | ✓ | — | — | Distributed |

Queue capability includes ordinary scheduling, execution history and job monitoring. Recurring
adds durable schedule reconciliation/materialization. Graph adds atomic batches, dependencies,
continuations and batch monitoring. The dashboard hides or returns 404 for unsupported graph
views.
Queue support includes scheduling, execution history and job monitoring. Recurring support stores
schedules and creates runs when they are due. Graph support adds batches, dependencies,
continuations and batch monitoring. The dashboard hides graph views when storage does not support
them.

## Tradeoffs

- In-memory is fastest and deterministic, but a restart loses pending jobs and history.
- Single-server acquires from memory and writes every transition to a full-capability SQL replica.
Startup restores the durable snapshot. It cannot provide multi-process failover.
- Distributed SQL coordinates leases, recurring schedules, graph transitions and fair-group
cursors in the database and is the full-featured scale-out option.
- In-memory is fast and predictable in tests, but a restart loses pending jobs and history.
- Single-server selects work in memory and writes every change to SQL. It restores that state after
a restart but cannot fail over to another process.
- Distributed SQL coordinates workers through the database. It supports multiple processes and
all job features.
- Redis offers efficient distributed queues and recurring work, but not batches, continuations or
fair-group acquisition.
fair queues.

## A custom provider

Implement `IJobStorage` for queue capability. Add `IRecurringJobStorage` and/or `IJobGraphStorage`
only when their atomicity contracts are honored. Implement `IJobStorageReplica` as well to qualify
for single-server mode. Providers must initialize idempotently, claim due work atomically, enforce
worker ownership and leases, make recurring materialization unique, paginate monitoring, tolerate
repeated async disposal, and make graph commit/release/cascade transitions atomic. See the compact
contract map in [API reference](/docs/Immediate.Jobs/api-reference#custom-storage-contracts).
Implement `IJobStorage` to support queues. Add `IRecurringJobStorage`, `IJobGraphStorage`, and
`IFairQueueStorage` only for features the provider supports.

Single-server storage needs two extra interfaces for restart recovery. `IJobStorageReplica`
claims the exact job IDs selected by the in-memory queue. `IJobGraphStorageReplica` loads incoming
continuation links at startup. A provider must implement both interfaces, plus recurring and graph
support, to use single-server mode.

Starting or disposing the provider more than once must be safe. It must save each claim, recurring
run and graph change in one operation so workers cannot create duplicates or overwrite each other.
It must also enforce leases and worker ownership, and return monitoring results in pages.

Run the `JobStorageConformanceSuite` from `Immediate.Jobs.Testing` with the same service
registration an application would use. Select the tests that match the provider's features. See
[Testing jobs](/docs/Immediate.Jobs/testing-jobs#test-a-storage-provider) and the contract summary
in [API reference](/docs/Immediate.Jobs/api-reference#custom-storage-contracts).
94 changes: 62 additions & 32 deletions src/content/docs/Immediate.Jobs/configuring-storage-providers.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Configuring storage providers
description: Configure in-memory, EF Core, LinqToDB and Redis storage and own their schemas correctly.
description: Configure built-in storage providers and manage their database schemas.
order: 11
group: Guides
---
Expand All @@ -12,11 +12,12 @@ group: Guides
## In-memory

```csharp
builder.Services.AddMyAppJobs(options => options.UseInMemory());
builder.Services.AddMyAppJobs()
.ConfigureStorage(storage => storage.UseInMemory());
```

This is also the default when no storage is selected. It is non-durable and single-node but
implements recurring, graph and fair-queue behavior for development and tests.
Select in-memory storage explicitly. It keeps data in one process and loses it on restart, but it
supports every job feature. Use it for development and tests.

## Entity Framework Core

Expand All @@ -39,8 +40,10 @@ builder.Services.AddDbContextFactory<JobsDbContext>(db =>
// db.UseSqlite(jobsConnectionString); // SQLite
// db.UseSqlServer(jobsConnectionString); // SQL Server

builder.Services.AddMyAppJobs(options =>
options.UseEntityFrameworkCore<JobsDbContext>());
builder.Services.AddMyAppJobs()
.ConfigureStorage(storage => storage
.UseEntityFrameworkCore<JobsDbContext>()
.UseSingleServer());

public sealed class AppDbContext(DbContextOptions<AppDbContext> options) : DbContext(options)
{
Expand Down Expand Up @@ -90,54 +93,81 @@ dotnet add package Immediate.Jobs.LinqToDB --prerelease
```

```csharp
using LinqToDB;
using LinqToDB.Data;
using LinqToDB.Extensions.DependencyInjection;

var dataOptions = new DataOptions().UsePostgreSQL(connectionString);
// new DataOptions().UseSQLite(connectionString);
// new DataOptions().UseSqlServer(connectionString);

await dataOptions.CreateImmediateJobsSchemaAsync(
schema: "background", // must be null for SQLite
CancellationToken.None
);
builder.Services.AddLinqToDBContext<JobsDataConnection>(() => dataOptions);

await using (var connection = new JobsDataConnection(dataOptions))
{
await connection.CreateImmediateJobsSchemaAsync(
schema: "background", // must be null for SQLite
CancellationToken.None
);
}

builder.Services.AddMyAppJobs()
.ConfigureStorage(storage => storage
.UseLinqToDB<JobsDataConnection>(schema: "background")
.UseSingleServer());

builder.Services.AddMyAppJobs(options =>
options.UseLinqToDB(dataOptions, schema: "background"));
public sealed class JobsDataConnection(DataOptions options) : DataConnection(options);
```

The application owns `DataOptions`, the matching ADO.NET driver and schema lifecycle. The helper
supports SQLite (without a named schema), PostgreSQL and SQL Server and creates the tables and
indexes for a fresh database.
Register the `DataConnection` type with dependency injection. Jobs resolves it when storage work
starts. The application owns `DataOptions`, the matching ADO.NET driver and the database schema.
The helper supports SQLite (without a named schema), PostgreSQL and SQL Server. It creates the
tables and indexes for a new database.

## Redis

```bash
dotnet add package Immediate.Jobs.Redis --prerelease
```

Pass a configuration string when Jobs should own the connection:
Register an `IConnectionMultiplexer`, then select Redis storage:

```csharp
builder.Services.AddMyAppJobs(options => options.UseRedis(
"localhost:6379",
redis =>
{
redis.Database = 1;
redis.KeyPrefix = "billing-jobs";
}
));
using StackExchange.Redis;

builder.Services.AddSingleton<IConnectionMultiplexer>(_ =>
ConnectionMultiplexer.Connect("localhost:6379"));

builder.Services.AddMyAppJobs()
.ConfigureStorage(storage => storage
.UseRedis()
.ConfigureRedis(redis =>
{
redis.Database = 1;
redis.KeyPrefix = "billing-jobs";
}));
```

Or pass an application-owned `IConnectionMultiplexer`; the provider will not dispose it. The
configuration-string overload owns and disposes its connection. `Database` defaults to `-1`
(server default), and `KeyPrefix` defaults to `immediate-jobs`. Prefixes cannot contain braces
because the provider adds its own Redis Cluster hash tag for atomic Lua operations.
The provider uses the registered connection and does not dispose it. The dependency injection
container disposes the connection in this example because it creates the singleton. If you
register an existing instance, its owner must dispose it. `Database` defaults to `-1` (server
default), and `KeyPrefix` defaults to `immediate-jobs`. The prefix cannot contain `{` or `}` because
Jobs uses those characters internally. Jobs validates these options at startup.

`ConfigureRedis` also accepts an `OptionsBuilder<RedisJobStorageOptions>` action. Use it when you
need configuration binding.

Redis always selects distributed mode and supports queue plus recurring capabilities. It does not
support graph workflows or fair queues.

<Callout type="warning" title="Preview schema ownership">
Call `ConfigureStorage` exactly once. With EF Core or LinqToDB, choose `UseSingleServer()` for one
scheduler process or `UseDistributed()` for more than one. Jobs defaults to single-server mode
when neither is selected. Redis always uses distributed mode.

<Callout type="warning" title="Database setup during preview">

Storage initialization is idempotent provider startup, not schema creation. Keep every
Immediate.Jobs provider package at the same preview revision as the core package, and create test
databases from the current EF model or `CreateImmediateJobsSchemaAsync` helper.
Starting Jobs does not create or update a database schema. Keep every Immediate.Jobs provider
package at the same preview version as the core package. Create test databases from the current EF
model or with `CreateImmediateJobsSchemaAsync`.

</Callout>
Loading