Summary
On Redis 8.8 and later, the RediSearch worker pool defaults to a multithreaded background executor (search-workers defaults to the core count rather than 0). Under concurrent write and query load, FT.SEARCH briefly loses read-your-writes consistency: documents that have already been written to the keyspace are not yet visible to a query issued immediately afterwards.
This is server behavior rather than a redisvl defect, but it affects a common redisvl usage pattern, so it is worth documenting.
index.load(records)
results = index.query(query) # may not include everything just loaded
Evidence
Harness: 8 threads, each creating an index, loading 3 documents, and immediately querying with FilterQuery(filter_expression="*", sort_by="title") expecting all 3. 60 iterations per thread, 480 total, against redis:latest (Redis 8.10.0).
| configuration |
shortfalls |
documents seen at first query |
search-workers 14 (image default), 1 CPU |
17/480 |
0, 1, or 2 instead of 3 |
search-workers 14 (image default), unconstrained CPU |
17/480 |
0, 1, or 2 instead of 3 |
search-workers 0 |
0/480 |
always 3 |
A single-threaded probe against an idle server shows no shortfall, so concurrency is required to surface this.
The writes themselves are never lost. Across 30 shortfall events, all 3 hashes were already present in the keyspace at the moment the query returned fewer, and the index caught up every time:
docs present in keyspace during every shortfall: True (3 of 3 expected)
index caught up in all cases: 30/30
convergence delay ms: min=0.8 median=1.6 max=2.6
So the window is short, roughly 1 to 3 ms in these runs, but it is real and it scales with load.
Why this matters for redisvl users
Any code that writes and then immediately reads through the index can observe incomplete results:
load() followed directly by query()- ingestion pipelines that write and verify in the same request
SemanticCache and MessageHistory style flows that store an entry and then immediately search for it
The failure mode is quiet: a query returns fewer rows rather than raising, so it can look like a relevance or filtering problem.
This is also the confirmed cause of a family of nondeterministic CI failures in this repository on the redis:latest matrix jobs (see #658 and #659), where tests loading a handful of documents and querying immediately saw 0, 1, or 2 of them.
Suggestions
- Document the behavior for users on Redis 8.8+, ideally near
load() in the user guide and API docs, including that it applies only under concurrent load and that the window is short.
- Note the server-side option for workloads that need synchronous indexing: starting Redis with
--search-workers 0 restores foreground indexing and eliminates the window, at the cost of the multithreaded indexing benefits.
- Consider whether any first-class support is warranted, for example an opt-in helper that waits for an expected document count or polls until an index reflects a just-completed load. This should be weighed carefully: polling has obvious downsides, and it may be better to simply document the behavior and leave the decision to callers.
Reproduction scripts used for the numbers above can be attached if useful.
Summary
On Redis 8.8 and later, the RediSearch worker pool defaults to a multithreaded background executor (
search-workersdefaults to the core count rather than 0). Under concurrent write and query load,FT.SEARCHbriefly loses read-your-writes consistency: documents that have already been written to the keyspace are not yet visible to a query issued immediately afterwards.This is server behavior rather than a
redisvldefect, but it affects a commonredisvlusage pattern, so it is worth documenting.Evidence
Harness: 8 threads, each creating an index, loading 3 documents, and immediately querying with
FilterQuery(filter_expression="*", sort_by="title")expecting all 3. 60 iterations per thread, 480 total, againstredis:latest(Redis 8.10.0).search-workers 14(image default), 1 CPUsearch-workers 14(image default), unconstrained CPUsearch-workers 0A single-threaded probe against an idle server shows no shortfall, so concurrency is required to surface this.
The writes themselves are never lost. Across 30 shortfall events, all 3 hashes were already present in the keyspace at the moment the query returned fewer, and the index caught up every time:
So the window is short, roughly 1 to 3 ms in these runs, but it is real and it scales with load.
Why this matters for redisvl users
Any code that writes and then immediately reads through the index can observe incomplete results:
load()followed directly byquery()SemanticCacheandMessageHistorystyle flows that store an entry and then immediately search for itThe failure mode is quiet: a query returns fewer rows rather than raising, so it can look like a relevance or filtering problem.
This is also the confirmed cause of a family of nondeterministic CI failures in this repository on the
redis:latestmatrix jobs (see #658 and #659), where tests loading a handful of documents and querying immediately saw 0, 1, or 2 of them.Suggestions
load()in the user guide and API docs, including that it applies only under concurrent load and that the window is short.--search-workers 0restores foreground indexing and eliminates the window, at the cost of the multithreaded indexing benefits.Reproduction scripts used for the numbers above can be attached if useful.