findex

Forward-driven Index — A memory index engine where inverted indices are driven by forward index updates.

Design Philosophy

The core principle of findex: inverted index updates are derived from forward index updates.

In traditional search/ads/recommendation systems, forward index and inverted index are often built and updated independently, or use a lambda architecture that separates real-time and offline into different segments. This leads to poor data consistency, complex pipelines, and high latency.

findex takes a different approach:

• Forward index is the source of truth: all data updates land on the forward index first
• Inverted index is derived from forward index: forward index changes automatically trigger incremental inverted index updates
• Build and serve in one process: no offline-build + online-load two-phase workflow

This means:

1. Write a record → forward index updated → inverted index updated in chain (single pipeline, naturally consistent)
2. No segment merging, no offline builds, no loading latency
3. Index is queryable immediately after write

Architecture

┌─────────────────────────────────────────────────────┐
│              Search Layer                             │
│                SeekPlan                               │
│       SQL-like query interface                        │
├─────────────────────────────────────────────────────┤
│              Storage Layer                            │
│                                                      │
│   ┌─ Table ─────────────────────────────────────┐   │
│   │  DataTable (forward index, source of truth)  │   │
│   │  IndexTable (inverted index, driven by fwd)  │   │
│   │  MultiValueTable (multi-value index)         │   │
│   ├─────────────────────────────────────────────┤   │
│   │  Storage (infrastructure)                     │   │
│   │  HashKV │ Buffer/Schema │ Mempool            │   │
│   │  Hashmap │ Klist │ Logger │ Util             │   │
│   └─────────────────────────────────────────────┘   │
│                                                      │
├─────────────────────────────────────────────────────┤
│              Data Layer                              │
│                Loader                                │
│     Load data from disk files / message queues       │
│     Supports full load (base) and incremental (inc)  │
└─────────────────────────────────────────────────────┘

Data Flow

Write path (forward drives inverted):

1. Loader receives external data (full files / incremental messages)
2. Write to DataTable (forward index), Mempool allocates memory
3. Forward index change triggers IndexTable update (insert/delete in sorted inverted list)
4. Sync update MultiValueTable (if multi-value fields exist)
5. Write complete, index is immediately queryable

Query path:

1. External request with SQL-like query
2. SeekPlan parses and orchestrates query plan
3. Lookup IndexTable (inverted index) to get candidate set
4. Conditioner filters, Joiner joins DataTable (forward index) to fill fields
5. Selector projects required fields, return results

Tech Stack

Dimension	Choice	Rationale
Language	C++17	filesystem, structured bindings, if constexpr
Build	Bazel	precise deps, reproducible builds, remote cache
Logging	spdlog	high-perf, header-only optional, flexible format
Serialization	Protobuf	compatible with upstream/downstream systems
Error handling	Return values	C++ exceptions disabled, use return values
Memory	Custom Mempool	deferred reclamation, stats, deterministic latency

Code Style

• Header guard: #pragma once
• Namespaces: findex::container, findex::hashkv, etc.
• Class names: PascalCase (HashkvFile, TimeMempool)
• Member variables: snake_case_ with trailing underscore
• Pointer prefix: p_ (e.g. p_mempool_, p_head_)
• Constants: kCamelCase (e.g. kVaddrNull, kMaxReaderCnt)
• Functions: PascalCase (GetBucketSize(), MutableMempool())
• Formatting: Google style, 120 columns, * left-aligned

Build

# Build all
./build.sh

# Debug mode
./build.sh -d

# Clean
./clean.sh

Progress

Storage Layer - Infrastructure

• Hashmap / Klist / Mempool (in-memory data structures and memory pool)
• Logger (logging system)
• Util (time utilities)
• HashKV file layer: HashkvFile / HashkvMline / HashkvUtil
• HashKV full engine: Block / Shard / GC / Engine
• Buffer & Schema: columnar encoding system

Storage Layer - Table

• DataTable (forward index table)
• IndexTable (inverted index table, driven by forward updates)
• MultiValueTable (multi-value index table)
• Scorer (scoring system)

Data Layer - Loader

• Full load (disk files)
• Incremental update (message queue)

Search Layer - SeekPlan

• Query parsing and orchestration
• Query operators: Conditioner / Joiner / Selector
• Batch queries and rate limiting