This repository contains a review-driven document ingestion pipeline built around Temporal workflows, FastAPI, SQLite, MinIO, and Marqo. It is designed for teams that need to normalize heterogeneous files, extract structured text, review and correct outputs, generate chunks, and publish searchable records into a vector index.
The system is intentionally operational, not just algorithmic. Documents move through explicit stages, every major output can be persisted as an artifact, and the operator UI is designed to inspect and manage the pipeline rather than hide it.
At a high level, the pipeline supports:
- ingestion of document files, images, office documents, and spreadsheets
- normalization into a canonical processing form
- OCR and extraction
- optional translation for non-English content
- chunk generation
- manual review and correction of pages, translations, and chunks
- indexing of approved chunks into Marqo
- operational inspection of workflow, artifacts, audit history, and index state
The system is suitable for:
- knowledge base ingestion
- multilingual document processing
- regulated or review-heavy ingestion workflows
- search and retrieval pipelines that need provenance and operator controls
The platform is composed of six main services:
api- FastAPI application exposing ingestion, review, artifact, search, and admin endpoints
worker- Temporal worker running OCR, translation, chunking, ingestion, and state-update activities
temporal- workflow orchestration and retry engine
minio- object storage for original uploads, normalized files, and stage artifacts
marqo- vector and lexical search index for approved chunks
ui- React operator console for dashboard, document review, search workbench, settings, and audit
Supporting service:
lang-detect- lightweight language detection service used before translation
+------------------+
| Operator UI |
| React console |
+---------+--------+
|
v
+------------------+ +--------+---------+ +------------------+
| Source Documents | -----> | API / Control | <----> | Temporal |
| PDFs, images, | | Plane | | orchestration |
| office, sheets | | FastAPI | | and retries |
+--------+---------+ +---+----------+---+ +--------+---------+
| | | |
| | | |
v v v v
+--------+---------+ +--------+--+ +---+---------------+ +------------------+
| MinIO | | SQLite | | Worker | | Lang Detect |
| originals, | | canonical | | normalize, OCR, | | language hints |
| normalized files, | | metadata, | | translate, chunk, | | before |
| artifacts | | pages, | | ingest | | translation |
+------------------+ | chunks | +---+---------------+ +------------------+
+-----+------+ |
| |
+--------------+
|
v
+--------+---------+
| Marqo |
| search index |
| approved chunks |
+------------------+
The pipeline is stage-based. Each stage has a purpose, a persistent state transition, and a corresponding operator surface.
The document has been accepted into the system and assigned a workflow identifier.
Typical outputs:
- document row in SQLite
- original file reference
- initial job record
The source file is normalized if needed and passed through OCR or native structured extraction.
Typical behavior:
- PDFs and office/image inputs are normalized toward a document-processing form
- CSV and XLSX inputs can be parsed without OCR
- OCR output is produced page by page conceptually, then persisted into the document state
Operators inspect extracted page content and correct OCR mistakes before downstream processing continues.
This is where the system becomes review-driven instead of fully automatic.
Non-English content is translated into a target language for downstream chunking and search.
The current implementation keeps translation provider and model metadata so translation outputs remain attributable.
Operators review machine translation before chunking. This is important in multilingual or domain-heavy corpora where terminology needs supervision.
Reviewed page content is transformed into chunks suitable for search and retrieval.
Each chunk is expected to remain traceable to:
- document
- page start
- page end
- chunk order
- run configuration
Operators can inspect, edit, exclude, and eventually tag chunks before ingestion.
This is also the right stage for future chunk tagging and reindex-dirty tracking.
Final gate before indexing approved chunks.
Approved chunks are written into Marqo using a passage-style schema.
The document is fully processed and indexed.
The workflow encountered a non-recoverable failure or exceeded retry limits.
The system is organized around a few core entity types.
Top-level business objects representing an ingested source.
Discrete runs such as ingestion, OCR-only, translation-only, chunking, or reingestion operations.
Persisted files or exports associated with a document and stage, such as:
- original uploads
- normalized files
- OCR JSON exports
- translation JSON exports
- chunk exports
- Marqo payload exports
OCR output and page-level review state.
Chunked text, review state, exclusion state, and page-span lineage.
Document-level view of what has been pushed to Marqo.
SQLite is the canonical metadata and review-state store.
It owns:
- document rows
- page rows
- chunk rows
- jobs
- artifact metadata
- audit logs
- search settings
- document/index status
MinIO stores document and stage artifacts.
Typical artifact types include:
- original uploads
- normalized PDF or spreadsheet outputs
- OCR page exports
- translation exports
- chunk exports
- Marqo payload snapshots
Temporal is the orchestration layer.
It is responsible for:
- retries
- workflow lifecycle
- review gates
- long-running task resilience
It is not the canonical store for edited content.
Marqo is the search-facing index.
It should be treated as a downstream projection of approved chunk state, not as the source of truth for content editing.
The ingestion layer accepts these source types:
- PDF documents
- text-heavy image assets:
.png.jpg.jpeg.webp.tif.tiff
- images:
- scanned pages, posters, forms, and photo captures are treated as OCR candidates
- office documents:
.doc.docx.ppt.pptx.xls.xlsx
- delimited and spreadsheet data:
.csv.xlsx
Normalization behavior by class:
- document-centric inputs
- PDF, image, Word, and PowerPoint inputs are normalized toward a PDF-like document-processing form before OCR
- spreadsheet-centric inputs
- CSV, XLS, and XLSX inputs can remain structured tabular artifacts and may use native extraction instead of OCR when that produces better outputs
- multilingual inputs
- OCR output can continue into language detection and translation before chunking
In practice this means the pipeline can ingest:
- scanned reports
- born-digital PDFs
- presentation decks
- office handbooks and manuals
- spreadsheets and rate sheets
- image-only notices and circulars
General behavior:
- document-like inputs are normalized toward PDF processing
- spreadsheet inputs can remain spreadsheet-oriented and skip OCR when native parsing is better
pipeline/ FastAPI app, Temporal workflows, activities, models, database logic
ui/ React operator console
lang-detect/ Language detection microservice
scripts/ Operational and maintenance scripts
docs/ Supporting design and operational notes
tests/ Automated tests
test_data/ Small local fixtures for tests and smoke checks
docker-compose.yml
Dockerfile
requirements.txt
Default local ports from docker-compose.yml:
- UI:
3000 - API:
8001 - Marqo:
8882 - Temporal:
7233 - Temporal UI:
8080 - MinIO API:
9000 - MinIO console:
9001
- Docker and Docker Compose
- an OCR provider API key exposed as
MISTRAL_API_KEY - enough local disk for SQLite, MinIO artifacts, and Marqo state
Optional but recommended:
- GPU runtime support if using a GPU-backed Marqo image
docker compose up -d --builddocker compose downUseful endpoints after startup:
curl http://localhost:8001/health
curl http://localhost:8882/
curl http://localhost:9000/minio/health/liveImportant runtime variables include:
MISTRAL_API_KEYTEMPORAL_HOSTMARQO_URLMINIO_ENDPOINTMINIO_ACCESS_KEYMINIO_SECRET_KEYMINIO_BUCKETDOCUMENT_DB_PATHLANG_DETECT_URLTRANSLATION_PROVIDERTRANSLATION_MODELTRANSLATION_PAGE_CONCURRENCYTRANSLATION_MAX_RETRIESTRANSLATION_RETRY_BASE_SECONDSTEMPORAL_MAX_CONCURRENT_ACTIVITIESDOCUMENT_METADATA_CSV_PATHDOCUMENT_DESCRIPTIONS_JSONL_PATHCORS_ORIGINSALLOWED_FILE_PATHS
Optional metadata files:
DOCUMENT_METADATA_CSV_PATH- optional manifest-style metadata enrichment file
DOCUMENT_DESCRIPTIONS_JSONL_PATH- optional per-document descriptions enrichment file
If these files are not present, the pipeline still works; metadata enrichment is simply reduced.
The simplest production-style deployment pattern is:
- expose the UI at a public hostname
- expose the API either:
- behind the same domain under
/api, or - at a separate internal hostname behind a reverse proxy
- behind the same domain under
- keep Temporal, MinIO, and Marqo internal to the deployment network
Recommended routing shape:
https://your-ui-host/-> UIhttps://your-ui-host/api/-> API
This keeps browser calls same-origin and avoids hardcoded environment-specific domains in the frontend.
The UI is an operations console, not just an upload form.
Current views include:
- dashboard
- new document
- document operations
- search workbench
- settings
- audit log
The document operations screen is intended to expose:
- current stage
- stage runtime
- artifacts
- jobs
- pages
- translations
- chunks
- Marqo state
- audit history
The API is organized around a few major groups.
POST /documentsPOST /uploadPOST /documents/batch
GET /documentsGET /documents/summaryGET /documents/{workflow_id}GET /documents/{workflow_id}/runtimeGET /documents/{workflow_id}/jobsGET /documents/{workflow_id}/stage-io
GET /documents/{workflow_id}/pagesPATCH /documents/{workflow_id}/pages/{page_number}POST /documents/{workflow_id}/pages/{page_number}/reset
Translation data is surfaced through the page model and review endpoints.
GET /documents/{workflow_id}/chunksPATCH /documents/{workflow_id}/chunks/{chunk_number}POST /documents/{workflow_id}/chunks/{chunk_number}/reset
POST /documents/{workflow_id}/approve-ocrPOST /documents/{workflow_id}/approve-translationPOST /documents/{workflow_id}/approve-chunksPOST /documents/{workflow_id}/approve-ingestion
GET /documents/{workflow_id}/artifactsGET /documents/{workflow_id}/artifacts/{artifact_id}GET /documents/{workflow_id}/artifacts/{artifact_id}/content
GET /documents/{workflow_id}/marqoGET /documents/{workflow_id}/marqo/chunksPOST /documents/{workflow_id}/reingestPOST /marqo/searchGET /marqo/indexes/{index_name}/settingsGET /marqo/indexes/{index_name}/stats
- audit endpoints
- search runtime settings endpoints
The easiest way to inspect the complete surface is to run the API and open the generated OpenAPI docs at:
http://localhost:8001/docs
The search workbench and API support a configurable Marqo retrieval surface, including:
- hybrid, tensor, or lexical modes
- candidate pool sizing
- final result limits
- hybrid alpha
- RRF tuning
- optional query expansion profile
- optional E5 query prefixing
- rerank mode selection
- per-document result diversity
The default example index name in this showcase branch is:
documents-index
The scripts/ directory contains operational helpers for:
- inspecting Marqo fields
- counting indexed records
- resetting or creating indexes
- bulk reingesting SQLite chunks into Marqo
- listing failed workflows
- terminating stuck workflows
These are intended as operator tools, not hidden one-off commands.
Run the test suite with:
pytestUseful quick checks:
python3 -m py_compile pipeline/*.py
cd ui && npm run buildThis repository favors explicitness over silent automation:
- review stages are visible
- artifacts are first-class
- runtime state is inspectable
- downstream indexing is a separate concern from content ownership
That makes it a good fit for teams that need document traceability and operational control rather than a black-box ingestion flow.