feat(ingestion): agentic per-document strategy selection

[Naseem77]

What

Adds agentic ingestion strategy selection to GraphRAG.ingest(...), mirroring the retrieval-side PathRouter. When auto=True, a planner inspects a sample of each document and picks the chunker, entity-extraction backend, and resolver per document — filling in only the strategies the caller did not pass explicitly.

Design

• Explicit chunker / extractor / resolver always win.
• On an empty/invalid plan or any planner error, it falls back to today's defaults (sentence / gliner / exact), so behavior is never silently lost.
• Two planners:
  • LLMIngestionPlanner (default) — one small guided LLM call.
  • HeuristicIngestionPlanner — zero-cost, feature-based.
• The planner may also tune parameters inside each strategy; all values are coerced and clamped to safe ranges (PARAM_SPECS), so the model can never produce an unsafe config.

Usage

# LLM planner picks strategies per document
await rag.ingest(source="doc.md", auto=True)

# zero-cost heuristic instead
from graphrag_sdk import HeuristicIngestionPlanner
await rag.ingest(source="doc.md", auto=True, planner=HeuristicIngestionPlanner())

# explicit strategy still wins; planner fills the rest
await rag.ingest(source="doc.md", auto=True, resolver=ExactMatchResolution())

Commits

1. IngestionPlan schema + param clamping
2. Strategy builders
3. Heuristic + LLM planners
4. Package exports
5. auto=/planner= wiring in GraphRAG.ingest
6. Tests: schema, parsing, tunable params
7. Tests: planners, builders, ingest(auto=) merge

Tests

38 passed — tests/test_ingestion_planner.py. Ruff clean on all changed files.

Summary by CodeRabbit

• New Features

  • Added automatic, per-document ingestion strategy selection during ingestion.
  • Exposed new ingestion-planning options at the top-level SDK for easier access.
  • Added support for choosing chunking, extraction, and resolution strategies based on document content or planner output.

• Bug Fixes

  • Improved fallback behavior so ingestion continues with default strategies when planning cannot be completed.
  • Added validation and safe parameter clamping for supported ingestion strategy settings.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: abec88b8-077a-4a85-83eb-0e8196ae3c34

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR introduces an agentic ingestion strategy planner module with heuristic and LLM-based implementations for selecting chunker, extractor, and resolver strategies. It wires this into GraphRAG.ingest() via new auto/planner parameters, adds public exports, and includes tests.

Changes

Agentic Ingestion Strategy Planner

Layer / File(s)	Summary
Plan data model, parsing, and clamping graphrag_sdk/src/graphrag_sdk/ingestion/ingestion_planner.py	Defines chunker/extractor/resolver option sets and defaults, PARAM_SPECS/clamp_params() for range-safe parameter coercion, the IngestionPlan dataclass with validation, default_plan()/parse_plan() for JSON/key-value parsing, and the LLM prompt guide.
Heuristic and LLM planners graphrag_sdk/src/graphrag_sdk/ingestion/ingestion_planner.py	Implements HeuristicIngestionPlanner (feature-based structural/sentence chunker choice) and LLMIngestionPlanner (prompt-driven plan generation with budget enforcement and fallback to defaults).
Strategy builder factories graphrag_sdk/src/graphrag_sdk/ingestion/ingestion_planner.py	Adds build_chunker, build_extractor, build_resolver, and build_ingestion_strategies to instantiate concrete pipeline components from an IngestionPlan.
GraphRAG.ingest() integration graphrag_sdk/src/graphrag_sdk/api/main.py	Adds auto/planner keyword-only parameters across ingest() overloads, forwards them through _ingest_batch/_ingest_single, and adds _plan_ingestion_strategies() to select strategies with error fallback.
Public exports graphrag_sdk/src/graphrag_sdk/__init__.py, graphrag_sdk/src/graphrag_sdk/ingestion/__init__.py	Re-exports the new planner classes and build_ingestion_strategies from package facades.
Tests graphrag_sdk/tests/test_ingestion_planner.py	Adds tests for plan defaults, parsing, heuristic/LLM planning, builder factories, _plan_ingestion_strategies merge behavior, and parameter clamping.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
    participant Caller
    participant GraphRAG as GraphRAG.ingest()
    participant PlanFn as _plan_ingestion_strategies()
    participant Planner as LLMIngestionPlanner
    participant LLM
    participant Builder as build_ingestion_strategies()

    Caller->>GraphRAG: ingest(source, auto=True)
    GraphRAG->>PlanFn: request missing strategies
    PlanFn->>Planner: plan(text, source, ctx)
    Planner->>LLM: ainvoke(prompt with _GUIDE + sample)
    LLM-->>Planner: response text
    Planner->>Planner: parse_plan(response)
    alt parse succeeds
        Planner-->>PlanFn: IngestionPlan
    else parse fails or error
        Planner-->>PlanFn: default_plan()
    end
    PlanFn->>Builder: build_ingestion_strategies(plan, llm, embedder, entity_types)
    Builder-->>PlanFn: (chunker, extractor, resolver)
    PlanFn-->>GraphRAG: strategies (explicit values preserved)
    GraphRAG->>GraphRAG: run ingestion pipeline

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: agentic per-document ingestion strategy selection.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/agentic-ingestion-planner

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

graphrag_sdk/tests/test_ingestion_planner.py (1)

358-361: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Avoid asserting on a private attribute.

ext.entity_extractor._threshold reaches into a protected member. If the internal implementation renames it, the test breaks without a public-contract change. Consider asserting via a public accessor/property if one exists, or exposing one for testability.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@graphrag_sdk/tests/test_ingestion_planner.py` around lines 358 - 361, The
test is asserting against the protected _threshold member on the object returned
by build_extractor, which ties the test to internal implementation details.
Update test_build_extractor_applies_threshold to verify the threshold through a
public API on the extractor if one exists, or add a public accessor/property on
the extractor type used by build_extractor so the test can assert behavior
without touching private state.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@graphrag_sdk/src/graphrag_sdk/api/main.py`:
- Around line 1576-1590: The auto-ingestion planner in `main.py` is only
receiving `source` when `text` is `None`, so it cannot plan from the actual
loaded file content. Update the `auto` path in the ingestion method that calls
`_plan_ingestion_strategies` to load or sample the file content before planning,
and pass that content to the planner alongside `source` so it can inspect
document structure instead of relying on the path alone.
- Around line 1313-1314: The sync wrapper is missing the new `auto` and
`planner` keywords that were added to the async ingest API, so `ingest_sync()`
currently drops them and raises `TypeError` for callers using the new feature.
Update `ingest_sync()` in `main.py` to accept `auto` and `planner` alongside the
existing ingest parameters, and forward both values through to the underlying
async ingest call used by `ingest_sync()` so the sync and async signatures stay
aligned.
- Around line 1795-1813: The fallback handling in the ingestion planner path is
incomplete: only active.plan() is protected, so invalid planner results or
failures inside build_ingestion_strategies still escape instead of using
defaults. Update the planner/fallback logic in the ingestion strategy method
around active.plan() and build_ingestion_strategies so any None/invalid plan or
construction exception is caught and returns the default chunker, extractor, and
resolver just like the existing planner failure branch.

---

Nitpick comments:
In `@graphrag_sdk/tests/test_ingestion_planner.py`:
- Around line 358-361: The test is asserting against the protected _threshold
member on the object returned by build_extractor, which ties the test to
internal implementation details. Update test_build_extractor_applies_threshold
to verify the threshold through a public API on the extractor if one exists, or
add a public accessor/property on the extractor type used by build_extractor so
the test can assert behavior without touching private state.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: cffcdfab-f699-41a6-b0bb-52dec86d5310

📥 Commits

Reviewing files that changed from the base of the PR and between 0ab92ba and 42313f1.

📒 Files selected for processing (5)

• graphrag_sdk/src/graphrag_sdk/__init__.py
• graphrag_sdk/src/graphrag_sdk/api/main.py
• graphrag_sdk/src/graphrag_sdk/ingestion/__init__.py
• graphrag_sdk/src/graphrag_sdk/ingestion/ingestion_planner.py
• graphrag_sdk/tests/test_ingestion_planner.py

[Copilot]

Pull request overview

Adds opt-in, per-document “agentic” ingestion strategy selection to GraphRAG.ingest(...) via a new ingestion planner module (LLM-backed or heuristic), with safe defaults and parameter clamping.

Changes:

• Introduces IngestionPlan, param clamping (PARAM_SPECS), plan parsing, and strategy builders (build_*) in a new ingestion_planner.py.
• Wires auto= / planner= into GraphRAG.ingest() + batch/sync variants and exposes planner APIs at package top-level.
• Adds end-to-end unit tests covering plan parsing, clamping, builders, planners, and ingest merge behavior; updates ingestion docs.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
graphrag_sdk/tests/test_ingestion_planner.py	New test suite for planner parsing/clamping/builders and ingest(auto=...) merge behavior.
graphrag_sdk/src/graphrag_sdk/ingestion/ingestion_planner.py	New ingestion-planning module (plans, planners, parsing, clamping, and builders).
graphrag_sdk/src/graphrag_sdk/ingestion/init.py	Re-exports planner-related symbols from graphrag_sdk.ingestion.
graphrag_sdk/src/graphrag_sdk/api/main.py	Adds auto/planner parameters and implements _plan_ingestion_strategies integration.
graphrag_sdk/src/graphrag_sdk/init.py	Top-level exports for planner APIs.
docs/ingestion.md	Documents auto=True usage and planner options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[galshubeli]

Review — agentic ingestion strategy selection

Nice work overall — the safety-first design is solid: strategy ids validated, params clamped with the cross-field guards (chunk_overlap < chunk_size, hard > soft) matching the real constructor constraints, and LatencyBudgetExceededError correctly re-raised while everything else degrades to defaults. Mirrors the retrieval PathRouter contract cleanly.

Issues found, by size:

Big

1. Double document load — _plan_ingestion_strategies fully loads the file to get a content sample, then IngestionPipeline loads it again. With auto=True that's two full parses per document (expensive for PDFs, multiplied in batch mode), and the sample load also runs for HeuristicIngestionPlanner, which mostly needs the path. Suggest caching the loaded document for the pipeline, or at least skipping the sample load when the planner doesn't need content and documenting the extra cost.

Medium

2. No end-to-end test of ingest(auto=True) — tests drive _plan_ingestion_strategies via a stub, but nothing exercises the real ingest() wiring (planner runs after loader resolution + ontology init). One integration test with a mocked LLM would close this.

Small

3. planner: Any | None in the public API — a small Protocol (async def plan(...) -> IngestionPlan | None) would document the contract and give type-checker support.
4. parse_plan partial-JSON gap — if the JSON contains any one of the three keys, the key:value fallback scrape is skipped for the others. Degrades safely; just noting.
5. Double clamping (in IngestionPlan.__post_init__ and again in build_*) — harmless/idempotent, but a one-line comment would prevent a future "cleanup".
6. _GUIDE omits semantic's force_summary_threshold/max_summary_tokens even though PARAM_SPECS allows them — either add to the guide or drop from the spec.
7. Prompt injection via document content can steer the planner; blast radius is well-bounded (valid strategies, clamped params — worst case a pricier config), worth one sentence in the docs' safety-guarantees list.
8. Docs link to blob/main/ingestion_planner.py, which only exists once this and feat: agentic retrieval - graph-walk, skills, MCP server, and MultiPath path router #274 land on main — fine at merge time.

Also noting the PR is stacked on #274's branch, so merge order matters.

Verdict: approve with minor changes — #1 is the one I'd want addressed (or consciously accepted) before merge; the rest is polish.