Tarkka

Tarkka is an open, agent-first research infrastructure platform for discovering, ingesting, normalizing, organizing, and serving evidence-grounded research to humans and AI agents.

License and releases

Tarkka is licensed under Apache-2.0. The license covers Tarkka software and project-authored documentation; it does not grant rights to research content acquired, processed, or referenced by Tarkka. Releases and public package publication are maintainer-controlled and will be announced through tagged GitHub releases.

The core is intentionally usable without an LLM, hosted service, or mandatory external API. External research providers, document parsers, databases, and future model providers live behind replaceable contracts.

What works today

Local research ingestion

tarkka ingest ./notes.md
tarkka inspect <document-id>
tarkka read <document-id> --section 0

# For JATS, LaTeX, EPUB, and semantic HTML: inspect preserved citations progressively.
tarkka citations list <document-id> --limit 20
tarkka citations show <document-id> <reference-id>
tarkka citations resolve <document-id>
# Traverse only locally persisted citation relations; this does not fetch sources.
tarkka citations traverse <work-id> --max-depth 1 --max-works 50

# Inspect preserved supplements, datasets, software, and alternate representations.
tarkka resources list <document-id> --limit 20
tarkka resources show <document-id> <resource-link-id>

# Record and inspect reviewable claim-to-evidence assessments.
tarkka verify record <claim-id> --kind supports --evidence <evidence-id> \
  --verifier human-review --verifier-version 1 --confidence 0.9
tarkka verify list <claim-id>
tarkka verify show <relation-id>

# Start agent/tool discovery with a small operation index, then load one schema.
tarkka capabilities list
tarkka capabilities show research.verify.candidates

# Expand normalized source content progressively, from a manifest to one section.
tarkka documents manifest <document-id>
tarkka documents sections <document-id> --limit 20
tarkka documents section <document-id> <section-id>
tarkka documents package <document-id> --section <section-id> --section <section-id>
tarkka documents package <document-id> --section <section-id> --save
tarkka documents saved-package <context-package-id>

Tarkka stores immutable source artifacts by SHA-256, records acquisition provenance, normalizes content into Document -> Section -> Passage, and exposes compact manifests before full content. Native-structure parsers also preserve bibliography entries, inline citations, and exact contexts; the citation CLI lists compact references before expanding a single reference's source text/context. citations resolve performs exact identifier resolution and creates a native cites relation only when the citing Work is explicit or uniquely linked to the Document. They also preserve source-observed resource relationships; the resources CLI follows the same compact-list then explicit-detail pattern without fetching or identity-resolving a target. Verification assessments are separately auditable and expand back to their exact source evidence.

Install the optional Docling integration for richer formats such as PDF, DOCX, PPTX, HTML, and images:

python -m pip install -e '.[docling]'
tarkka ingest ./paper.pdf

Docling is an adapter, not a core dependency.

Scholarly discovery

# Default AUTO policy (currently prefers OpenAlex for broad discovery)
tarkka discover "machine learning MLB game outcome prediction"

# One provider
tarkka discover "pitcher fatigue" --provider semantic-scholar

# Selected providers
tarkka discover "baseball forecasting" \
  --provider openalex \
  --provider crossref

# Exhaustive enabled-provider fan-out
tarkka discover "baseball forecasting" --provider all

Current scholarly adapters:

• OpenAlex
• Crossref
• Semantic Scholar

Discovery supports provider selection, bounded concurrent fan-out, retries/rate-limit handling, provider-specific continuation cursors, DOI-first deduplication, and reproducible SearchSnapshots. Providers remain independent adapters; cross-provider identity and enrichment happen in application services rather than providers calling one another.

Work and document persistence use local JSON by default and therefore need no database service. To opt into PostgreSQL, install the optional extra and select the backend explicitly; a database URL alone never changes the default. On a new database, apply the current schema migrations first:

uv sync --extra postgres
export TARKKA_DATABASE_URL=postgresql://localhost/tarkka
tarkka db upgrade
TARKKA_WORK_BACKEND=postgres \
tarkka work show <work-id>

TARKKA_DOCUMENT_BACKEND=postgres \
tarkka documents manifest <document-id>

Install the optional MCP transport to expose the same staged, read-only document services to an MCP client over stdio. The server deliberately lists capabilities and document structure before returning a selected section's source-preserving passage text:

uv sync --extra mcp
tarkka-mcp

The initial MCP tools are research_capabilities, research_operation_schema, document_manifest, document_sections, and document_section. They use the same TARKKA_DOCUMENT_BACKEND selection as the document CLI; context-package saves and other writes remain CLI/application-service operations until their audit boundaries are exposed explicitly. Exact section expansion is rejected above the same 8,000 estimated-token budget used for document context packages, so one MCP call cannot force an unbounded document response.

For transparent, opt-in local telemetry, set a JSONL destination before starting the server. Events contain only tool name, outcome/error code, response byte count, estimated tokens, and latency; they never contain document text, request arguments, or identifiers:

export TARKKA_MCP_TELEMETRY_PATH=./var/tarkka-mcp-usage.jsonl
tarkka-mcp

Agent-first design

Tarkka uses progressive disclosure to conserve context:

capabilities
  -> manifests
  -> summaries / structured findings
  -> evidence
  -> full sections/documents
  -> raw artifacts

Agents should retrieve the smallest representation that can answer the current question, then expand only when necessary. See AGENTS.md, CLAUDE.md, and docs/CONTEXT_EFFICIENCY.md.

AGENTS.md is the shared repository instruction file for Codex, Claude, and other coding agents. CLAUDE.md contains only Claude-specific context-loading guidance so architectural rules are not duplicated.

The first portable staged skill is available at:

skills/research-discovery/SKILL.md

Architecture principles

• provider-neutral and domain-neutral core
• no LLM required for core operation
• immutable content-addressed raw artifacts
• provenance as a first-class concern
• claims, evidence, and citations remain distinct
• rights/access/redistribution policy modeled separately from software licensing
• PostgreSQL is the reference production metadata store
• adapters/plugins remain replaceable
• domain-specific semantics belong in domain packs
• CLI/API/MCP/SDK should share application services

Current roadmap position

Foundation through the initial structured extraction/source-intelligence workflow is delivered. Evidence verification is substantially complete for the local workflow, and the first agent-serving foundation now offers compact transport-neutral capability discovery for implemented discovery, verification, citation traversal, and source-resource inspection services.

Immediate engineering sequence:

evidence verification
  -> bounded source/citation expansion
  -> compact agent capability discovery
  -> manifest / summary / evidence expansion services
  -> MCP serving and saved context packages

See docs/ROADMAP.md and docs/MILESTONE_3.md for details.

Development

python -m pip install -e '.[dev]'
ruff check .
mypy
pytest

CI validates Ruff, strict mypy, and pytest across Python 3.11–3.13. Optional Docling integration is verified separately with a CPU-only workflow.

Documentation

Start with:

1. AGENTS.md
2. docs/PROJECT_CHARTER.md
3. docs/ARCHITECTURE.md
4. docs/ROADMAP.md

Then load task-specific documents on demand. The repository intentionally avoids requiring agents or humans to read the entire documentation set before doing focused work.