Capture, preserve, and explore your own sessions across AI platforms (ChatGPT, Claude.ai, Gemini, NotebookLM, Qwen, DeepSeek, Perplexity, Grok, Kimi) plus command-line tools (Claude Code, Codex, Gemini CLI, Antigravity CLI). Data is preserved locally in canonical format (parquet), even if you delete it from the server.
The repository contains the code, documentation, and DVC pointers; personal data, browser profiles, and private operating notes stay outside Git. This makes it possible to rebuild a working machine without publishing the archive itself.
This tool is for personal use, with your own accounts. It uses the platforms' internal APIs authenticated with cookies from your own login (access you already have). It is not a tool for scraping data from other users or for bypassing terms of use — and should not be used that way.
AI platforms have limited official exports, often broken, with no guarantee of retention. You have no way of knowing whether an old conversation will be accessible 6 months from now, or whether a new feature will disappear taking data with it.
This project solves that by capturing everything locally:
- Conversations, projects, knowledge files, artifacts (canvas, deep research reports, slide decks)
- Generated images (DALL-E, Nano Banana), user uploads, mind maps
- Voice messages (transcripts), thinking blocks (reasoning), tool calls
- Chats deleted on the server — preserved in the local cumulative archive and in the published canonical base
Output in parquet (unified schema across all 13 sources), ready for analysis in pandas/DuckDB/Quarto/whatever you prefer.
All 13 sources have implemented capture/copy, consolidation where applicable, canonical parsing, and descriptive visualization (Quarto):
| Source | Type | Coverage |
|---|---|---|
| ChatGPT | web | branches, voice, DALL-E, projects, custom GPT |
| Claude.ai | web | thinking, tool use+MCP, project_docs with inline content |
| Perplexity | web | threads + pages + spaces + 9 artifact types |
| Qwen | web | 8 chat types (search, research, dalle, etc.), projects |
| DeepSeek | web | R1 reasoning (thinking in ~31% of msgs), token usage |
| Gemini | web | multi-account (2 Google accounts), 8 models |
| NotebookLM | web | 2 active accounts plus a separate legacy archive; 9 output types (audio, video, slide deck, etc.) |
| Grok | web | conversations, workspaces, tool events, assets, scheduled tasks |
| Kimi | web | chats, installed skills, tool events, signed asset downloads |
| Claude Code | CLI | local sessions (~/.claude/projects/), subagents |
| Codex | CLI | local sessions (~/.codex/sessions/), exact latency per tool call |
| Gemini CLI | CLI | local sessions (~/.gemini/tmp/) |
| Antigravity CLI | CLI | local trajectories (~/.gemini/antigravity-cli/brain/) |
The automated test suite covers extractors, reconcilers, parsers, the canonical schema, dashboard, and unification. Known limitations and gaps are documented in extractor engineering's known limitations.
Prerequisites: Python ≥3.12, macOS or Linux. Windows not tested.
git clone <repo>
cd multi-ai-session-data-extractor
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
playwright install chromiumLogin (once per platform — opens a browser, you log in manually, close):
python scripts/chatgpt-login.pySync web data (capture + consolidation), then parse it:
python scripts/chatgpt-sync.py
python scripts/chatgpt-parse.pyResult:
data/raw/ChatGPT/— raw capture (cumulative, keeps binaries)data/merged/ChatGPT/— consolidated version (also keeps conversations deleted from the server)data/processed/ChatGPT/*.parquet— canonical format for analysis
Repeat the 2 commands for other web platforms (claude-login.py,
gemini-sync.py, etc.). CLI sources are copied and parsed by their respective
*-sync.py commands. Details in docs/SETUP.md.
If you are restoring an existing personal archive rather than starting a new
one, first restore your private DVC configuration and run dvc pull. The
DVC runbook explains that recovery path and the data
retention policy.
extractor → reconciler → parser → unify
raw → merged → processed (per-source) → unified (cross-source)
- Extractor downloads via the platform's internal API (authenticated with your cookie).
- Reconciler consolidates what you just captured with what you already had — preserving records that disappeared from the server.
- Parser converts the raw JSON into parquet with a unified schema:
Conversation,Message,ToolEvent,Branch(and a few auxiliaries per platform —ProjectDoc,NotebookLMOutput, etc.). - Unify consolidates the parquets from the 13 sources into a single
data/unified/with 13 parquet files (4 canonical + 9 auxiliaries), ready for cross-platform analysis.
Full schema in src/schema/models.py. Capture and parser terminology is in
the extractor engineering glossary.
Login is always with a visible window (once per platform — you need to log in manually). Capture after that varies:
| Platform | Capture |
|---|---|
| Claude.ai, Gemini, NotebookLM, Qwen, DeepSeek, Grok, Kimi | No visible window |
| ChatGPT, Perplexity | Visible window (Cloudflare detects scraping without a window) |
If you run Claude.ai/Gemini/NotebookLM/Qwen/DeepSeek/Grok/Kimi and see a window open during capture: something is wrong (likely an expired cookie). For ChatGPT/Perplexity: expected behavior.
Each web platform has 2-4 scripts in scripts/. Pattern:
python scripts/<plat>-login.py # once — manual login in the browser
python scripts/<plat>-sync.py # capture + consolidation
python scripts/<plat>-parse.py # merged -> canonical parquetCommon web-sync flags (availability varies by source):
--full— force full recapture (skips the incremental path)--no-binaries— skip asset downloads (images, slide decks, etc.)--no-reconcile— skip consolidation (capture only)--dry-run— show what would happen without executing
Full list of commands per platform: docs/operations/pipeline.md.
Local Streamlit visualization — cross-platform totals, per-platform status, links to the descriptive documents:
PYTHONPATH=. .venv/bin/streamlit run dashboard.pyOpens at http://localhost:8501. Read-only over what sync produced — does not write or edit.
Details in docs/operations/dashboard.md.
Platform and cross-platform profiles cover data schema, coverage, distributions, and examples. They share templates to avoid duplication.
QUARTO_PYTHON="$(pwd)/.venv/bin/python" quarto render notebooks/chatgpt.qmd
QUARTO_PYTHON="$(pwd)/.venv/bin/python" quarto render notebooks/00-overview.qmd
To view the generated HTMLs locally:
./scripts/serve-qmds.sh openPYTHONPATH=. .venv/bin/pytest # everything
PYTHONPATH=. .venv/bin/pytest tests/parsers/ # parsers only- docs/README.md — full index
- docs/SETUP.md — detailed setup, first login, and troubleshooting
- docs/operations/dvc-runbook.md — DVC operational guide (canonical-current data vault and local recovery)
- docs/extractor-engineering/known-limitations.md — known extractor gaps and limitations
- docs/operations/pipeline.md — common commands per platform
- docs/extractor-engineering/glossary.md — capture and parser terms
- platform engineering records — empirical behavior per platform
- docs/SECURITY.md — credentials and ToS policy
- docs/CONTRIBUTING.md — contributor guide
- Capture once, never downgrade. Once something is captured, it stays local. Reruns only fetch new items.
- Preservation above all. Conversations/files deleted on the
server remain local with the
is_preserved_missing=Trueflag. - The canonical schema is the boundary. Parsers deliver parquet in a unified schema; analysis consumes parquet. No platform particularities leak into the analysis stage.
- Fall back safely in suspicious cases. If a discovery listing drops materially versus known history, the extractor must not trust it blindly; the applicable platform guardrail preserves or refetches known records.
MIT — see LICENSE.
Issues and PRs welcome. Details in docs/CONTRIBUTING.md. The technical guide for a new platform is adding-a-platform.md.