xTap

Passively capture tweets as you browse X/Twitter

Installation · How It Works · Stealth · Output Format · Configuration · License

xTap is a browser extension (Chrome + Firefox) that silently intercepts the GraphQL API responses X/Twitter already sends to your browser and saves every tweet you encounter as structured JSONL. No scraping, no extra requests — just a tap on the data already flowing through.

Features

Zero footprint — no additional network requests; captures what your browser already receives
Structured output — each tweet saved as a clean JSON object with author, metrics, media, and more
Article support — long-form X articles are captured when X returns the full body, with inline image references and Draft.js block structure
Video download — download videos from tweets using yt-dlp (or direct MP4 fallback) via the extension popup. Requires the HTTP daemon. Note: unlike passive capture, video downloads make additional network requests to X and are not stealth.
Image download — opt-in toggle in the popup ("Download images automatically") fetches photos from pbs.twimg.com to <output_dir>/media/<tweet_id>/<filename> as you browse. If enabled after browsing, already-seen photo tweets are rechecked for media download without bumping the captured-tweet counter. Daemon-side; rate-limited; logs to media-manifest.jsonl. Note: also not stealth — adds requests to the Twitter CDN.
Pause / resume — click the extension icon to toggle capture on the fly
Live counter — badge on the extension icon shows tweets captured this session
Multi-tab aware — multiple X tabs feed into the same service worker with shared deduplication
Debug logging — optional toggle to write timestamped service worker logs to a date-rotated file
Debug dashboard — internal extension page with live capture events, transport health, and a parser sandbox for testing GraphQL response parsing
Cross-platform — works on macOS, Linux, and Windows

How It Works

        X/Twitter GraphQL responses
                    │
                    ▼
     ┌────────────────────────────┐
     │     content-main.js        │  MAIN world
     │    patches fetch & XHR     │
     └──────────────┬─────────────┘
                    │ CustomEvent (random name)
                    ▼
     ┌────────────────────────────┐
     │     content-bridge.js      │  ISOLATED world
     │   relays to service worker │
     └──────────────┬─────────────┘
                    │ chrome.runtime.sendMessage
                    ▼
     ┌────────────────────────────┐
     │     background.js          │  Service worker
     │   parse, dedup, batch      │
     └──────────┬─────────┬───────┘
                │         │
          HTTP  │         │ native messaging
     (all data) │         │ (token bootstrap only)
                │         │
                ▼         ▼
     ┌──────────────┐  ┌──────────────┐
     │ xtap_daemon  │  │ xtap_host.py │
     │ (HTTP)       │  │ (stdio)      │
     └──────┬───────┘  └──────────────┘
            │
            ▼
       tweets-YYYY-MM-DD.jsonl

A MAIN world content script patches fetch and XMLHttpRequest.open() to observe GraphQL responses as they arrive
Payloads are relayed via a random-named CustomEvent to an ISOLATED world bridge, which forwards them to the service worker
The service worker parses, normalizes, deduplicates, and batches tweets
Batches are sent to disk via the HTTP daemon (xtap_daemon.py), a standalone process on 127.0.0.1:17381 managed by launchd (macOS), systemd (Linux), or Scheduled Task (Windows). On macOS, it runs outside browser TCC sandboxes and can write to protected paths like ~/Documents and iCloud Drive
At startup, the extension retrieves the daemon's auth token via native messaging (xtap_host.py over Chrome/Firefox native messaging). This is a one-time bootstrap — all data flows through HTTP

Is This Safe to Use?

X is rolling out stricter detection for automation and bots. The key line: "If a human is not tapping on the screen, the account and all associated accounts will likely be suspended."

xTap is not a bot. It doesn't post, like, follow, scroll, or make any API calls on your behalf. It sits in the background and reads the responses X already sent to your browser while you browse normally. From X's server-side perspective, your account looks identical to any other user — because you are a normal user. There is no extra traffic to detect.

The risk of automation enforcement applies to tools that act as you (auto-liking, auto-following, automated scrolling, headless browsers). xTap does none of that. It's the equivalent of keeping DevTools open and saving the Network tab — just automated into structured JSONL.

Stealth Measures

Even though passive interception is inherently low-risk, xTap avoids leaving unnecessary traces:

No extra network requests — only reads responses the browser already received; nothing to spot in a network log
Native-looking API patches — fetch and XMLHttpRequest.prototype.open are patched with toString() overrides that return [native code], passing the most common runtime integrity checks
No expando properties — XHR URL tracking uses a WeakMap instead of attaching properties to the XHR instance, which would be trivially detectable
Random event channel — the MAIN↔ISOLATED world bridge uses a CustomEvent with a per-page-load random name; the <meta> beacon that communicates the name is removed immediately after the bridge reads it
Zero DOM footprint — no injected UI, no page modifications; everything lives in the popup and service worker
Zero console output in page context — all logging happens in the service worker and parser, which run outside the page's JavaScript environment
Minimal permissions — only storage, nativeMessaging, and alarms (the alarm wakes the service worker to deliver buffered tweets; it is not observable by web pages); no webRequest, no host permissions beyond x.com / twitter.com / 127.0.0.1
Jittered flush timing — batches are flushed on a randomized interval to avoid a clockwork-regular pattern

These measures don't make detection impossible — a determined page script could still compare prototype references or probe for patched behavior — but they avoid the low-hanging signals that fingerprinting scripts typically check. More importantly, there's nothing to detect server-side because xTap generates zero network activity of its own.

Installation

Requirements

	Requirement
Browser	Google Chrome or Mozilla Firefox (128+)
Runtime	Python 3
OS	macOS, Linux, or Windows
`yt-dlp` (optional)	For best-quality video downloads

1. Load the extension

Chrome:

Open chrome://extensions
Enable Developer mode (top right)
Click Load unpacked and select the xtap/ directory
Copy the extension ID shown on the card (used by native host install)

Firefox (128+):

Create a Firefox copy of the extension directory (so your Chrome manifest stays unchanged)
In that copy, replace manifest.json with manifest.firefox.json (rename it to manifest.json)
Open about:debugging#/runtime/this-firefox
Click Load Temporary Add-on...
Select the Firefox copy's manifest.json

Firefox uses the fixed extension ID from manifest.firefox.json: xtap@mkubicek.dev.

Thanks to Vincent Koc for the Firefox support contribution.

2. Install the native host

macOS

cd native-host
./install.sh <your-extension-id> chrome

For Firefox:

cd native-host
./install.sh firefox

This installs the native messaging host (for auth token bootstrap) and an HTTP daemon (xtap_daemon.py) that runs via launchd. The daemon runs independently of the browser process tree and has its own TCC permissions, so it can write to protected paths like ~/Documents and iCloud Drive. The installer captures your current PATH so the daemon can find tools like yt-dlp.

The extension automatically detects the daemon via the native host's auth token. If the daemon is not running, the extension will show a red "!" badge and an error in the popup.

Linux

cd native-host
./install.sh <your-extension-id> chrome

For Firefox:

cd native-host
./install.sh firefox

This installs the native messaging host and an HTTP daemon (xtap_daemon.py) that runs as a systemd user service. The daemon enables video downloads and provides the same HTTP transport as macOS.

Windows (PowerShell)

cd native-host
.\install.ps1 -ExtensionId <your-extension-id> -Browser chrome

For Firefox:

cd native-host
.\install.ps1 -Browser firefox

This installs the native messaging host and an HTTP daemon (xtap_daemon.py) as a Windows Scheduled Task that starts at logon. The daemon enables video downloads and provides the same HTTP transport as macOS/Linux.

3. Browse X

Open x.com and browse normally. The badge counter on the extension icon shows how many tweets have been captured this session. Click the icon to see stats and pause/resume capture.

After updating the extension: Reload xTap in your extension manager (chrome://extensions or about:debugging#/runtime/this-firefox), then hard-reload any open X tabs (Cmd+Shift+R / Ctrl+Shift+R). The content scripts that intercept API responses are injected at page load, so stale scripts from before the update won't connect to the new service worker.

Upgrading from a previous version

After updating the extension files:

Re-run the installer (install.sh on macOS/Linux, install.ps1 on Windows) — this updates the daemon configuration and picks up new Python code
Reload the extension in your browser extension manager
Hard-reload any open X tabs (Cmd+Shift+R / Ctrl+Shift+R)

From versions before v0.20.0 on macOS/Linux: Re-running install.sh is required — the native messaging manifest now points to a wrapper script (~/.xtap/xtap_host_wrapper.sh) that uses an absolute Python path, fixing native host launch failures on macOS where Chrome's minimal environment couldn't find python3.

From versions before v0.19.0: The native messaging host (xtap_host.py) no longer handles tweet writing — all data now flows through the HTTP daemon exclusively. Re-running install.sh is required to update the daemon's service configuration (adds XTAP_LOG_LEVEL support). The extension will show a red "!" badge if the daemon is not running, instead of silently falling back to native messaging.

From versions before v0.13.0 on macOS: Re-running install.sh is required for video download support — the daemon needs an updated launchd configuration to find yt-dlp on your PATH.

Troubleshooting

If the extension shows "Not connected" or a red "!" badge:

Check if the daemon is running:

curl http://127.0.0.1:17381/status
# Should return: {"ok": true, "version": "..."}

Check daemon logs:
```
cat ~/.xtap/daemon-stderr.log
```
The daemon logs startup diagnostics (Python version, output directory, token status) on every start. Common issues:
- FATAL: ~/.xtap/secret not found — run install.sh first
- FATAL: Cannot bind to 127.0.0.1:17381 — another instance is already running
- Import errors — check Python version (python3 --version, requires 3.x)
Check native host errors (token bootstrap failures):
```
cat ~/.xtap/host-error.log
```
This file is created when the native messaging host crashes. It includes the Python version, script path, and full traceback.
Enable debug logging for detailed request-level daemon logs:
```
export XTAP_LOG_LEVEL=debug
cd native-host && ./install.sh <extension-id> chrome
```
Then check ~/.xtap/daemon-stderr.log for per-request details (method, path, duration, tweet counts).
Verify the native messaging manifest points to the correct path:
```
# Chrome (macOS):
cat ~/Library/Application\ Support/Google/Chrome/NativeMessagingHosts/com.xtap.host.json
# Firefox (macOS):
cat ~/Library/Application\ Support/Mozilla/NativeMessagingHosts/com.xtap.host.json
```
The path field should point to ~/.xtap/xtap_host_wrapper.sh (macOS/Linux) or xtap_host.bat (Windows). The wrapper uses an absolute Python path so native messaging works even in Chrome's minimal environment. If it still points directly at xtap_host.py, re-run install.sh.

Configuration

Output directory

The easiest way to change where tweets are saved is through the extension popup — click the xTap icon and enter your preferred path in the Output directory field.

Alternatively, set the XTAP_OUTPUT_DIR environment variable and re-run the installer — the daemon runs as a system service (launchd/systemd/Scheduled Task), so the variable must be baked into the service definition; exporting it in a shell or before launching the browser has no effect:

export XTAP_OUTPUT_DIR="$HOME/Documents/xtap-data"
./native-host/install.sh <extension-id> [chrome|firefox]

On Windows, set it as a user environment variable ([Environment]::SetEnvironmentVariable('XTAP_OUTPUT_DIR', 'D:\\path', 'User')) and re-run install.ps1.

Setting	Default	Description
Popup "Output directory"	(empty — uses default)	Overrides the output path per-session
`XTAP_OUTPUT_DIR` env var	`~/Downloads/xtap`	Fallback when no popup setting is configured (set at install time)
Debug Dashboard	—	Accessible via popup link; shows live capture events, transport health, debug logging and discovery mode toggles, and parser sandbox

macOS note: On macOS, the HTTP daemon (installed via install.sh) runs outside browser TCC sandboxes and can write to protected paths like ~/Documents and iCloud Drive after a one-time macOS permission prompt.

Download tuning

These daemon environment variables are optional. Re-run the installer after changing them so the service definition picks them up.

Setting	Default	Description
`XTAP_IMAGE_DELAY_MS`	`100`	Delay between background image requests
`XTAP_MAX_FILE_MB`	`50`	Max size for one downloaded image
`XTAP_MAX_MEDIA_MB`	(unlimited)	Max cumulative image bytes per daemon process
`XTAP_MAX_VIDEO_MB`	`500`	Max size for one direct MP4 fallback video download; set `0` or less to disable the cap
`XTAP_CONN_TIMEOUT_S`	`15`	Daemon per-connection socket timeout — drops idle/stalled connections; bad or `<=0` values fall back to the default (the timeout is never disabled)
`XTAP_SHUTDOWN_GRACE_S`	`10`	Max seconds the daemon waits for in-flight requests at shutdown before force-exiting; bad or `<=0` values fall back to the default

Output Format

Output is written to daily files (tweets-YYYY-MM-DD.jsonl). Each line is a self-contained JSON object:

{
  "id": "1234567890",
  "url": "https://x.com/handle/status/1234567890",
  "created_at": "2024-01-01T00:00:00.000Z",
  "author": {
    "id": "987654321",
    "username": "handle",
    "display_name": "Display Name",
    "verified": false,
    "is_blue_verified": true,
    "follower_count": 1234
  },
  "text": "Full tweet text...",
  "lang": "en",
  "metrics": {
    "likes": 10,
    "retweets": 5,
    "replies": 2,
    "views": 1000,
    "bookmarks": 1,
    "quotes": 0
  },
  "media": [],
  "urls": [],
  "hashtags": [],
  "mentions": [],
  "in_reply_to": null,
  "quoted_tweet_id": null,
  "conversation_id": "1234567890",
  "is_retweet": false,
  "retweeted_tweet_id": null,
  "is_subscriber_only": false,          // true for subscriber-only tweets
  "is_article": true,                   // present only for long-form articles
  "article": {                          // present only for long-form articles
    "title": "Article Title",
    "text": "Rendered plain text with ![img](media/<id>/file.png) refs",
    "blocks": [],                       // raw Draft.js content_state blocks
    "media": [{                         // article image references
      "id": "...",
      "url": "https://pbs.twimg.com/...",  // original CDN URL
      "filename": "image.png",
      "local_path": "media/<tweet_id>/image.png",
      "width": 1200,
      "height": 800
    }]
  },
  "source_endpoint": "HomeTimeline",    // which GraphQL endpoint
  "captured_at": "2024-01-01T00:00:00.000Z"
}

For regular tweets, is_article and article are absent. For articles, text contains a markdown-style rendering of the article with inline image references pointing to media/<tweet_id>/. Article stubs without content_state.blocks are skipped rather than saved as incomplete rows.

Media file convention

When the "Download images automatically" toggle is on, the daemon writes photos to:

<output_dir>/media/<tweet_id>/<basename(media.url)>

Top-level photo media[] entries do not carry a local_path field — the path is derived by convention so consumers can reconstruct it directly from tweet.id + the URL basename. Article media (tweet.article.media[]) does include local_path because that path is also embedded in the rendered article markdown (![](media/<id>/file.png)) so the article body works as a self-contained document. Download status (success / 404 / quota / oversize / blocked) is appended to <output_dir>/media-manifest.jsonl.

Project Structure

xTap/
├── manifest.json              # Chrome MV3 extension manifest
├── manifest.firefox.json      # Firefox MV3 extension manifest (generated — do not edit)
├── background.js              # Service worker — parsing, dedup, transport
├── content-main.js            # MAIN world — patches fetch/XHR, emits events
├── content-bridge.js          # ISOLATED world — relays events to service worker
├── popup.html/js/css          # Extension popup UI
├── debug.html/js/css          # Debug dashboard (live events, transport health, parser sandbox)
├── icons/                     # Extension icons
├── lib/                       # Shared utilities
└── native-host/
    ├── xtap_core.py              # Shared file I/O logic
    ├── xtap_host.py              # Native messaging host — token bootstrap only (Python, stdio)
    ├── xtap_daemon.py            # HTTP daemon
    ├── com.xtap.daemon.plist     # launchd plist template (macOS)
    ├── com.xtap.daemon.service   # systemd unit template (Linux)
    ├── com.xtap.host.json        # Native host manifest template (Chrome)
    ├── com.xtap.host.firefox.json # Native host manifest template (Firefox)
    ├── install.sh                # Installer for macOS / Linux
    ├── install.ps1               # Installer for Windows
    ├── xtap_host.bat             # Windows native host wrapper
    └── xtap_daemon.bat           # Windows daemon wrapper

Development

After modifying extension files (background.js, lib/, content-*.js, popup.*), reload the extension in your browser (chrome://extensions or about:debugging#/runtime/this-firefox) and hard-reload any open X tabs.

Debug dashboard: Click "Debug Dashboard" in the popup to open a live view of capture events, transport health, and a parser sandbox for testing extractTweets against raw GraphQL JSON. Debug logging and discovery mode toggles are also here — enable debug logging to write timestamped service worker logs to debug-YYYY-MM-DD.log, or discovery mode to log endpoint response shapes to the console.

Dev mode: When loaded unpacked (developer mode), the extension prefers chrome.storage.session for the seenIds dedup cache, and falls back to chrome.storage.local if session storage APIs are unavailable. When session storage is available, reloading the extension automatically clears the cache — no need to manually clear storage between test runs.

After modifying Python host files (xtap_core.py, xtap_host.py, xtap_daemon.py), the native host picks up changes on next browser restart. To restart the HTTP daemon immediately:

macOS (launchd):

launchctl kickstart -k gui/$(id -u)/com.xtap.daemon   # restart
launchctl bootout gui/$(id -u)/com.xtap.daemon        # stop
launchctl print gui/$(id -u)/com.xtap.daemon          # status
tail -f ~/.xtap/daemon-stderr.log                     # logs

Linux (systemd):

systemctl --user restart com.xtap.daemon   # restart
systemctl --user stop com.xtap.daemon      # stop
systemctl --user status com.xtap.daemon    # status
journalctl --user -u com.xtap.daemon -f    # logs

Windows (Scheduled Task, PowerShell):

Stop-ScheduledTask -TaskName xTapDaemon; Start-ScheduledTask -TaskName xTapDaemon  # restart
Stop-ScheduledTask -TaskName xTapDaemon                                            # stop
Get-ScheduledTask -TaskName xTapDaemon                                             # status
Get-Content ~\.xtap\daemon-stderr.log -Tail 50 -Wait                               # logs

Testing

python3 -m pytest tests/ -v
node --test tests/*.test.mjs

CI runs these on every push to main with coverage uploaded to Codecov.

Parser fixture packs live under tests/fixtures/. Raw captures stay local in tests/fixtures/private-raw/ (gitignored), while committed anonymized packs live in tests/fixtures/sanitized/. The anonymization methodology and review checklist are documented in tests/fixtures/FIXTURES.md.

License

MIT — use it however you like.

Navigation Menu

Repository files navigation

xTap

Features

How It Works

Is This Safe to Use?

Stealth Measures

Installation

Requirements

1. Load the extension

2. Install the native host

3. Browse X

Upgrading from a previous version

Troubleshooting

Configuration

Output directory

Download tuning

Output Format

Media file convention

Project Structure

Development

Testing

License

About

Releases

Packages

Used by

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
icons		icons
lib		lib
native-host		native-host
scripts		scripts
tests		tests
.coveragerc		.coveragerc
.gitignore		.gitignore
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
background.js		background.js
content-bridge.js		content-bridge.js
content-main.js		content-main.js
debug.css		debug.css
debug.html		debug.html
debug.js		debug.js
manifest.firefox.json		manifest.firefox.json
manifest.json		manifest.json
popup.css		popup.css
popup.html		popup.html
popup.js		popup.js

Latest commit

History

Folders and files

Repository files navigation

xTap

Features

How It Works

Is This Safe to Use?

Stealth Measures

Installation

Requirements

1. Load the extension

2. Install the native host

3. Browse X

Upgrading from a previous version

Troubleshooting

Configuration

Output directory

Download tuning

Output Format

Media file convention

Project Structure

Development

Testing

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages