Twitter (X) Scraper

Twitter (X) data, powered by Bright Data.

This repository provides two approaches to accessing Twitter (X) data at scale:

Method 1: Bright Data Twitter (X) Scraper API (Recommended) - A fully managed, enterprise-grade scraping API that handles proxies, CAPTCHAs, and scaling automatically.
Method 2: Bright Data Twitter (X) Datasets - Ready-to-download, pre-collected Twitter (X) datasets, no scraping required.

Why Use Bright Data for Twitter (X) Scraping?

Twitter (X) scraping comes with several challenges:

Rate Limiting: Twitter (X) aggressively rate-limits automated requests.
Authentication Barriers: Most data requires login; automated access is detected.
Dynamic Content Loading: JavaScript-rendered content requires headless browser.
IP Blocking: Repeated requests from the same IP may result in permanent bans.
API Restrictions: Free API access is severely limited since 2023.

Bright Data's Twitter (X) Scraper API solves these problems with:

✅ Built-in rotating proxies: Bypass IP-based rate limits automatically
✅ CAPTCHA solving: Handles bot detection without any extra setup
✅ Structured data output: Receive clean JSON ready for analysis
✅ No infrastructure needed: Cloud-managed scraping at any scale
✅ 99.9% uptime SLA: Reliable data collection for business-critical workflows

Method 1: Bright Data Twitter (X) Scraper API

The Bright Data Twitter (X) Scraper API is a fully managed solution requiring zero infrastructure setup.

Getting Started with the Twitter (X) Scraper API

Sign up for a free Bright Data account
Navigate to the Twitter (X) Scraper API
Get your API token from the dashboard
Install the requests library: pip install requests
Run any of the scripts in twitter_scraper_api_codes/

1. Twitter Posts by URL

Collect detailed data from specific Twitter/X posts using their direct URLs, including engagement metrics, author details, and full post content.

Input Parameters

Field	Type	Required	Description
`url`	string	Yes	The URL of the Twitter/X post to scrape
`limit`	integer	No	Maximum number of results to return
`include_errors`	boolean	No	Include error details in the response
`notify`	url	No	Webhook URL to notify when collection is complete
`format`	enum	No	Output format: `JSON`, `NDJSON`, `JSON Lines`, `CSV`

Sample Response

{
  "date_posted": "2026-04-16T17:14:49.000Z",
  "db_source": "1776417080791",
  "description": "\ud83d\udea8 Collin is set to tee off shortly for his first round at the RBC Heritage! \n\nA lot of low scores out there, with the a...",
  "id": "2044826862996656217",
  "name": "Morikawa T*****r",
  "user_posted": "Mor***waT***ker***"
}

👉 View Full Python Code

2. Twitter Profile Posts

Collect posts from a Twitter/X profile, with control over the maximum number of posts to retrieve per profile.

Input Parameters

Field	Type	Required	Description
`url`	string	Yes	The URL of the Twitter/X profile
`max_number_of_posts`	integer	No	Maximum number of posts to collect (default: 100)
`limit`	integer	No	Maximum number of results to return
`include_errors`	boolean	No	Include error details in the response
`notify`	url	No	Webhook URL to notify when collection is complete
`format`	enum	No	Output format: `JSON`, `NDJSON`, `JSON Lines`, `CSV`

Sample Response

{
  "biography": "Information is the heart and lifeblood of Organisations. Data needs to be Fit_for_purpose, Robust \u0026amp; Secure. Cyber-security Independent Audits by the Experts,",
  "date_joined": "2012-06-13T10:12:31.000Z",
  "external_link": "http://www.mgba.co.uk/",
  "followers": 82,
  "following": 126,
  "id": "BrightData",
  "is_business_account": false,
  "is_government_account": false,
  "is_verified": false,
  "location": "Birmingham, UK",
  "posts_count": 189,
  "profile_image_link": "https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png",
  "profile_name": "BrightData",
  "subscriptions": 0,
  "url": "https://x.com/BrightData",
  "x_id": "607080773"
}

👉 View Full Python Code

3. Twitter Date Range Posts

Collect Twitter/X posts from a specific profile within a defined date range, useful for historical data collection and trend analysis.

Input Parameters

Field	Type	Required	Description
`url`	string	Yes	The URL of the Twitter/X profile
`start_date`	string	Yes	Start date for collection in YYYY-MM-DD format
`end_date`	string	Yes	End date for collection in YYYY-MM-DD format
`limit`	integer	No	Maximum number of results to return
`include_errors`	boolean	No	Include error details in the response
`notify`	url	No	Webhook URL to notify when collection is complete
`format`	enum	No	Output format: `JSON`, `NDJSON`, `JSON Lines`, `CSV`

Sample Response

{
  "biography": "Information is the heart and lifeblood of Organisations. Data needs to be Fit_for_purpose, Robust \u0026amp; Secure. Cyber-security Independent Audits by the Experts,",
  "date_joined": "2012-06-13T10:12:31.000Z",
  "external_link": "http://www.mgba.co.uk/",
  "followers": 82,
  "following": 126,
  "id": "BrightData",
  "is_business_account": false,
  "is_government_account": false,
  "is_verified": false,
  "location": "Birmingham, UK",
  "posts_count": 189,
  "profile_image_link": "https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png",
  "profile_name": "BrightData",
  "subscriptions": 0,
  "url": "https://x.com/BrightData",
  "x_id": "607080773"
}

👉 View Full Python Code

Method 2: Bright Data Twitter (X) Datasets

For use cases where you need ready-to-use data without writing any scraping code, the Bright Data Twitter (X) Dataset offers pre-collected, regularly updated data available for instant download.

Why use the dataset instead of the API?

📦 Instant access: No setup, no code, no waiting for collection
🔄 Regularly updated: Fresh data refreshed on a consistent schedule
📊 Multiple formats: Download as JSON, JSONL, or CSV
🌍 Massive scale: Millions of records across all major Twitter (X) categories
✅ Fully compliant: Ethically sourced and legally cleared data

👉 Explore the Twitter (X) Dataset

Data Collection Approaches

Feature	Bright Data Scraper API	Bright Data Datasets
Setup required	API token only	None
Real-time data	✅ Yes	❌ Pre-collected
Custom queries	✅ Full control	❌ Fixed schema
Proxies included	✅ Built-in rotating	N/A
CAPTCHA solving	✅ Automatic	N/A
Scale	Unlimited	Unlimited
Structured output	✅ JSON / NDJSON / JSON Lines / CSV	✅ JSON / JSONL / CSV
Support	Enterprise 24/7	Enterprise 24/7

🔗 Learn more: https://brightdata.com/products/web-scraper/twitter

Name		Name	Last commit message	Last commit date
twitter_scraper_api_codes		twitter_scraper_api_codes
Proxies and scrapers GitHub bonus banner.png		Proxies and scrapers GitHub bonus banner.png
README.md		README.md
requirements.txt		requirements.txt
twitter_date_range_posts.json		twitter_date_range_posts.json
twitter_posts.json		twitter_posts.json
twitter_profile_posts.json		twitter_profile_posts.json

Navigation Menu

Repository files navigation

Twitter (X) Scraper

Table of Contents

Why Use Bright Data for Twitter (X) Scraping?

Method 1: Bright Data Twitter (X) Scraper API

Getting Started with the Twitter (X) Scraper API

1. Twitter Posts by URL

Input Parameters

Sample Response

2. Twitter Profile Posts

Input Parameters

Sample Response

3. Twitter Date Range Posts

Input Parameters

Sample Response

Method 2: Bright Data Twitter (X) Datasets

Data Collection Approaches

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Twitter (X) Scraper

Table of Contents

Why Use Bright Data for Twitter (X) Scraping?

Method 1: Bright Data Twitter (X) Scraper API

Getting Started with the Twitter (X) Scraper API

1. Twitter Posts by URL

Input Parameters

Sample Response

2. Twitter Profile Posts

Input Parameters

Sample Response

3. Twitter Date Range Posts

Input Parameters

Sample Response

Method 2: Bright Data Twitter (X) Datasets

Data Collection Approaches

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages