Python projects grouped by domain, one folder per project.
Right now that is one project, and it is the interesting part of this repository, so it is described here rather than only linked:
Scrapes Indeed job listings through Cloudflare's managed challenge, turns each posting into a typed record with Claude, and writes it to CSV, XLSX and Postgres.
The pipeline is four files:
scraper.py fetch + extract Patchright browser, AgentQL element queries, pagination
extractor.py transform Claude tool-use with a forced JSON schema
store.py load pandas -> CSV/XLSX, SQLAlchemy upsert -> Postgres
main.py orchestrate CLI: --query, --location, --pages, --no-llm, --check
Getting past bot detection is a stack of levers, not one trick, and the
project's README ranks them by how much they actually matter — good IP first,
clever JS patching last. The specific finding: tf-playwright-stealth only
patches JavaScript-level tells, while Cloudflare also fingerprints Playwright's
Chrome DevTools Protocol leak — Runtime.enable, navigator.webdriver, the
HeadlessChrome user agent. That is why the first version hit "Additional
Verification Required". Patchright, a source-patched drop-in replacement
for Playwright, closes the CDP leak, which is the difference between the
scraper working and not.
On top of that: real Chrome over a persistent user-data directory, so the fingerprint stays warm and challenge cookies survive between runs; a headed browser with randomised human-like delays; and AgentQL, which matches elements by meaning instead of by CSS selector, so extraction survives Indeed changing its markup.
Full write-up, including the configuration table and the proxy fallback →
utils.robots_ok() checks robots.txt before fetching and honours
Crawl-delay. Delays are randomised, and the scraper stops early on empty
pages. The anti-bot work here is for learning and for sites you are authorised
to scrape.
agentql · patchright · playwright · anthropic · pandas · openpyxl ·
SQLAlchemy · psycopg2 · python-dotenv
Abdul Kabir Jawed — Full-Stack & Agentic AI Engineer, Karachi, Pakistan GitHub · LinkedIn