BookStores Datasets

Automated book price tracking for Turkish online bookstores — scrape → diff → publish → visualize.

bookdata is a clean, testable web-scraping pipeline that tracks book prices across Turkish bookstores. It scrapes categories and products asynchronously, normalizes the data, computes price changes, and maintains an append-only CSV dataset that lives on Kaggle. A scheduled CI job renders an interactive Plotly dashboard and publishes it to GitHub Pages — fully hands-free.

Highlights

• ⚡ Async scraping — httpx-based concurrent fetcher with retries, backoff, per-host rate limiting and robots.txt support
• 🧹 Port-adapter architecture — clean separation of concerns, mockable and fully unit-tested
• 🧾 Structured-data extraction — JSON-LD (Product/Book/ItemList) parsed first, DOM used as fallback; ISBN, currency and stock availability captured
• 🔗 LLM-free product matching — cross-store matching engine: ISBN → publisher bucket → RapidFuzz title+author fuzzy, with MATCH / REVIEW / DIFFERENT confidence bands
• 📈 Price-diff engine — vectorized pandas diff so the dataset only grows with real price changes
• 🗂️ Centralized ignore rules — one global ignore_categories.txt applies to every store
• 🤖 Zero-ops automation — GitHub Actions runs on a CRON schedule and on demand
• 📊 Interactive dashboard — Plotly charts in a single small HTML file (CDN-hosted)
• 🔓 Open source by default — no credentials or personal IDs in the codebase; all configured via environment

Architecture

                 ┌────────────────────────────── GitHub Actions ─────────────────────────────┐
                 │   schedule / workflow_dispatch                                             │
                 ▼                                                                           │
        ┌──────────────┐    ┌──────────────┐   ┌───────────────┐   ┌───────────────────┐      │
        │ fetch_        │    │ pipeline     │   │ dataset       │   │ Kaggle Publisher  │      │
        │ categories    ├───▶│ filter →     ├──▶│ store (CSV)   ├──▶│ (append-only)     │      │
        │ fetch_        │    │ standardize  │   │ price diff    │   └───────────────────┘      │
        │ products      │    │ → merge      │   └───────┬───────┘                             │
        └──────────────┘    └──────────────┘           │                                      │
                                                       ▼                                      │
                                        ┌──────────────────────┐   ┌───────────────────┐       │
                                        │ analyze + dashboard  │──▶│ GitHub Pages       │──────┘
                                        │ (Plotly HTML)        │   │ (public dashboard) │
                                        └──────────────────────┘   └───────────────────┘

Project structure

src/bookdata/
├── cli.py              # Typer CLI (scrape / categories / report / publish / inspect / stores)
├── config.py           # Settings from environment variables
├── logging_setup.py    # Central log configuration
├── models.py           # Category / Product data models (isbn, currency, availability)
├── errors.py           # Typed fetch/robots error hierarchy
├── matching.py         # Cross-store matching engine (ISBN → publisher → fuzzy title+author)
├── analyze.py          # Price changes, weekly trends, summary stats
├── dashboard.py        # Plotly dashboard renderer (single-file HTML)
├── pipeline/
│   ├── runner.py       # Orchestrates the scrape flow + store registry
│   ├── extract.py      # JSON-LD product extraction ("structured data first")
│   ├── filter.py       # Applies global ignore rules to categories
│   ├── products.py     # Concurrent product collection
│   ├── standardize.py  # Normalizes scraped rows into the dataset schema
│   └── merge.py        # Computes price diffs against the last known price
└── adapters/
    ├── http.py         # Async HTTP client (retries, rate limiting, robots.txt)
    ├── kaggle.py       # Kaggle dataset publisher
    ├── storage.py      # CSV dataset store
    └── stores/
        ├── base.py     # StorePort abstract interface (domain-based resolution)
        ├── bkm.py      # BKM Kitap adapter
        └── kitapyurdu.py  # Kitap Yurdu adapter

Quick start

Requires uv.

uv sync                 # install runtime dependencies
uv sync --extra report  # + plotly for the dashboard

Usage

Every command runs through the bookdata CLI:

Command	Description
bookdata scrape <store>	Category → filter → products → standardize → price diff → append to dataset
bookdata categories <store>	List a store's categories (ignore rules applied)
bookdata inspect <url>	Diagnose any URL: adapter match, HTTP status, JSON-LD + detected fields
bookdata match	Match the same book across stores (ISBN → publisher → fuzzy); REVIEW rows → CSV
bookdata report	Generate the interactive dashboard from all datasets
bookdata publish <store>	Upload the dataset to Kaggle
bookdata stores	List registered store adapters

Examples:

uv run bookdata scrape bkm
uv run bookdata scrape kitapyurdu
uv run bookdata categories bkm -n 20
uv run bookdata inspect https://www.bkmkitap.com/kitap
uv run bookdata match --match-threshold 0.95 --review-threshold 0.75
uv run bookdata report -o Report/index.html
uv run bookdata publish bkm

KY / BKM shortcuts are accepted for kitapyurdu / bkm.

Cross-store matching

bookdata match groups the same book across stores without an LLM:

1. ISBN equal → MATCH (deterministic)
2. No ISBN → same publisher (Brand) bucket → title (Model) fuzzy similarity + author fuzzy confirmation
3. Confidence bands: ≥ 0.95 → MATCH, 0.75–0.95 → REVIEW, below → DIFFERENT
4. Same title, different publisher → edition variant → REVIEW (human decides)

Author participates fuzzily too — name order/spelling varies across stores ("Rowling, J.K." vs "J. K. Rowling"). Ambiguous pairs are exported to Match/review_<date>.csv for manual review, and confident cross-store matches feed the dashboard's store price-comparison chart.

Ignore rules

ignore_categories.txt holds one pattern per line (blank lines and # comments are ignored). Any category whose name or URL contains a pattern is skipped — for every store. This is the single source of truth for "not a book" categories (stationery, toys, accessories, music, film…). Override the file location with BOOKDATA_IGNORE_FILE.

Configuration

All settings are environment-driven — nothing is hardcoded:

Variable	Default	Purpose
BOOKDATA_STORE	bkm	Default store for commands that need one
BOOKDATA_DATA_DIR	Data	Where *_Datasets.csv files live
BOOKDATA_LOG_DIR	logs	Log output directory
BOOKDATA_LOG_LEVEL	INFO	Log verbosity
BOOKDATA_IGNORE_FILE	ignore_categories.txt	Global category ignore patterns
BOOKDATA_CONCURRENCY	12	Parallel HTTP requests
BOOKDATA_TIMEOUT	20	Request timeout (seconds)
BOOKDATA_RETRY_ATTEMPTS	3	Retries per request
BOOKDATA_MIN_INTERVAL	0.2	Min seconds between requests per host
BOOKDATA_MAX_PAGES	50	Max pagination pages per category
BOOKDATA_RESPECT_ROBOTS	false	Respect robots.txt (Disallow) rules before fetching
BOOKDATA_MATCH_THRESHOLD	0.95	Title similarity for MATCH
BOOKDATA_REVIEW_THRESHOLD	0.75	Title similarity floor for REVIEW
BOOKDATA_AUTHOR_MATCH_THRESHOLD	0.85	Author similarity needed to confirm a MATCH
BOOKDATA_KAGGLE_DATASET	—	Kaggle dataset id (owner/slug) for publishing
KAGGLE_USERNAME / KAGGLE_KEY	—	Kaggle API credentials

Automation

GitHub Actions drives everything — every workflow runs on a CRON schedule and can be triggered manually from the Actions tab.

Workflow	Trigger	What it does
scrape.yml	CRON (2×/day) + manual	Scrapes both stores (matrix: bkm / kitapyurdu), updates the Kaggle datasets, commits logs. Manual run lets you pick all / bkm / kitapyurdu
generate_report.yml	CRON (weekly) + manual	Pulls datasets from Kaggle, renders the dashboard, deploys it to GitHub Pages
ci.yml	every push + PR + manual	Runs ruff check, ruff format --check, and pytest on every commit

Scrape workflows download the latest dataset from Kaggle, compute price diffs against it, re-upload the updated dataset, and commit fresh logs — so the repository always shows recent activity and the data stays authoritative on Kaggle.

Setup

1. Add repository secrets: KAGGLE_USERNAME, KAGGLE_KEY.
2. Add repository variables: KAGGLE_DATASET_BKM, KAGGLE_DATASET_KY (e.g. owner/your-dataset).
3. Enable GitHub Pages (source: GitHub Actions) for the dashboard deployment.

Dashboard features

The generated dashboard is a single self-contained HTML page:

• Summary cards — record/product counts, last scrape date, rising/falling stats
• Store comparison — average price change per store
• Category analysis — change distribution across categories and stores
• Top movers — 10 most increased / 10 most decreased titles
• Weekly trend — average price over time
• Store comparison — matched books (via the matching engine) with per-store prices, top price gaps

Plotly loads from a CDN, keeping the HTML tiny while staying fully interactive.

Adding a store

1. Create an adapter in src/bookdata/adapters/stores/ implementing StorePort (use bkm.py or kitapyurdu.py as a template).
2. Register it in STORE_REGISTRY in src/bookdata/pipeline/runner.py.
3. Add any store-specific patterns to ignore_categories.txt if needed.

Development

uv run pytest                 # run tests
uv run ruff check src tests   # lint
uv run ruff format src tests  # format

The test suite covers filtering, standardization, merging, storage, analysis and dashboard rendering — and the CI workflow runs it on every push.

License

MIT © BookStores Datasets contributors