Extract clean(er), readable text from web pages via trafilatura.
Earlier versions of this project used the Postlight Parser, which required Node.js and shelling out to its command-line driver, plus html2text for the Markdown/plain-text conversions. Both have been replaced by trafilatura, a well-maintained Python library that consistently tops content-extraction benchmarks and emits HTML, Markdown, and plain-text natively. Everything now runs in a single Python process with a single dependency.
Clone this repository and install the dependencies with uv:
$ uv sync
Or with a classic virtual environment:
$ python3 -m venv .venv
$ source .venv/bin/activate
(reader) $ pip install -r requirements.txt
(reader) $ ./reader.py -h
usage: reader.py [-h] [-f {json,html,md,txt}] [-w BODY_WIDTH] [-t FORMAT] source
Get a cleaner version of a web page for reading purposes. This script fetches a URL (or reads
local HTML) and extracts the main content and metadata via [trafilatura](https://trafilatura.readthedocs.io/), outputting the document as JSON, Markdown, plain-text, or
HTML.
positional arguments:
source URL to fetch and parse, or path to a local HTML file (use "-" to read
HTML from stdin)
options:
-h, --help show this help message and exit
-f {json,html,md,txt}, --format {json,html,md,txt}
output format (default: json)
-w BODY_WIDTH, --body-width BODY_WIDTH
character offset at which to hard-wrap lines of markdown and plain-text
content (default: None)
-t FORMAT, --table-format FORMAT
tabulate format for data tables in plain-text content (default: simple)
When wrapping markdown, lines whose markup would break if split across lines (headings, table rows, horizontal rules, and fenced code blocks) are left intact, and long tokens such as URLs are never split.
Layout tables (common on older, table-based sites) are unwrapped into ordinary paragraphs so their contents read naturally, while genuine data tables are preserved; decorative tables with no text (image/spacer scaffolding) are dropped. In plain-text output, data tables are rendered as aligned text via tabulate, in any format tabulate supports (-t/--table-format, default simple):
City Population
----------- ------------
Springfield 30,720
Shelbyville 12,654
Rendered tables are never disturbed by -w line-wrapping, regardless of the chosen table format.
The source can be a URL (fetched by trafilatura), a local HTML file, or - to read HTML from stdin — so you can also feed it pages saved locally or fetched by other tools (curl, a headless browser, etc.).
The default output is JSON containing trafilatura's extracted metadata alongside the content in three forms: HTML (.content.html), Markdown (.content.markdown), and plain-text (.content.text):
(reader) $ ./reader.py https://www.paulgraham.com/greatwork.html | jq .
{
"title": "How to Do Great Work",
"author": null,
"url": "https://www.paulgraham.com/greatwork.html",
"hostname": "paulgraham.com",
"description": null,
"sitename": "paulgraham.com",
"date": "2023-01-01",
"categories": [],
"tags": [],
"fingerprint": null,
"id": null,
"license": null,
"language": null,
"image": null,
"pagetype": null,
"filedate": "2026-08-19",
"content": {
"html": "<html>...</html>",
"markdown": "July 2023 If you collected lists of techniques for doing great work...",
"text": "July 2023 If you collected lists of techniques for doing great work..."
},
"word_count": 11807
}
The extracted HTML content is accessible from .content.html, or directly with --format=html:
(reader) $ ./reader.py https://www.paulgraham.com/greatwork.html -f html
As a convenience, the -f/--format option can output the whole document as Markdown, including some of the human-relevant metadata:
(reader) $ ./reader.py https://www.paulgraham.com/greatwork.html --format=md
---
title: "How to Do Great Work"
url: "https://www.paulgraham.com/greatwork.html"
sitename: "paulgraham.com"
date: "2023-01-01"
words: 11874
---
# [How to Do Great Work](https://www.paulgraham.com/greatwork.html)
July 2023
If you collected lists of techniques for doing great work in a lot of
different fields, what would the intersection look like? I decided to find
out by making it.
...
The front matter includes the human-relevant metadata fields that are present (empty fields are omitted), quoted as YAML-safe scalars.
Similarly, the whole document can be formatted as plain-text:
(reader) $ ./reader.py https://www.paulgraham.com/greatwork.html --format=txt -w 80
title: How to Do Great Work
url: https://www.paulgraham.com/greatwork.html
sitename: paulgraham.com
date: 2023-01-01
words: 11874
July 2023
If you collected lists of techniques for doing great work in a lot of different
fields, what would the intersection look like? I decided to find out by making
it.
...
One use case for this script is to convert content from the web to a format that is suitable for reading in your terminal. Here's a short shell pipeline to extract the content and feed the converted plain-text to your $PAGER of choice for easy reading:
#!/bin/sh
# Read a web page as clean plain text in $PAGER.
# Usage: newspaper.sh <url>
set -eu
"path/to/reader.py" "$1" -w 80 -f txt | "${PAGER:-less}"