Polite web-fetch + parse toolkit for general-purpose crawling and content extraction

---

Install

pnpm add @sdk-e/crawler

Quick Start

import { crawl } from "@sdk-e/crawler";

const result = await crawl({
  url: "https://example.com",
  extractAs: "markdown",
});

console.log(result.content.slice(0, 200));

Documentation

• Usage Guide — Core concepts, extraction modes, recursive crawling, hooks
• API Reference — Full API types and options
• Examples — Code examples and patterns
• Advanced Topics — Proxy rotation, rate limiting, retries, JS rendering, robots.txt

Contributing

See CONTRIBUTING.md for development setup and guidelines.

License

MIT — SDK Enterprises