Scryshot

Scryshot is a desktop application for crawling websites and building browsable offline mirrors.

Unlike simpler download tools that work best on static sites, Scryshot is designed to deal with modern JavaScript-heavy pages, builder-generated layouts, client-side navigation, and interactive widgets. Its goal is to produce local copies that are not only saved, but still usable when opened outside the live web.

Why Scryshot

Many mirroring tools are good at downloading files, but struggle when a website:

• renders content through JavaScript
• uses buttons instead of normal links
• loads pages through AJAX or SPA-style navigation
• rewrites the DOM after load
• depends on runtime-generated assets and routes

Scryshot focuses on that gap. It combines browser automation, interactive discovery, asset rewriting, and offline-safe page generation to preserve as much navigability as possible in the mirrored output.

What It Does

• Crawls a site from a configurable starting URL
• Discovers pages through normal links and interactive controls
• Mirrors HTML pages and static assets to a local folder
• Rewrites asset references for local browsing
• Tries to convert JavaScript-driven navigation into local mirror-safe links
• Supports different interactive discovery profiles for speed vs coverage
• Provides desktop controls for project setup, test runs, history cleanup, and output browsing

Core Capabilities

Interactive Discovery

Scryshot does not rely only on <a href> links. It can probe interactive UI elements such as buttons, custom widgets, and JavaScript-driven controls to discover pages that would otherwise be invisible to a traditional crawler.

Offline-Friendly Mirror Generation

When a page is captured, Scryshot rewrites mirrored output so that it can be browsed locally more reliably. This includes local asset mapping, link normalization, redirect handling, and mirror-oriented page generation.

Better Support for JS-Heavy Sites

Scryshot uses Playwright and Crawlee to inspect the live page, observe runtime behavior, and capture signals that classic HTML-only crawlers miss. This makes it better suited to builder-based or SPA-like websites.

Desktop Workflow

The application includes a desktop UI for:

• project management
• crawl execution
• test runs against a single page
• progress monitoring
• mirror launching and folder browsing
• run history inspection

How It Works

At a high level, Scryshot follows this flow:

1. Open a page in a real browser context
2. Wait for the page to stabilize enough for analysis
3. Discover candidate links and interactive controls
4. Probe interactive elements on a safe page clone
5. Infer navigable destinations from browser, DOM, and network signals
6. Capture a mirror-ready snapshot of the page
7. Rewrite links and assets for local browsing
8. Save the mirrored output and enqueue new pages

Tech Stack

• Electron
• React
• TypeScript
• Playwright
• Crawlee
• better-sqlite3
• Tailwind CSS
• i18next

Repository Structure

• app/ - main Electron application
• tests/ - test project and automated verification
• scripts/ - support scripts
• CHANGELOG.md - project changelog
• Scryshot.slnx - Visual Studio solution entry point

Getting Started

Requirements

• Node.js
• npm
• Windows is currently the primary development environment

Install

cd app
npm install

cd tests
npm install

Run the App

cd app
npm run dev

Run Type Checking

cd app
npm run typecheck

Run Tests

cd tests
npm test

Current Status

Scryshot is actively evolving and already supports:

• project-based crawling
• offline mirror output
• interactive discovery modes
• test-run workflows for single-page debugging
• improving support for JavaScript-driven navigation

The project is especially focused on improving mirror quality for sites that rely heavily on runtime rendering and non-semantic navigation.

Changelog

See CHANGELOG.md for notable changes.

License

MIT