Detect drift. Preserve trust.
Name decided 2026-08-20 (see TD-004). All
user-visible naming reads from src/config/brand.ts.
Status: feature-complete and deployed. M0–M14 done: the pipeline is closed end to end, the interface is built and audited, and the submission package is assembled.
Live: https://scraper-three-azure.vercel.app/
One item is still open — the demo video is recorded but not yet published. The remaining checklist is in docs/SUBMISSION.md.
Most scraping systems reason like this:
HTTP 200 + selector matched + schema valid = SUCCESS
That inference is wrong, and it fails in the most expensive way possible.
Consider a product page. Before a redesign:
<div class="pricing">
<span class="price">₹899</span>
</div>After the redesign the site adds a struck-through list price, which also
carries the price class, placed first in the document:
<div class="price-stack">
<span class="price price--was">₹1,299</span>
<span class="price price--now" data-testid="product-price">₹899</span>
</div>.price still matches. The text is still valid currency. The schema still
validates. Every health check is green.
The number is now 44% wrong, and nothing anywhere reports a problem.
A scraper that crashes gets fixed within the hour. A scraper that lies quietly poisons every downstream decision until someone notices by accident.
Other systems tell you when a scraper broke. Signalhart tells you when your scraper is lying.
Bright Data runs the scraper. Signalhart decides whether to believe it:
- Detect — field-level plausibility, not null-checks. Temporal drift, type drift, cross-field coherence, selector ambiguity.
- Diagnose — distinguish the price changed from we started reading the wrong element.
- Recover — tiered repair, from deterministic through AI-assisted to escalation.
- Verify — every repair passes a validation gate. Nothing is adopted on trust.
This is the technically interesting part, and it is nearly free.
The extraction engine records more than a value. For every field it keeps which selector matched, that candidate's position in the preference list, how many elements matched, and the DOM ancestor chain.
Two of those settle the question:
value changed and fingerprint changed → the page was restructured value changed and fingerprint identical → the world changed
In the example above, .price goes from 1 match to 2, and its path moves
from div.pricing > span.price to div.price-stack > span.price--was.
The value alone is indistinguishable from a genuine 44% price rise. The
provenance is not ambiguous at all.
All of it is captured during the run that detected the problem, so diagnosis costs no extra page load.
supported source -> url-safety -> extraction plan -> Bright Data collector
-> interpreter (value + provenance + digest) -> normalisation
-> ingestion -> append-only history -> trust engine
|-> trusted
+-> suspicious -> diagnosis -> recovery -> verification gate
|
archive reading <-------------------------------------+
(how long, how much warning, did the repair hold?)
Full detail in docs/ARCHITECTURE.md.
Extraction plans are data, not code. The collector is a generic plan interpreter rather than a hardcoded scraper, so a repair is a new plan version in a table — versioned, diffable, shadow-testable, reversible in one query, and applied in seconds rather than the 15 minutes a code refactor takes. (TD-002)
History is append-only, and each stored field carries two hashes: one over the value, one over the provenance fingerprint (which selector won, how many elements it matched, where in the DOM it sat). Comparing both against the previous run is the whole of drift detection:
| value | fingerprint | reading |
|---|---|---|
| same | same | nothing happened |
| changed | same | the world changed — a real price move |
| changed | changed | the page was restructured — suspect the data |
| same | changed | extraction is degrading; the wrong value is coming |
See it happen, with no network call and no credit spent:
npm run ingest:demo # the hashes moving, on three scrapes
npm run validate:demo # run says ok; the value is questionable
npm run trust:demo # run says ok, value plausible, extraction DRIFTED
npm run history:demo # the warning, measured — and a repair that decayedtrust:demo is the one to run first. Seven daily scrapes of a fixture
page, judged three ways as each arrives:
| day | run.status | is the value believable? | has the extraction moved? |
|---|---|---|---|
| 4 | ok 3/3 | plausible | TRUSTED |
| 5 | ok 3/3 | plausible | DRIFTED |
| 6 | ok 3/3 | questionable | DRIFTED |
Day 5 is the whole product. The redesign has shipped, the price really is
₹899, every other layer in the stack is correct to report success — and
the number is already being read from a different element. Day 6 is the
same page one deploy later, where it becomes ₹1,299 and stays a clean
200 OK. (TD-008,
TD-013)
A real 899 → 849 price cut on day 2 is not flagged, which matters as much: a monitor that alerts on a sale teaches its operator to close the alert unread.
"Caught a day early" is only worth saying if it can be checked, so the archive measures it: the distance between the run where the extraction was flagged and the run where the value stopped being believable. On the flagship archive that is 1 run / 24 hours.
It is published with its denominators, and a warning of zero is reported
as zero. npm run history:demo runs the same repair — same engine, same
gate, same passed — on two pages, and the pooled median comes out at 0.5
runs rather than 24 hours because one of them had no warning window at
all. That is deliberate
(TD-024): a figure averaged only over the
comfortable cases would not survive being audited.
The same reading answers a question the verification gate structurally
cannot. The gate proves a repair read the right value from the page in
front of it, on the day it was accepted. Whether it held is a property
of the runs since — and on the second page it did not: the
data-testid the repair chose was renamed days later, extraction fell
through to its backup selector, and the price it reported was still
correct. regressed, not failed, and nothing else in the stack would
have noticed.
Full detail in docs/BRIGHT_DATA.md.
Hackathon rule 5 disqualifies off-the-shelf Scrapers Library scrapers. Ours is not a fixed-site scraper at all — it is a generic extraction-plan interpreter that:
- accepts an extraction plan as input rather than hardcoding selectors;
- returns provenance for every field — matched selector, candidate index, match count, DOM ancestor chain;
- returns a bounded structural digest of ranked replacement candidates for any field that failed, in the same run.
Points 2 and 3 are what make the trust engine possible, and no library scraper emits them.
Stated explicitly, because Scraper Studio ships its own Self-Healing tool and we are not claiming it:
| Concern | Owner |
|---|---|
| Fetching, unblocking, proxying, rendering | Bright Data |
| Scraper code repair (tier 3 escalation) | Bright Data Self-Healing |
| Knowing something is wrong | Signalhart |
| Real change vs extraction drift | Signalhart |
| Proposing and validating a plan-level repair | Signalhart |
| Refusing an unvalidated repair | Signalhart |
Their documentation states the gap we fill: self-healing "only activates when you request it, not when site changes occur."
Next.js 15 · React 19 · TypeScript (strict) · Tailwind 4 · Vitest · Bright Data Scraper Studio · Postgres/Supabase (migrated and RLS-locked; a JSON file store backs the offline demos) · Gemini (optional — the recovery engine runs its deterministic tier and escalates when no key is present, which is a working configuration rather than a degraded one)
npm install
cp .env.example .env.local # fill in credentials
npm run verify # typecheck + lint + test
npm run trust:demo # free: the silent break, caught a day early
npm run history:demo # free: the warning, measured
npm run seed:ui # free: give the UI real rows to render
npm run dev # / · /overview · /history · /collect · /method · /aboutThe free tier is 5,000 page loads/month, one credit each. SCRAPER_MODE
defaults to replay and the client refuses billable calls in that
mode. Record once, replay forever:
npm run bd:poc -- --live # one credit, records a cassette
npm run bd:poc # free from then on/collect runs the whole pipeline on demand: choose one of three
supported public sources, give it a page address, and watch the six
stages complete into four verdicts on a run that did not exist a moment
before.
It is a catalogue, not a URL box, and that is the design. The collector is a generic extraction-plan interpreter, which is what makes a repair a row rather than a deploy — and would also make an unguarded endpoint in front of it an open fetcher on somebody else's infrastructure. So a request carries a source id and an address, and nothing else it might carry is read:
| supplied by | what |
|---|---|
| the request | a source id from a fixed set, and an address |
src/config/sources.ts |
the host allowlist, the path scope, the plan |
| server environment | the API token and the collector id |
Host matching is equality against a written-down list, never
includes — which would accept books.toscrape.com.attacker.test — and
lib/url-safety runs first, because an allowlist alone accepts
https://books.toscrape.com@169.254.169.254/, where the allowed host is
in the credentials and the authority is the cloud metadata endpoint. The
same pure function validates in the browser for instant feedback and on
the server for real. Rationale and the full threat list:
TD-030.
Credit discipline is unchanged. In replay mode a page with no cassette produces the credit guard, and the screen reports the refusal rather than spending to avoid it — the example addresses are marked recorded · free or not recorded from a real cassette-key computation, so the constraint is legible before the button is pressed. The demo does not depend on this route: nothing else imports it, so a target site being down costs the reader one screen and nothing else.
It works with no credentials at all. Three recordings ship with the
repository in data/examples/cassettes/, and a replay composes no
request — so it needs the collector id (part of the cassette key) and no
API token. A deployment holding zero Bright Data secrets still serves the
Books to Scrape example targets, for free, forever. Those recordings hold
the parsed result and its provenance, not page markup, and come from a
sandbox published expressly for scraping practice.
See .env.example. Nothing is required to run the test
suite, and no new variable was added for live collection — it reads
the same BRIGHT_DATA_* and SCRAPER_MODE the scripts do.
data/examples/field-history.json— three stored observations of one field, including the silent break. Generated bynpm run ingest:demo.data/examples/validation.json— M3 judging the silent-break run that reportsstatus=ok. Generated bynpm run validate:demo.data/examples/trust.json— M4 on two consecutive runs: the warning while the value is still correct, and the break it predicted. Generated bynpm run trust:demo.data/examples/history.json— the archive reading over two stories: one repair that held, and the same repair decaying on a page that moved again. Generated bynpm run history:demo.data/examples/observation.json— a live collector run. Generated bynpm run bd:poc.data/examples/cassettes/— three promoted recordings, so/collectworks on a fresh clone with no credentials. See that directory's README for what a cassette holds and how to promote another.
732 tests, offline and credential-free. npm run verify runs typecheck,
lint and the suite; 38 further Postgres contract tests run against a real
database with npm run test:db.
The suite covers the nine routes as well as the services under them. A route test renders the real page component over a store seeded by the real pipeline, substituting only the composition root — which is the choice of database, not a collaborator — so nothing in it asserts that a mock returned its configured value (TD-027). The demo seed and the route tests share one scenario module, so a change that breaks a screen breaks a test rather than a rehearsal.
Two of them are worth naming. src/services/collection/target.test.ts
is written as attempts rather than as assertions — each case is an
address somebody could plausibly send, and the expectation is that it
does not resolve. run.test.ts drives the entire collection composition
over a stub speaking the Bright Data wire protocol as
docs/BRIGHT_DATA.md records it, with rows produced
by the real interpreter: wiring was the one part of the pipeline with no
test, and wiring is where a bug survives a green suite.
The fixtures in src/services/extraction/fixtures.ts cover all four
cells of the value/fingerprint quadrant: a baseline page, the same page
repriced (the world moved), a redesign that still reads the right number
(the extraction moved), a loud break where the field disappears, and a
silent break where extraction succeeds and is wrong. The silent one
is the product spec; the precursor is the one the product is proud of. A
sixth covers the case the archive exists for: the page a repair's chosen
handle disappears from, where a correctly verified repair quietly decays.
Claude Code was used as an AI coding assistant during development. Product direction, architecture decisions, implementation review, testing, verification, UI/UX decisions and final integration were directed and reviewed by the participant.
Three families are bundled as Latin-subset .woff2 under app/_fonts/,
all under the SIL Open Font License 1.1 and unmodified: Newsreader
(display, driven on its optical-size axis), Instrument Sans
(interface) and JetBrains Mono (selectors, hashes and every numeral
in a column). They are committed rather than fetched so a fresh clone
builds with no network and no credentials. Attribution and upstream
sources are in app/_fonts/LICENSE.md.
Nine routes, in two registers.
The ledger — /overview, /history, /collect, and the three
source screens.
Hairlines, verdict stamps, real columns, no pill anywhere and exactly one
filled surface in the whole product (a repair waiting for a decision).
State is carried in four channels that survive greyscale: position says
which judge spoke, rule weight says how severe, a dash pattern says the
judge declined to answer, and ink says what it said.
The chassis — /, /method, /about, plus the figures now sitting
above each ledger. A second type scale for pages that are posters rather
than records; five hand-drawn SVG figures (app/_components/figures.tsx)
that answer is anything wrong in under a second, where the table
answers what exactly happened on run 6; and a page ground tinted by the
worst verdict currently on screen, computed from the same
SourceTrust.level the stamps are drawn from.
The series — four more figures, all hand-drawn SVG
(app/_components/series.tsx, value-series.tsx), each answering a
question the tables could not:
- The value series. The number over time, drawn on top of the element it was read from. A line that holds flat across a band boundary is the warning this product exists to give — same price, new element — and it is invisible on the price chart every other monitor draws. Only for fields whose values are numbers, and never smoothed: a run that extracted nothing breaks the line rather than being crossed by it.
- The pulse strip. One dot per run beside each field, so a
driftedverdict can be read against whether the field has been steady for eleven runs or flapping all month. - The provenance chain. Page → element → selector → value → value verdict → extraction verdict, drawn severed at the link that failed.
- The repair funnel. The broken selector, the candidates the recorded evidence offered, the one chosen, and the verification gate — drawn as a gate, because a model may propose and may not publish.
There is no chart library, no component library, no icon set and no CSS framework in the interface. There is also no health score anywhere — the four judges are kept in four vocabularies precisely so they cannot be averaged, and a figure that averaged them would undo the judging stack in one SVG.
DESIGN-SPEC.md describes the interface rather than the pipeline: the
two type scales, the OKLCH palette with its computed contrast table, the
motion tokens, the figures and the reasoning behind each. The decision
records are TD-026 (the ledger), TD-028 (the chassis) and TD-029 (the
series, and the one condition under which a value chart is honest) in
docs/TECHNICAL_DECISIONS.md.
docs/SUBMISSION.md is the package: the compliance checklist with the file or command that proves each line, and the demo inventory. The repository and the deployment are live; the one item still open is publishing the recorded demo video. docs/FINAL_AUDIT.md is the audit behind it — 22 checks, every one executed rather than reviewed, with the two findings written up and the documentation claims they corrected.
| Rule | Status |
|---|---|
| Custom Scraper Studio scraper | ✅ plumbline-plan-interpreter / c_msxh3vh52lt427jfje, Bright Data run j_msykl0h32fwjl9m4eo — response committed at data/examples/collector-row.json |
| Not a Scrapers Library scraper | ✅ generic plan interpreter |
| Public data only | ✅ enforced in code — SSRF-aware URL validation, and interactive collection is restricted to a server-side catalogue of three public sites (TD-030); no auth/paywall targets |
| No private or personal data | ✅ product, pricing and bibliographic fields only |
| Work done after start | ✅ repository initialised 2026-08-17 |
| Public repository | ✅ https://github.com/dhananjay-123/scraper — live at https://scraper-three-azure.vercel.app/ |
| README | ✅ |
| Example structured output | ✅ data/examples/, nine files, one from the live run |
| Demo video | ✅ recorded — script in docs/DEMO_SCRIPT.md |
| Bright Data explanation | ✅ docs/BRIGHT_DATA.md |
| AI disclosure | ✅ above |