Release v2.7.0 — deep attack-surface platform (passive ASM) - #2
Merged
Conversation
The Shannon-entropy floor (MIN_ENTROPY_THRESHOLD=3.5) was applied
uniformly to every detector. Structural/provider patterns (AKIA…,
ghp_…, sk_live_…, PEM, fixed-format hex/UUID tokens) are high-precision
by shape, but a genuinely low-entropy yet well-formed live key — e.g. an
AWS key ID at ~3.27 bits/char — was silently dropped before it ever
reached AI validation. That is a false negative, the worst failure mode
for a secret scanner.
Entropy handling is now class-aware:
• generic keyword=value catch-all keeps the full 3.5 bar (needs a
randomness signal to stay quiet);
• structural detectors clear only a low anti-degenerate floor
(MIN_STRUCTURAL_ENTROPY=2.5) that still rejects obvious junk such as
"AKIAAAAAAAAAAAAAAAAA" (~0.6 bits) while catching modest-entropy
real keys.
SecretPattern gains an entropy_gated flag (default False; True only on
the generic pattern). Adds MIN_STRUCTURAL_ENTROPY config + .env.example
docs. New tests lock in both directions (low-entropy structural key
detected; degenerate junk still rejected; generic gate unchanged).
Precision/recall stays 1.000/1.000; suite 187 -> 191, ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
…eview
Bucketing sent a finding to Confirmed only when is_valid AND confidence
>= threshold, and to manual review only when AI validation was
*unavailable* (sentinel). A structural/provider match the AI *actively*
rejected with a real confidence matched neither bucket and was silently
dropped — so a genuine live key the AI merely under-called on (e.g. a
page that gave it no surrounding context) vanished with no trace.
Add classify_validated() returning confirmed | review | drop:
• AI unavailable (sentinel) -> review (unchanged)
• is_valid and confidence >= min -> confirmed (unchanged)
• structural match NOT confidently dismissed by the AI -> review
(new: shape-anchored keys get human eyes instead of deletion)
• everything else, incl. any generic catch-all 'no' -> drop
(preserves aggressive generic filtering / 'no false positives in
Confirmed')
Report: broaden the manual-review section heading and the SARIF note to
cover both reasons; the per-row Note already carries the AI's reasoning.
Adds 6 classifier tests. Suite 191 -> 197, ruff clean, precision/recall
unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
…ASM slice 1)
First discovery layer of the passive attack-surface pipeline: given a
domain, expand it into its known subdomain surface from Certificate
Transparency (crt.sh) using public data only — no packet is ever sent to
the target, so it runs before an engagement is signed.
New backend/recon.py:
• extract_registrable_domain() — normalise URL / host[:port] / bare
domain to the registrable domain, with a two-label public-suffix
table (incl. .co.uk, .com.au, and Bangladesh .*.bd). Returns None for
IP literals and single-label inputs (nothing to enumerate).
• parse_crtsh_json() — pure, deterministic parse of a crt.sh response
into sorted, deduped, in-scope hostnames; strips wildcards, rejects
emails and look-alike suffixes (evil.example.com.attacker.net).
• enumerate_subdomains_ct() — async CT query that FAILS CLOSED: any
network/parse error yields an empty result + error string, never an
exception that could abort a scan.
CLI: `python cli.py <domain> --subdomains` prints discovered subdomains
as JSON (accepts a bare domain since it never fetches the target).
Config: CRTSH_URL, SUBDOMAIN_ENUM_TIMEOUT, MAX_SUBDOMAINS (+ .env.example).
18 new tests (parsing, normalisation, fail-closed, limit); network mocked.
Suite 197 -> 215, ruff clean. Passive/authorized posture preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Live Pi testing exposed crt.sh returning 502 / timing out — a single CT
source is not production-grade, and the timeout surfaced as a blank error
string. Make enumeration reliable the way subfinder/amass do: query
several passive sources with retries and merge.
• Second CT source: Certspotter (/v1/issuances), parsed by the new pure
parse_certspotter_json(). Still passive — a third-party CT aggregator,
never the target.
• enumerate_subdomains() now queries crt.sh + Certspotter, unions and
dedupes. `error` is set only if EVERY source fails, so one flaky
source no longer zeroes out a good result; `sources` lists what worked.
• _get_with_retries(): backoff retries on transient statuses
(429/500/502/503/504) and timeouts, and every error names its type
(fixes the blank "crt.sh request failed: " seen on the Pi).
• SubdomainResult.source -> sources (list). CLI prints the source list.
• Config: CERTSPOTTER_URL, SUBDOMAIN_ENUM_RETRIES.
enumerate_subdomains_ct kept as a backward-compatible alias.
Tests: Certspotter parsing, both-source merge, one-source-down still OK,
retry-then-succeed, all-fail names each source, limit. Sleeps neutralised
so retry paths run instantly. Suite 215 -> 219, ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
…ASM slice 2)
Closes the discovery-to-findings loop. backend/orchestrator.py takes a
single domain and: enumerates its subdomains (passive CT), probes which
hosts are live, runs the existing passive secret+posture scan against
each, and aggregates everything into one DeepScanResult / combined report.
This is the "deepest scan, not just the surface" step — a surface scan of
one host becomes a full-domain assessment.
• run_deep_scan(): enumerate -> probe -> scan-each -> aggregate, with all
collaborators injected (enumerate/scan/client) so it is unit-testable
with no network. IP / non-enumerable targets degrade to a single scan.
• SSRF guard: a discovered host that resolves to a private/internal
address is skipped (recorded, not scanned) unless ALLOW_PRIVATE_TARGETS
— a wildcard/misissued cert can name internal hosts, and the scanner
itself has no built-in guard, so orchestration must enforce it.
• Resilience: MAX_TARGETS cap, concurrent liveness probing
(PROBE_CONCURRENCY), per-host error isolation (one host raising is
recorded and the run continues).
• report.generate_deep_scan_html(): one client deliverable across the
domain — subdomain surface, live hosts, per-host confirmed/needs-
review/posture with an EXPOSURE/CLEAN verdict.
• CLI: `python cli.py <domain> --deep-scan -o report.html`
(+ --max-targets). Config: MAX_TARGETS, PROBE_CONCURRENCY, PROBE_TIMEOUT.
10 orchestrator tests (probe filtering, aggregation, dead-host exclusion,
max-targets, per-host error isolation, IP fallback, SSRF-guard skip);
network fully mocked. Suite 219 -> 229, ruff clean. Passive posture intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
The passive alternative to directory/content brute-forcing: instead of
guessing paths against the target (active, noisy, needs an RoE), recover
the URLs it has already exposed from public web archives — no request
touches the target.
backend/historical.py:
• Two sources merged: Wayback Machine CDX (matchType=domain, so apex +
every subdomain) and CommonCrawl (two-hop: newest index via collinfo,
then its CDX API). Same multi-source + backoff-retry + fail-closed
design as the subdomain layer; error is set only if all sources fail.
• Pure parsers parse_wayback_cdx() / parse_commoncrawl_jsonl() with
in-scope host filtering (host == domain or *.domain).
• HistoricalResult exposes urls, a unique-path view (the "hidden
directories"), and js_urls() — the highest-value scan seeds, since
bundled JS is where secrets leak most.
• CLI: `python cli.py <domain> --historical`. ENABLE_COMMONCRAWL toggle
+ timeout/retry/limit config.
Surfaces forgotten endpoints and stale bundles a live crawl never links
to, and gives the testphp-style empty-crawl case real coverage. 10 new
tests (parsers, merge, one-source-down, retry, all-fail, limit); network
mocked. Suite 229 -> 239, ruff clean. Passive posture intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Turns discovery into findings — the payoff of the passive discovery
chain. A secret sitting in a forgotten/archived JS bundle that no live
page links to now gets fetched and confirmed.
• scanner.run_scan() gains seed_urls: externally-supplied asset URLs are
fetched (reusing fetch_url) and added to the scan set after the live
crawl, deduped against it and capped by MAX_SEED_URLS. Backward-
compatible (defaults to none).
• orchestrator.run_deep_scan(include_historical=True) recovers the
domain's historical URLs once (Wayback/CommonCrawl) and routes each
host its own archived JS bundles as seeds. discover_historical_fn is
injected for testing; DeepScanResult carries a historical_urls count.
• CLI: `--deep-scan --with-historical`. Combined report gains a
"Historical URLs" stat; deep-scan summary line reports the count.
Tests: run_scan seed fetch + dedup + no-seed no-op (collaborators mocked);
orchestrator routes per-host seeds and reports the count, and passes none
when not requested. Suite 239 -> 244, ruff clean. Passive posture intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Live testphp.vulnweb.com --deep-scan returned 0 hosts: the target was
reduced to the registrable domain vulnweb.com, CT enumeration returned
nothing that run, the bare apex was not live, and the host the user
actually typed (testphp.vulnweb.com) was never scanned.
• Always include the originally-specified host as the first scan
candidate — enumeration can never drop the target the caller named.
• When include_historical, harvest hostnames from the recovered archive
URLs and add them as candidates too (not just JS seeds). Historical
now enriches HOST discovery as well as paths, so a flaky/empty CT
source no longer zeroes out a run — the archive's testphp/testasp/
rest/www hosts get scanned. Candidates are filtered in-scope and
deduped; historical discovery moved ahead of host selection.
Tests: specified host scanned despite empty CT enumeration; archive-only
hostnames become scanned candidates. Suite 244 -> 246, ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
The combined report summarised each host's confirmed/needs-review as
numbers but never listed the findings, so a run reporting "22 needs-
review" gave the operator no way to actually review them.
• DeepScanResult.to_dict() now aggregates confirmed_findings and
needs_review_findings across all hosts, tagging each with its host of
origin (_host) for provenance.
• generate_deep_scan_html() renders two new detail tables — "Confirmed
Findings (all hosts)" and "Flagged for Manual Review (all hosts)" —
with host, severity, type, location and reasoning, capped at 250 rows
each. Adds severity-badge styling.
Now the deliverable is complete: the needs-review candidates a deep scan
surfaces are inspectable, not just counted. Suite 246 -> 247, ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Live testing surfaced 22 needs-review items that were really ~2 tokens: web archives store one file under many cache-buster query strings (api_data.js?v=1596328976710, ?v=1596328981612, …), and the historical seed layer fed all 11 variants as separate scan targets, so the same secrets were re-found in each. HistoricalResult.js_urls() now dedupes by (host, path), keeping one URL per unique JS file. Fewer redundant fetches, and a finding count that reflects distinct files instead of cache-bust noise. (The 22 were also all "AI validation skipped" — the CLI run lacked GEMINI_API_KEY, so every structural match correctly routed to manual review rather than being dropped; and the tokens are apiDoc documentation examples the AI triages when enabled. This dedupe fixes the multiplier.) Test: 11 api_data.js?v=… + api_project.js collapse to 2 seeds. Suite 247 -> 248, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
The whole deep-ASM pipeline is now drivable from the web UI, not just the
CLI.
Backend:
• POST /api/deep-scans — runs orchestrator.run_deep_scan as a streaming
background task (mirrors the single-scan endpoint): returns a scan_id,
per-host progress flows over the existing /ws/logs/{scan_id} socket.
DeepScanRequest validates + caps inputs.
• run_deep_scan gained a broadcast hook: it emits enumerate / historical
/ probe / per-host / deep_scan_complete events and forwards the socket
to each host's run_scan, so the live terminal shows real progress.
• Report endpoint serves the combined multi-target report
(generate_deep_scan_html) for results flagged deep_scan.
Frontend:
• A DEEP toggle turns the target box into a whole-domain scan (bare
domain in → enumerate + historical + probe + scan every live host).
• The WS handler tracks deep mode and finalises on deep_scan_complete
instead of the first per-host scan_complete; a new case reports the
aggregate totals.
Tests: API route registered, auth required, input caps, scan starts and
returns a scan_id (orchestrator + save mocked, offline). JS syntax-checked.
Suite 248 -> 252, ruff clean. Passive posture intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
… (slices 5 & 4)
backend/surface.py mines already-fetched assets (passive — no new target
requests beyond the deeper-crawl fetches) for two things:
Slice 5 — extract_endpoints(): URLs/paths referenced in JS/HTML
(fetch()/axios targets, /api/… routes) resolved to absolute URLs. A
live crawl never links to these; run_scan now fetches the same-site
.js endpoints ONE LEVEL DEEPER so code-referenced bundles get
secret-scanned too.
Slice 4 — extract_referenced_hosts(): the external hosts an asset talks
to (CDNs, APIs, third parties), aggregated across a scan into the
associated-asset graph (third-party / connected-infra surface).
Wiring:
• run_scan returns discovered_endpoints + associated_hosts; the surface
step runs after asset collection, bounded by MAX_ENDPOINT_SEEDS /
MAX_DISCOVERED_ENDPOINTS, behind EXTRACT_SURFACE.
• DeepScanResult aggregates associated_hosts across hosts.
• Single-target and deep reports gain an "Attack Surface Intelligence"
section (associated hosts + endpoints referenced in code).
All extractor regexes are bounded (no nested quantifiers) — verified
ReDoS-safe on pathological input (0.006s). 8 surface tests + a run_scan
integration test (endpoints/hosts populated, deeper .js fetched). Suite
252 -> 260, ruff clean. Passive posture intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
My earlier `git add -A` swept in backend/data/secretnode.db-journal (the scan database's rollback journal, created when the API tests init the DB). .gitignore already excluded *.db/*.db-wal/*.db-shm but not *.db-journal, so it slipped through. A scan database can hold real findings and must never be committed. Untrack the journal and add the missing ignore rule. (The committed file was the sandbox test DB — synthetic data only — but the rule gap is the real fix.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
`make run` failed on the Pi with "uvicorn: not found" (Error 127): the target invoked the bare `uvicorn` console script, which isn't on PATH even though the module is importable (python cli.py works fine). Use `python -m uvicorn` so it runs under the same interpreter as the CLI, and switch --loop uvloop -> auto so a missing uvloop no longer hard-fails the start (auto uses uvloop when present, else asyncio). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
The highest-severity passive finding a deep scan can surface: a subdomain
whose DNS still points (CNAME) at a de-provisioned third-party service —
an unclaimed S3 bucket, deleted GitHub Pages site, removed Heroku app —
can be re-claimed by an attacker and used to serve content from a hostname
your users trust.
backend/takeover.py:
• Curated fingerprint catalogue (16 services) — GitHub Pages, S3,
Heroku, Netlify, Shopify, Fastly, Zendesk, Surge, Bitbucket, Ghost,
Pantheon, Tumblr, Wordpress, Readme, Cargo, Webflow.
• High precision: check_takeover() flags only on a service's SPECIFIC
unclaimed-resource body signature (generic 404s excluded); when
CNAMEs resolve, one must point at the service (guards look-alike
strings on unrelated hosts). Body-signature-alone flags only when
CNAMEs are unresolvable.
• Stdlib DNS (socket.gethostbyname_ex), concurrent scan pass, fails
closed per host.
Orchestrator runs a concurrent takeover pass over every in-scope host
(a takeover target is often not a "normal" live host); DeepScanResult
carries takeover_findings + a takeover_risks total; the combined report
gains a prominent "Subdomain Takeover Risks" section + KPI.
Tests: fingerprint matches (GitHub/S3/Fastly), precision guards
(wrong-service CNAME, clean page), detect/scan flows, orchestrator
integration. Suite 260 -> 269, ruff clean. Passive posture intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
A domain deep scan scanned hosts one at a time; a 10-host domain took ~10×
a single scan. Scan hosts in parallel with a bounded semaphore
(HOST_SCAN_CONCURRENCY, default 3 — each host scan is itself concurrent,
so the bound stays Pi-friendly).
• Results collected in target order; per-host error isolation preserved
(one host failing is recorded and the run continues).
• Progress emitted as [k/N] host — done/error events over the WebSocket.
Test proves genuine parallelism (max in-flight > 1) that never exceeds
the semaphore bound, and that all hosts are still scanned. Suite
269 -> 270, ruff clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Promote the deep-ASM epic to a release: bump version to 2.7.0 across pyproject, the report generator's fallback, and the dashboard, and finalize the changelog (Unreleased -> 2.7.0). Since 2.6.0 (verification-first detection): SecretNode now takes a domain and enumerates subdomains (Certificate Transparency x2), recovers historical URLs (Wayback/CommonCrawl), mines JS for referenced endpoints and third-party hosts, probes liveness, checks each host for subdomain-takeover risk, and scans live pages + archived bundles for secrets and posture — concurrently — into one reviewable report, from the CLI or the dashboard. Passive and authorized-scope throughout. Suite 187 -> 270, ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Grows SecretNode from a single-URL secret scanner into a passive attack-surface platform. Give it a domain and it discovers the whole surface, then scans it — all passively, authorized-scope only.
The pipeline:
domain → enumerate subdomains → recover historical URLs → probe liveness → check takeover risk → scan live pages + archived bundles → one reviewable report, drivable from the CLI or the dashboard.Delivered as small, test-backed slices:
Discovery
recon.py)historical.py).jsendpoints one level deeper. (surface.py)Orchestration
MAX_TARGETScap, and per-host error isolation; combined client report. (orchestrator.py)HOST_SCAN_CONCURRENCY) instead of sequential.takeover.py)Dashboard / API
POST /api/deep-scansstreaming background endpoint + a DEEP toggle in the web UI to run whole-domain scans and download the combined report.Detection-quality fixes (carried over)
Every extractor regex is bounded/ReDoS-safe. Version bumped to 2.7.0 (single-sourced). Full detail in
CHANGELOG.md.Type of change
Checklist
ruff check backend/passespytestpasses (added/updated tests for the change) — suite 187 → 270, all greenseverity,cwe, andremediation(takeover findings carry severity + CWE-350 + remediation)🤖 Generated with Claude Code
https://claude.ai/code/session_01V6XK23ZKmYtCovRte9a73E
Generated by Claude Code