Release v2.7.0 — deep attack-surface platform (passive ASM)

.env.example

@@ -57,8 +57,53 @@ PORT=8000
 CONCURRENCY_LIMIT=20
 # Seconds per HTTP request before timeout.
 FETCH_TIMEOUT=20.0
-# Shannon-entropy floor (bits/char). Raise to 4.0 to cut Gemini cost / false matches.
+# Shannon-entropy floor (bits/char) for the GENERIC keyword=value catch-all.
+# Raise to 4.0 to cut Gemini cost / false matches.
 MIN_ENTROPY_THRESHOLD=3.5
+# Low anti-degenerate entropy floor for STRUCTURAL/provider detectors (AKIA…, ghp_…,
+# sk_live_…). Only rejects obvious junk (e.g. all-identical chars); keep it well below
+# MIN_ENTROPY_THRESHOLD so genuinely modest-entropy live keys are not dropped.
+MIN_STRUCTURAL_ENTROPY=2.5
+
+# ── Passive reconnaissance (subdomain enumeration) ──────────────────────────
+# Certificate Transparency front-end used to enumerate subdomains passively.
+# Discovery never contacts the target — only this CT source.
+CRTSH_URL=https://crt.sh
+# Seconds to wait for the CT-log query before failing closed (empty result).
+SUBDOMAIN_ENUM_TIMEOUT=30
+# Safety cap on how many discovered subdomains to return.
+MAX_SUBDOMAINS=500
+# ── Multi-target deep scan (domain → enumerate → probe → scan each host) ─────
+# Max live hosts scanned in one --deep-scan run.
+MAX_TARGETS=25
+# Parallel liveness probes when checking which discovered hosts are up.
+PROBE_CONCURRENCY=10
+# Seconds per liveness probe before a host is treated as unreachable.
+PROBE_TIMEOUT=10
+# How many hosts a deep scan scans in parallel. Each host scan is itself
+# concurrent, so keep this modest on a Raspberry Pi (3 is a good default).
+HOST_SCAN_CONCURRENCY=3
+# ── Historical path discovery (passive, from public web archives) ───────────
+# Recover historically-exposed URLs (the passive alternative to brute-forcing).
+WAYBACK_CDX_URL=http://web.archive.org/cdx/search/cdx
+COMMONCRAWL_COLLINFO=https://index.commoncrawl.org/collinfo.json
+# Set false to skip CommonCrawl and use the Wayback Machine only (faster).
+ENABLE_COMMONCRAWL=true
+HISTORICAL_TIMEOUT=30
+HISTORICAL_RETRIES=2
+# Safety cap on how many historical URLs to return.
+MAX_HISTORICAL_URLS=2000
+# Cap on externally-supplied seed assets fetched per scan (e.g. historical JS
+# bundles fed into a --deep-scan --with-historical run).
+MAX_SEED_URLS=200
+# ── Surface intelligence (endpoints referenced in code + associated hosts) ──
+# Mine fetched JS/HTML for referenced endpoints and external hosts, and fetch
+# same-site .js endpoints one level deeper. Set false to disable.
+EXTRACT_SURFACE=true
+# Same-site .js endpoints (referenced in code) to fetch for the deeper crawl.
+MAX_ENDPOINT_SEEDS=50
+# Cap on how many discovered endpoints to store/report.
+MAX_DISCOVERED_ENDPOINTS=300
 # Skip JS assets larger than this many bytes (default 5 MB).
 MAX_ASSET_BYTES=5242880
 # Only surface AI-confirmed findings at or above this confidence (0-100).

.gitignore

@@ -18,6 +18,7 @@ venv/
 backend/data/*.db
 backend/data/*.db-wal
 backend/data/*.db-shm
+backend/data/*.db-journal
 *.sqlite3
 
 # OS / editor

CHANGELOG.md

@@ -3,6 +3,102 @@
 All notable changes to SecretNode are documented here. This project adheres to
 [Semantic Versioning](https://semver.org/).
 
+## [2.7.0] — Deep attack-surface platform (passive)
+
+SecretNode grows from a single-URL secret scanner into a **passive attack-surface platform**: give it
+a domain and it enumerates the subdomain surface, recovers historically-exposed URLs from public
+archives, mines JavaScript for referenced endpoints and third-party hosts, probes which hosts are
+live, checks each for **subdomain-takeover risk**, and scans them all — live pages *and* archived
+bundles — for exposed credentials and misconfigurations, aggregated into one reviewable report and
+drivable from the CLI **or** the dashboard. Every layer stays passive and authorized-scope only.
+Test suite **187 → 270**, all green; ruff clean. See `docs/TECHNICAL-AUDIT-AND-ROADMAP.md`.
+
+### Changed
+- **Concurrent host orchestration (deep-dive slice D5).** A domain deep scan now scans its hosts in
+  parallel with a bounded semaphore (`HOST_SCAN_CONCURRENCY`, default 3) instead of one at a time —
+  a large multi-host domain finishes far faster. Results are collected in target order, per-host
+  error isolation is preserved (one host failing never sinks the run), and progress is emitted as
+  `[k/N] host — done` events. A test proves the parallelism is real *and* stays within the bound.
+
+### Added
+- **Subdomain-takeover detection (deep-dive slice D1).** New `backend/takeover.py` flags hosts whose
+  DNS still points (via CNAME) at an **unclaimed third-party service** (S3, GitHub Pages, Heroku,
+  Netlify, Shopify, Fastly, Zendesk, …) — a hijackable subdomain an attacker can claim to serve
+  content from the target's domain. High-precision by design: a host is flagged only when the
+  response carries a service's *specific* unclaimed-resource signature (generic 404s excluded), with
+  the CNAME recorded as corroborating evidence. The deep scan runs a concurrent takeover pass over
+  every in-scope host; results surface as CRITICAL/HIGH findings with a "Subdomain Takeover Risks"
+  section + KPI in the combined report. Passive (DNS + one GET), stdlib-only, ReDoS-free.
+- **Surface intelligence: endpoints + associated-host graph (deep-ASM slices 5 & 4).** New
+  `backend/surface.py` mines every fetched asset (passively, no new target requests) for two things:
+  **(5)** URLs/paths referenced in the JavaScript — `fetch()`/`axios` targets, `/api/…` routes a live
+  page crawl never links to — and then fetches same-site `.js` endpoints **one level deeper** so
+  code-referenced bundles get secret-scanned too; and **(4)** the external hosts each asset talks to
+  (CDNs, APIs, third parties), aggregated into an **associated-asset graph**. `run_scan` now returns
+  `discovered_endpoints` + `associated_hosts`; both reports gain an "Attack Surface Intelligence"
+  section. Extractor regexes are bounded/ReDoS-safe. Config: `EXTRACT_SURFACE`, `MAX_ENDPOINT_SEEDS`,
+  `MAX_DISCOVERED_ENDPOINTS`.
+- **Dashboard domain-mode + deep-scan API (deep-ASM slice 6).** The whole deep-ASM pipeline is now
+  drivable from the web UI, not just the CLI. New `POST /api/deep-scans` runs a domain-wide deep scan
+  as a streaming background task — per-host progress flows over the existing `/ws/logs/{scan_id}`
+  WebSocket (`run_deep_scan` gained a `broadcast` hook and emits enumerate/probe/per-host/complete
+  events), and the report endpoint serves the combined multi-target report for deep results.
+  Frontend: a **DEEP toggle** turns the target box into a whole-domain scan (bare domain in →
+  enumerate + historical + probe + scan-all), finalising on `deep_scan_complete` rather than
+  per-host. API tests added (route, auth, input caps, start). Passive; authorized-scope only.
+- **Historical bundles fed into the scan (deep-ASM slice 3.5).** `run_scan()` gains a `seed_urls`
+  parameter — externally-supplied asset URLs are fetched and scanned alongside the live crawl,
+  deduped against it (capped by `MAX_SEED_URLS`). `run_deep_scan(include_historical=True)` now
+  recovers the domain's historical JS bundles (Wayback/CommonCrawl) and routes each host its own
+  archived bundles as seeds, so a secret in a forgotten bundle **no live page links to** still gets
+  fetched and confirmed. CLI: `python cli.py <domain> --deep-scan --with-historical`; the combined
+  report gains a "Historical URLs" metric. This turns discovery into findings — the payoff of the
+  whole passive discovery chain.
+- **Historical path discovery (deep-ASM slice 3).** New `backend/historical.py` recovers a domain's
+  historically-exposed URLs from **public web archives (Wayback Machine + CommonCrawl)** — the
+  passive alternative to directory/content brute-forcing, so no request ever touches the target. Two
+  sources merged with backoff retries and fail-closed handling (matching the subdomain layer);
+  surfaces forgotten endpoints, stale JS bundles and old admin paths a live crawl would never link
+  to. `HistoricalResult` exposes the raw URLs, the unique-path view ("hidden directories"), and a
+  `js_urls()` helper (highest-value scan seeds). CLI: `python cli.py <domain> --historical`. Config:
+  `WAYBACK_CDX_URL`, `COMMONCRAWL_COLLINFO`, `ENABLE_COMMONCRAWL`, `HISTORICAL_TIMEOUT`,
+  `HISTORICAL_RETRIES`, `MAX_HISTORICAL_URLS`.
+- **Multi-target orchestration (deep-ASM slice 2).** New `backend/orchestrator.py` closes the loop
+  from discovery to findings: a single domain → passive subdomain enumeration → liveness probe of
+  each host → the existing passive secret+posture scan per live host → one aggregated
+  `DeepScanResult`. Includes a per-host **SSRF guard** (a discovered host that resolves to a
+  private/internal address is skipped unless `ALLOW_PRIVATE_TARGETS=true`), a `MAX_TARGETS` cap,
+  concurrent probing, and per-host error isolation (one host failing never sinks the run). New
+  combined client report `report.generate_deep_scan_html()` (subdomain surface + live hosts +
+  per-host confirmed/needs-review/posture). CLI: `python cli.py <domain> --deep-scan -o report.html`.
+  Config: `MAX_TARGETS`, `PROBE_CONCURRENCY`, `PROBE_TIMEOUT`.
+- **Passive subdomain enumeration (deep-ASM slice 1).** New `backend/recon.py` expands a domain
+  into its known subdomain surface from **Certificate Transparency** — fully passive, it never
+  contacts the target, so it runs before a client engagement is signed. Queries **two independent CT
+  sources (crt.sh + Certspotter)** with backoff retries and merges them, so a single flaky/rate-
+  limited source (crt.sh 502s often) no longer zeroes out a good result; the result lists which
+  sources succeeded and only reports an error if *all* fail. `extract_registrable_domain()`
+  normalises URL/host/IP inputs (two-label public-suffix table incl. `.bd`). Exposed via the CLI:
+  `python cli.py <domain> --subdomains`. First layer of the passive attack-surface pipeline
+  (subdomains → historical paths → associated assets → existing secret/posture scan).
+
+### Fixed
+- **False-negative: structural keys wrongly entropy-gated.** The Shannon-entropy floor
+  (`MIN_ENTROPY_THRESHOLD=3.5`) was applied uniformly to every detector, silently dropping
+  genuinely low-entropy but well-formed provider keys (e.g. an AWS key ID at ~3.27 bits) before
+  they ever reached AI validation — the worst failure mode for a scanner. Entropy is now
+  class-aware: the *generic* keyword=value catch-all keeps the full 3.5 bar, while
+  *structural/provider* detectors (AKIA…, ghp_…, sk_live_…, PEM, fixed-format tokens) only clear a
+  low anti-degenerate floor (`MIN_STRUCTURAL_ENTROPY=2.5`) that still rejects obvious junk like
+  `AKIAAAAAAAAAAAAAAAAA`. Precision/recall stays 1.000/1.000.
+- **False-negative: AI-dismissed structural matches silently dropped.** A finding was routed to
+  manual review only when AI validation was *unavailable*; a structural/provider match the AI
+  *actively* rejected with a real confidence matched no bucket and was discarded — so a live key the
+  AI merely under-called on (e.g. lacking page context) vanished with no trace. New
+  `classify_validated()` sends any structural match the AI does **not confidently dismiss** to
+  manual review instead of dropping it; the generic catch-all keeps aggressive filtering, so the
+  "no false positives in Confirmed" promise holds. Suite **187 → 197**.
+
 ## [2.6.0] — Detection quality, safety & attack-surface breadth
 
 A measured capability pass grounded in a fresh audit vs 2026 secret-scanning SOTA

Makefile

@@ -20,7 +20,10 @@ bench: ## Measure detection-layer precision/recall on the labelled corpus (R2)
 	cd backend && SECRETNODE_API_KEY=bench python -m bench.run_bench
 
 run: ## Start the server (requires .env with SECRETNODE_API_KEY)
-	cd backend && uvicorn main:app --host 0.0.0.0 --port 8000 --loop uvloop
+	# `python -m uvicorn` uses the same interpreter that runs the CLI, so it works
+	# even when the `uvicorn` console script isn't on PATH; --loop auto uses uvloop
+	# when available and falls back to asyncio otherwise (no hard uvloop dependency).
+	cd backend && python -m uvicorn main:app --host 0.0.0.0 --port 8000 --loop auto
 
 docker: ## Build and run via docker compose
 	docker compose up --build

backend/cli.py

@@ -27,6 +27,9 @@
 import sys
 from urllib.parse import urlparse
 
+import historical
+import orchestrator
+import recon
 import report
 import scanner
 
@@ -78,11 +81,120 @@ def build_parser() -> argparse.ArgumentParser:
     ap.add_argument("-o", "--output", help="Write report to this file (default: stdout)")
     ap.add_argument("--fail-on-findings", action="store_true",
                     help="Exit non-zero if any confirmed findings (use as a CI gate)")
+    ap.add_argument("--subdomains", action="store_true",
+                    help="Passive attack-surface discovery: enumerate the target's subdomains "
+                         "via Certificate Transparency (crt.sh) and print them. Never contacts "
+                         "the target. Authorized use only.")
+    ap.add_argument("--historical", action="store_true",
+                    help="Passive path discovery: recover historically-exposed URLs for the domain "
+                         "from public web archives (Wayback Machine + CommonCrawl) and print them. "
+                         "Never contacts the target. Authorized use only.")
+    ap.add_argument("--deep-scan", dest="deep_scan", action="store_true",
+                    help="Domain-wide deep scan: enumerate subdomains, probe which hosts are live, "
+                         "scan each, and write a combined report. Passive; authorized use only.")
+    ap.add_argument("--max-targets", dest="max_targets", type=int, default=orchestrator.MAX_TARGETS,
+                    help=f"Max live hosts to scan in a --deep-scan run (default: {orchestrator.MAX_TARGETS})")
+    ap.add_argument("--with-historical", dest="with_historical", action="store_true",
+                    help="In a --deep-scan, also recover historical JS bundles from public archives "
+                         "(Wayback/CommonCrawl) and scan them as seeds — catches secrets in forgotten "
+                         "bundles no live page links to. Slower; passive.")
     return ap
 
 
+async def _run_deep_scan(args) -> int:
+    """Domain-wide deep scan: enumerate → probe live hosts → scan each → combined report."""
+    result = await orchestrator.run_deep_scan(
+        args.target,
+        max_crawl_pages=max(1, args.crawl),
+        verify=args.verify,
+        only_verified=args.only_verified,
+        max_targets=max(1, args.max_targets),
+        include_historical=args.with_historical,
+    )
+    if args.output:
+        report_fmt = (args.format if args.format in ("json",) else "html")
+        body = (json.dumps(result.to_dict(), indent=2) if report_fmt == "json"
+                else report.generate_deep_scan_html(result.to_dict()))
+        with open(args.output, "w", encoding="utf-8") as fh:
+            fh.write(body)
+        print(f"Deep-scan report written to {args.output}", file=sys.stderr)
+    else:
+        sys.stdout.write(json.dumps(result.to_dict(), indent=2) + "\n")
+
+    t = result.to_dict()["totals"]
+    hist = f", {t['historical_urls']} historical URL(s)" if t.get("historical_urls") else ""
+    print(
+        f"SecretNode deep scan of {result.domain}: {t['subdomains']} subdomain(s), "
+        f"{t['live_hosts']} live, {t['hosts_scanned']} scanned{hist} — "
+        f"{t['confirmed']} confirmed, {t['needs_review']} needs-review, "
+        f"{t['posture_issues']} posture issue(s).",
+        file=sys.stderr,
+    )
+    if args.fail_on_findings and t["confirmed"]:
+        return 1
+    return 0
+
+
+async def _run_subdomain_enum(target: str) -> int:
+    """Passive subdomain discovery mode: expand a domain into its known subdomain
+    surface from Certificate Transparency and print the results as JSON."""
+    domain = recon.extract_registrable_domain(target)
+    if domain is None:
+        raise SystemExit(
+            f"Cannot enumerate subdomains for {target!r} — need a domain "
+            "(subdomain enumeration does not apply to bare IP addresses)."
+        )
+    async with scanner.build_client() as client:
+        result = await recon.enumerate_subdomains(client, domain)
+    print(json.dumps(result.to_dict(), indent=2))
+    sources = ", ".join(result.sources) if result.sources else "none"
+    print(
+        f"SecretNode: discovered {result.count} subdomain(s) for {domain} "
+        f"(sources: {sources})." + (f" [error: {result.error}]" if result.error else ""),
+        file=sys.stderr,
+    )
+    return 0
+
+
+async def _run_historical(target: str) -> int:
+    """Passive path discovery: recover historically-exposed URLs from public
+    archives (Wayback + CommonCrawl) — the passive alternative to brute-forcing."""
+    domain = recon.extract_registrable_domain(target)
+    if domain is None:
+        raise SystemExit(
+            f"Cannot run historical discovery for {target!r} — need a domain "
+            "(does not apply to bare IP addresses)."
+        )
+    async with scanner.build_client() as client:
+        result = await historical.discover_historical_urls(client, domain)
+    print(json.dumps(result.to_dict(), indent=2))
+    sources = ", ".join(result.sources) if result.sources else "none"
+    print(
+        f"SecretNode: recovered {result.count} historical URL(s) "
+        f"({len(result.paths)} unique path(s), {len(result.js_urls())} JS) for {domain} "
+        f"(sources: {sources})." + (f" [error: {result.error}]" if result.error else ""),
+        file=sys.stderr,
+    )
+    return 0
+
+
 def main(argv: list[str] | None = None) -> int:
     args = build_parser().parse_args(argv)
+
+    # Passive discovery mode: enumerate subdomains and exit. Accepts a bare domain
+    # (no scheme) since it never fetches the target — only Certificate Transparency.
+    if args.subdomains:
+        return asyncio.run(_run_subdomain_enum(args.target))
+
+    # Passive historical path discovery from public archives; accepts a bare domain.
+    if args.historical:
+        return asyncio.run(_run_historical(args.target))
+
+    # Domain-wide deep scan: enumerate → probe → scan each live host. Accepts a
+    # bare domain; the orchestrator applies the SSRF guard per discovered host.
+    if args.deep_scan:
+        return asyncio.run(_run_deep_scan(args))
+
     if not args.target.startswith(("http://", "https://")):
         raise SystemExit("Target must start with http:// or https://")
     assert_public_target(args.target)