diff --git a/.env.example b/.env.example index 17f82bf..d488f4d 100644 --- a/.env.example +++ b/.env.example @@ -151,3 +151,40 @@ MAX_SOURCEMAP_SOURCES=200 # response for missing/weak security headers, version disclosure, and insecure # cookies. Pure analysis of a response the target already serves. SCAN_HTTP_POSTURE=true + +# Inline SSR state decoding (v2.7.6). Server-rendered apps embed bootstrap state +# in the HTML (__NEXT_DATA__, __NUXT__, __INITIAL_STATE__, __APOLLO_STATE__). +# The raw HTML is already scanned, so a plainly-embedded secret is caught either +# way; this decodes the JSON so a credential mangled by \uXXXX escaping is still +# recognised. Local decode only — no extra requests, scan stays passive. +# Asset caching (v2.7.7). Sends If-None-Match / If-Modified-Since on re-scans. +# An asset that is BOTH unchanged (304) AND produced no finding last time is +# skipped entirely; one that previously had a finding is always refetched so the +# finding cannot silently vanish from a report. No response body is ever cached +# — only validators, a content hash, and a clean/dirty flag — because a client's +# JavaScript can contain live credentials. Cache entries expire after 24h. +# Client reports redact the credential by default (sk_abc…******…3fc4 (51 chars)). +# A report gets emailed, forwarded and archived, so writing a live secret into +# one turns the deliverable itself into a second exposure — and the RoE we ask +# clients to sign promises redaction. The redacted form stays greppable, so a +# developer can still identify exactly which key to rotate. Set true only when +# you deliberately need the full value in the artefact. +REPORT_FULL_SECRETS=false + +ASSET_CACHE=true + +SCAN_INLINE_JSON=true +MAX_INLINE_JSON_BYTES=2000000 + +# ── Scan politeness / resilience (v2.7.5) ────────────────────────────────── +# Retries use exponential backoff with equal jitter so concurrent workers do not +# all retry on the same tick (a thundering herd against a host that just asked +# for relief). RETRY_MAX_BACKOFF caps any single wait, including a server's +# Retry-After. +RETRY_MAX_BACKOFF=30.0 +# Adaptive per-host throttle: when a host answers 429/503 we start pacing +# requests to THAT host only, growing by THROTTLE_STEP per signal up to +# THROTTLE_MAX_DELAY, and decaying as it recovers. Costs nothing while a host is +# healthy. Raise THROTTLE_STEP to be gentler on fragile client infrastructure. +THROTTLE_STEP=0.5 +THROTTLE_MAX_DELAY=5.0 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a4db184..fb549c6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -37,6 +37,13 @@ jobs: - name: Run test suite run: pytest + # A scanner's whole value is that its findings can be trusted, so a + # precision regression is release-blocking, not a warning. This measures + # the deterministic detection layer against the labelled corpus and fails + # the build if precision or recall drops. + - name: Detection quality gate (precision/recall) + run: make bench + docker-build: runs-on: ubuntu-latest steps: diff --git a/CHANGELOG.md b/CHANGELOG.md index fe0efd5..c3e1080 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,260 @@ All notable changes to SecretNode are documented here. This project adheres to [Semantic Versioning](https://semver.org/). +## [2.7.9] — Deep QA before release: reports no longer leak the credential + +A full pre-PR QA pass: booted the application, ran real end-to-end scans against a local target +with planted secrets, exercised every export format, the CLI, and a re-scan to prove the cache. It +surfaced one serious defect and one cosmetic one. + +### Fixed +- **Client reports contained the full, live credential.** The CSV column was named + `matched_value_partial` but wrote `raw_match` verbatim, and the HTML report did the same. A report + is emailed, forwarded and archived — writing a working secret into one turns the deliverable + itself into a second exposure, and directly contradicts the Rules of Engagement we ask clients to + sign ("only the minimum evidence needed to prove a finding, with sensitive data redacted"). + Values are now redacted to `sk_798…******…3fc4 (51 chars)`: still greppable, so a developer can + identify exactly which key to rotate, but not usable. `REPORT_FULL_SECRETS=true` opts back in for + the case where an operator deliberately needs the full value. SARIF was already clean. +- **Version drift in the dashboard.** The footer and boot log were hardcoded to v2.7.1 and never + touched by the runtime `/api/health` sync, so a client demo showed a stale version indefinitely. + The footer is now synced too and the boot log no longer hardcodes a version. `report.py`'s + fallback constant was also stale. + +### Verified end to end (not just unit-tested) +- Application boots; `/api/health` reports the correct version; dashboard serves. +- Real scan of a local target found **both** of this session's headline features working in a live + system: the v2.7.2 **ElevenLabs detector**, and a v2.7.6 recovery of an Anthropic key that was + `\uXXXX`-escaped inside `__NEXT_DATA__` (with a raw `<` in the blob, exercising the v2.7.8 fix). +- **Re-scan** repopulated findings rather than losing them, and `asset_cache` correctly recorded + both assets as `was_clean=False` so they will always be refetched. +- HTML / CSV / SARIF all generate and stamp the right version; SARIF carries all 63 rules. +- CLI produces correct CSV output. +- **SSRF guard confirmed working** — refused a localhost target until `ALLOW_PRIVATE_TARGETS` was + explicitly set for the lab run. +- Inline dashboard JavaScript passes `node --check` after the mobile-layout edits. + +Suite **349 → 353**, all green; ruff clean; bench 1.000/1.000. + +## [2.7.8] — Pre-release review: three real bugs, and a quality gate that can fail + +A deliberate self-review of everything shipped in v2.7.2–v2.7.7 before opening the PR. It found +three genuine defects — two of them in the very code written to prevent that class of failure — +and one process gap that mattered more than any of them. + +### Fixed +- **Inline-JSON extraction truncated at the first raw `<`.** `_INLINE_SCRIPT_RE` used a `[^<]` + character class, so any blob containing a literal `<` — prose like `"a < b"`, embedded HTML + fragments, templates — was cut short and failed to parse, losing every secret after that point. + Now lazily matched to the `` terminator (linear-time, still ReDoS-free: verified at + 0.18s on hostile input). +- **A previously-dirty asset could be lost on 304 when `RETRY_ATTEMPTS=1`.** The refetch was done + via `continue`, which consumed a retry attempt; with only one attempt configured the request was + never re-issued and the asset was dropped — **the exact "a finding silently vanishes" failure + the cache was designed to prevent.** The unconditional refetch now happens inline. +- **An unprompted `304` with no cache entry burned a retry.** Now refetched immediately, and a + server that answers 304 even unconditionally terminates instead of spinning. + +### Added — the detection quality gate is now real +- **The benchmark corpus covers this session's work.** 27 → **45 samples**. All nine v2.7.2 AI/ML + detectors, the v2.7.6 inline-SSR path, and **eight hard negatives** chosen to be structurally + confusable with the new patterns: `sk_` + wrong-length hex, a 64-hex SHA digest that resembles an + OpenRouter key, provider-shaped placeholders, benign inline JSON. Shipping nine detectors with + zero corpus coverage meant the "measured precision" claim did not cover them. +- **`make bench` can fail.** It previously printed a report and always exited 0, so it could not + block anything. It now enforces `BENCH_MIN_PRECISION` / `BENCH_MIN_RECALL` (default 1.0), prints + the offending samples, and exits non-zero. +- **CI runs it.** A precision regression is now release-blocking. Verified end to end by + deliberately loosening the ElevenLabs regex: precision falls 1.000 → 0.917, the two false + positives are named, and the build fails. + +Suite **337 → 349**, all green; ruff clean; bench 1.000/1.000 on the larger corpus. + +## [2.7.7] — R10: asset caching for re-scans + +Closes roadmap item **R10**. Re-scanning a target refetched every asset from scratch — wasted +bandwidth on the client's servers and wasted CPU on ours, since most assets neither change between +engagements nor contain anything. + +### Added +- **Conditional GET on re-scans.** SecretNode now sends `If-None-Match` / `If-Modified-Since` from + a per-target cache of HTTP validators (`asset_cache` table, 24h TTL). +- **Correctness rule for 304s.** A `304 Not Modified` is acted on by history, not blindly: + - asset was **clean** last scan → **skipped entirely** (unchanged + previously clean means still + clean), and never enters the scan text; + - asset previously **yielded a finding** → **refetched unconditionally**, so the finding is + reproduced. A finding that disappeared from a report would read as *resolved*, which is a + dangerous lie to tell a client. +- `purge_asset_cache()` for clearing one target's cache or all of it. +- Config: `ASSET_CACHE` (default on). + +### Privacy: no response bodies are cached +The obvious implementation stores bodies so a 304 can still be scanned. We deliberately do not: a +client's JavaScript can contain live credentials, and caching it would leave a long-lived copy of +their secrets on our disk — which the engagement's confidentiality terms do not allow. The cache +holds **only** the validators, a truncated content hash, and a clean/dirty flag. That is enough to +skip the overwhelming majority of assets, and a test asserts no body content is ever persisted. + ++10 tests, including a **mutation check** (with `ASSET_CACHE=false`, five cache tests fail, proving +they exercise the real path) and a real-SQLite round-trip covering upsert, per-target isolation and +purge. Suite **327 → 337**, all green; ruff clean. + +## [2.7.6] — Inline SSR state decoding + +Server-rendered apps embed their bootstrap state in the HTML — Next.js writes +`', + "Anthropic API Key"), +] + +# Hard negatives: strings that structurally resemble the new detectors but are +# not credentials. These are where false positives actually come from. +_AI_NEGATIVES: list[Sample] = [ + # sk_ + hex but the wrong length for ElevenLabs + Sample("sk-hex-short", f'id = "sk_{_hex(20)}"', None), + Sample("sk-hex-long", f'id = "sk_{_hex(72)}"', None), + # right prefix, obvious placeholder + Sample("elevenlabs-placeholder", 'ELEVENLABS_API_KEY = "sk_your_api_key_here"', None), + Sample("hf-placeholder", 'HF_TOKEN = "hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"', None), + # a 64-hex blob that is a digest, not an OpenRouter key + Sample("sha256-digest", f'integrity = "{_hex(64)}"', None), + # docs-style examples + Sample("groq-docs", 'api_key="gsk_YOUR_GROQ_KEY"', None), + Sample("openrouter-docs", 'Authorization: Bearer sk-or-v1-', None), + # benign inline JSON must not produce anything + Sample("inline-json-benign", + '', None), +] + +CORPUS: list[Sample] = _POSITIVES + _NEGATIVES + _AI_POSITIVES + _AI_NEGATIVES diff --git a/backend/bench/run_bench.py b/backend/bench/run_bench.py index ee484b4..7323a0a 100644 --- a/backend/bench/run_bench.py +++ b/backend/bench/run_bench.py @@ -25,6 +25,8 @@ from __future__ import annotations +import os + from bench.corpus import CORPUS, Sample from scanner import extract_secrets @@ -96,5 +98,32 @@ def format_report(m: dict) -> str: return "\n".join(lines) +# Quality floor enforced in CI. A detector change that drops below either bar +# fails the build instead of silently shipping: a scanner's whole value is that +# its findings can be trusted, so precision regressions are release-blocking. +MIN_PRECISION = float(os.environ.get("BENCH_MIN_PRECISION", "1.0")) +MIN_RECALL = float(os.environ.get("BENCH_MIN_RECALL", "1.0")) + + +def main() -> int: + m = evaluate() + print(format_report(m)) + failures = [] + if m["precision"] < MIN_PRECISION: + failures.append(f"precision {m['precision']:.3f} < required {MIN_PRECISION:.3f}") + if m["recall"] < MIN_RECALL: + failures.append(f"recall {m['recall']:.3f} < required {MIN_RECALL:.3f}") + if failures: + print("\nDETECTION QUALITY GATE FAILED:") + for f in failures: + print(f" ✗ {f}") + print("\nEither fix the detector, or — if the corpus is wrong — update it " + "deliberately in the same commit and say why.") + return 1 + print("\n quality gate: PASS " + f"(precision >= {MIN_PRECISION:.2f}, recall >= {MIN_RECALL:.2f})") + return 0 + + if __name__ == "__main__": # pragma: no cover - print(format_report(evaluate())) + raise SystemExit(main()) diff --git a/backend/main.py b/backend/main.py index 5f643e3..30a0285 100644 --- a/backend/main.py +++ b/backend/main.py @@ -42,10 +42,11 @@ # 3.12+, and uvicorn already selects the event loop itself via --loop (auto/uvloop), # so the speed-up is preserved without a module-level global side effect. -from scanner import run_scan, ScanState +from scanner import run_scan, ScanState, load_asset_cache, drain_asset_cache import orchestrator from storage import ( init_db, save_scan, load_scans, load_scan, get_previous_scan_for_target, + get_asset_cache, save_asset_cache, mark_false_positive, unmark_false_positive, get_suppressed_fingerprints, list_false_positives, ) @@ -327,6 +328,10 @@ async def _run() -> None: ) suppressed_fps = await get_suppressed_fingerprints(request.target_url) + # Prime the conditional-GET cache so unchanged, previously-clean + # assets can be skipped instead of re-downloaded. + load_asset_cache(await get_asset_cache(request.target_url)) + result = await run_scan( target_url=request.target_url, scan_id=scan_id, @@ -340,6 +345,7 @@ async def _run() -> None: ) _registry[scan_id]["meta"] = result await save_scan(scan_id, result) + await save_asset_cache(request.target_url, drain_asset_cache()) except asyncio.CancelledError: logger.info("Scan %s task cancelled", scan_id) await manager.broadcast_scan(scan_id, { diff --git a/backend/report.py b/backend/report.py index af8ca32..c564949 100644 --- a/backend/report.py +++ b/backend/report.py @@ -19,6 +19,7 @@ import csv import html import io +import os import json from datetime import datetime, timezone from pathlib import Path @@ -45,7 +46,7 @@ def _tool_version() -> str: return s.split("=", 1)[1].strip().strip('"').strip("'") except Exception: pass - return "2.7.1" + return "2.7.9" _TOOL_VERSION = _tool_version() @@ -55,6 +56,29 @@ def _severity_of(finding: dict[str, Any]) -> str: return str(finding.get("severity", "MEDIUM")).upper() +# Client reports get emailed, forwarded and archived. Writing a live credential +# into one turns the deliverable itself into a second exposure — and the RoE we +# ask clients to sign promises the opposite: "only the minimum evidence needed +# to prove a finding, with sensitive data redacted". +# +# A redacted value still does its job: the developer can grep for the prefix and +# identify exactly which key to rotate, without the report handing an attacker a +# working credential. An operator who genuinely needs the full value (handing it +# to the client's engineer for rotation) can opt in explicitly. +REPORT_FULL_SECRETS = os.environ.get("REPORT_FULL_SECRETS", "false").lower() == "true" + + +def redact_secret(value: str) -> str: + """Mask the middle of a credential, keeping enough to identify it.""" + if not value: + return "" + if REPORT_FULL_SECRETS: + return value + if len(value) <= 12: + return "*" * len(value) + return f"{value[:6]}…{'*' * 6}…{value[-4:]} ({len(value)} chars)" + + def _sort_key(finding: dict[str, Any]) -> tuple[int, int, str]: """Sort by severity (critical first), then AI confidence (high first), then secret type for stable ordering.""" @@ -153,7 +177,7 @@ def finding_row(f: dict[str, Any]) -> str: {html.escape(f.get('source_url', f.get('target_url','')))} {f.get('confidence',0)}% {badge}{ver_badge(f)} - {html.escape(f.get('raw_match',''))} + {html.escape(redact_secret(f.get('raw_match','')))} {html.escape(f.get('reason',''))} """ @@ -400,13 +424,13 @@ def generate_csv_report(scan: dict[str, Any]) -> str: f.get("source_url", f.get("target_url", "")), f.get("confidence", 0), "NEW" if f.get("is_new", True) else "RECURRING", f.get("verified", "disabled"), f.get("verified_detail", ""), f.get("impact", ""), - f.get("raw_match", ""), f.get("reason", ""), f.get("found_at", ""), + redact_secret(f.get("raw_match", "")), f.get("reason", ""), f.get("found_at", ""), ]) for f in scan.get("needs_review_findings", []): writer.writerow([ "NEEDS_REVIEW", _severity_of(f), f.get("cwe", ""), f.get("secret_type", ""), f.get("source_url", f.get("target_url", "")), - "", "", "", "", f.get("impact", ""), f.get("raw_match", ""), + "", "", "", "", f.get("impact", ""), redact_secret(f.get("raw_match", "")), f.get("reason", ""), f.get("found_at", ""), ]) return buf.getvalue() diff --git a/backend/scanner.py b/backend/scanner.py index ab3e95e..526b358 100644 --- a/backend/scanner.py +++ b/backend/scanner.py @@ -13,11 +13,13 @@ import logging import math import os +import random import re import time import uuid from dataclasses import dataclass, field from datetime import datetime, timezone +from email.utils import parsedate_to_datetime from typing import Any, Callable, Coroutine from urllib.parse import urljoin, urlparse @@ -246,6 +248,11 @@ def _meta(self) -> "SecretPattern": # Secret Pattern Registry # ───────────────────────────────────────────────────────────────────────────── +# The loose keyword=value catch-all. Named here because the de-duplication pass +# needs to know which detector is the fallback: when a provider-specific detector +# has already typed a credential, the generic claim on the same value is dropped. +GENERIC_SECRET_TYPE = "Generic High-Entropy Secret" + SECRET_PATTERNS: list[SecretPattern] = [ SecretPattern( name="AWS Access Key", @@ -337,7 +344,7 @@ def _meta(self) -> "SecretPattern": severity="HIGH", ), SecretPattern( - name="Generic High-Entropy Secret", + name=GENERIC_SECRET_TYPE, regex=re.compile( r"(?i)(?:api[_-]?key|secret|token|password|passwd|auth)\s*[=:]\s*['\"]([A-Za-z0-9\-_.~+/]{20,80})['\"]" ), @@ -382,6 +389,71 @@ def _meta(self) -> "SecretPattern": description="Anthropic (Claude) API key", severity="CRITICAL", ), + # ── AI / ML provider keys ──────────────────────────────────────────────── + # Modern AI stacks leak these constantly: the key is shipped to the browser + # because a frontend calls the provider directly instead of proxying through + # a backend. Each pattern below is structural (distinctive prefix + fixed + # length), so it stays high-precision without the generic entropy gate. + SecretPattern( + name="ElevenLabs API Key", + regex=re.compile(r"\b(sk_[a-f0-9]{48})\b"), + description="ElevenLabs text-to-speech API key", + severity="HIGH", + remediation=( + "Revoke this key in the ElevenLabs dashboard and issue a replacement. A leaked key " + "lets anyone consume the account's character quota and credits, generate audio, and " + "reach private/cloned voice models. Never ship it to the browser: proxy " + "text-to-speech calls through your own backend so the key stays server-side." + ), + ), + SecretPattern( + name="Groq API Key", + regex=re.compile(r"\b(gsk_[A-Za-z0-9]{52})\b"), + description="Groq inference API key", + severity="CRITICAL", + ), + SecretPattern( + name="Hugging Face Access Token", + regex=re.compile(r"\b(hf_[A-Za-z0-9]{34,40})\b"), + description="Hugging Face user access token", + severity="CRITICAL", + ), + SecretPattern( + name="Replicate API Token", + regex=re.compile(r"\b(r8_[A-Za-z0-9]{37,45})\b"), + description="Replicate model-inference API token", + severity="CRITICAL", + ), + SecretPattern( + name="Perplexity API Key", + regex=re.compile(r"\b(pplx-[A-Za-z0-9]{32,64})\b"), + description="Perplexity AI API key", + severity="CRITICAL", + ), + SecretPattern( + name="xAI API Key", + regex=re.compile(r"\b(xai-[A-Za-z0-9]{64,100})\b"), + description="xAI (Grok) API key", + severity="CRITICAL", + ), + SecretPattern( + name="OpenRouter API Key", + regex=re.compile(r"\b(sk-or-v1-[a-f0-9]{64})\b"), + description="OpenRouter aggregated-inference API key", + severity="CRITICAL", + ), + SecretPattern( + name="LangSmith API Key", + regex=re.compile(r"\b(lsv2_(?:pt|sk)_[a-f0-9]{32}_[a-f0-9]{10})\b"), + description="LangSmith / LangChain tracing API key", + severity="HIGH", + ), + SecretPattern( + name="Pinecone API Key", + regex=re.compile(r"\b(pcsk_[A-Za-z0-9_]{40,90})\b"), + description="Pinecone vector-database API key", + severity="CRITICAL", + ), SecretPattern( name="Slack Token", regex=re.compile(r"\b(xox[baprs]-[0-9A-Za-z\-]{12,72})\b"), @@ -689,6 +761,155 @@ def redact_snippet(snippet: str, secret_value: str) -> str: _WAF_BLOCK_CODES = frozenset({401, 403, 406, 429, 503}) +# ── Politeness: adaptive per-host throttle (v2.7.5) ────────────────────────── +# +# Being a good guest on a client's infrastructure is part of the engagement, not +# an afterthought: an authorized scan that trips rate limiting looks like an +# attack to their SOC and gets the scanner blocked mid-assessment. Two mechanics: +# +# • Jittered backoff. A deterministic 2**attempt makes every concurrent worker +# retry on the same tick — a thundering herd that keeps the host saturated +# exactly when it asked for relief. Randomising the delay spreads retries out. +# • Adaptive throttle. When a host answers 429/503 we start pacing subsequent +# requests to *that host only*, growing the pace while it keeps complaining +# and decaying it as it recovers. Cost is zero while a host is healthy. + +THROTTLE_MAX_DELAY = _env_float("THROTTLE_MAX_DELAY", 5.0) +THROTTLE_STEP = _env_float("THROTTLE_STEP", 0.5) +RETRY_MAX_BACKOFF = _env_float("RETRY_MAX_BACKOFF", 30.0) + +# host -> current politeness delay in seconds. Module-level so every worker +# sharing the event loop cooperates on the same host budget. +_host_delays: dict[str, float] = {} + + +def reset_throttle() -> None: + """Clear all learned per-host pacing (used between scans and in tests).""" + _host_delays.clear() + + +def _throttle_penalise(host: str) -> float: + """Record a rate-limit signal from `host`; return the new delay.""" + delay = min(_host_delays.get(host, 0.0) + THROTTLE_STEP, THROTTLE_MAX_DELAY) + _host_delays[host] = delay + return delay + + +def _throttle_reward(host: str) -> None: + """A clean response — relax pacing for `host`, forgetting it once healthy.""" + current = _host_delays.get(host) + if current is None: + return + relaxed = current - (THROTTLE_STEP / 2) + if relaxed <= 0: + _host_delays.pop(host, None) + else: + _host_delays[host] = relaxed + + +async def _throttle_wait(host: str) -> None: + """Pace a request to a host that has recently rate-limited us.""" + delay = _host_delays.get(host, 0.0) + if delay > 0: + await asyncio.sleep(delay) + + +# ── R10 · conditional fetch (asset caching) ────────────────────────────────── +# +# Re-scanning a target refetched every asset from scratch. Most assets do not +# change between engagements and most contain nothing, so that is wasted +# bandwidth on the client's servers and wasted CPU on ours. +# +# We send If-None-Match / If-Modified-Since and act on a 304 as follows: +# • asset was clean last time -> skip it entirely (unchanged + previously +# clean means still clean), returning CACHED_CLEAN. +# • asset had a finding -> refetch unconditionally, so the finding is +# reproduced rather than silently vanishing from the report. A finding that +# disappears reads as "resolved", which would be a dangerous lie. +# +# No response body is ever cached: a client's JavaScript can hold live +# credentials and we do not keep copies of it. Only validators + a content hash. + +ASSET_CACHE_ENABLED = os.environ.get("ASSET_CACHE", "true").lower() == "true" + +# Sentinel distinguishing "unchanged, previously clean, skip" from a real body +# and from a failure (None). +CACHED_CLEAN = "\x00__SECRETNODE_CACHED_CLEAN__" + +# Scan-scoped cache state. Module-level for the same reason _host_delays is: +# every fetch in a scan shares it without threading a parameter through +# spider_target and its six call sites. +_asset_cache_in: dict[str, dict[str, Any]] = {} +_asset_cache_out: dict[str, dict[str, Any]] = {} + + +def load_asset_cache(entries: dict[str, dict[str, Any]]) -> None: + """Prime the conditional-GET cache for a scan.""" + _asset_cache_in.clear() + _asset_cache_in.update(entries or {}) + _asset_cache_out.clear() + + +def drain_asset_cache() -> dict[str, dict[str, Any]]: + """Validators observed during this scan, for persisting afterwards.""" + return dict(_asset_cache_out) + + +def extract_headers_without_validators(headers: dict[str, str] | None) -> dict[str, str] | None: + """Strip conditional-request headers so a refetch actually returns a body.""" + if not headers: + return None + stripped = {k: v for k, v in headers.items() + if k.lower() not in ("if-none-match", "if-modified-since")} + return stripped or None + + +def _usable_body(body: str | None) -> bool: + """A real, scannable body — not a failure and not a cache hit.""" + return bool(body) and body != CACHED_CLEAN + + +def mark_asset_dirty(url: str) -> None: + """Record that `url` yielded a finding, so a future 304 refetches it + instead of skipping — a finding must never silently vanish.""" + if url in _asset_cache_out: + _asset_cache_out[url]["was_clean"] = False + + +def _backoff_delay(attempt: int) -> float: + """Exponential backoff with equal jitter, capped. + + Equal jitter (half fixed, half random) keeps a guaranteed minimum pause while + still de-synchronising concurrent workers, which full jitter alone does not. + """ + ceiling = min(RETRY_BACKOFF_BASE ** attempt, RETRY_MAX_BACKOFF) + return (ceiling / 2) + random.uniform(0, ceiling / 2) + + +def _parse_retry_after(raw: str | None, fallback: float) -> float: + """Parse a Retry-After header into seconds. Never raises. + + RFC 7231 allows either delta-seconds *or* an HTTP-date. Before v2.7.5 this was + parsed with a bare float(), so a spec-compliant date raised ValueError, hit the + generic handler, and made the scanner abandon the asset outright — a false + negative caused by the server behaving correctly. + """ + if not raw: + return fallback + raw = raw.strip() + try: + return max(0.0, float(raw)) + except ValueError: + pass + try: + when = parsedate_to_datetime(raw) + if when.tzinfo is None: + when = when.replace(tzinfo=timezone.utc) + return max(0.0, (when - datetime.now(timezone.utc)).total_seconds()) + except Exception: # noqa: BLE001 — a malformed header must never break a scan + return fallback + + def _browser_headers(user_agent: str) -> dict[str, str]: """A realistic modern-Chrome header set. Client-Hints + Sec-Fetch-* are what modern WAFs look for; sending them lets an authorized scan reach a @@ -762,6 +983,8 @@ async def fetch_url( url: str, semaphore: asyncio.Semaphore, broadcast: Broadcaster | None = None, + cache: dict[str, dict[str, Any]] | None = None, + cache_out: dict[str, dict[str, Any]] | None = None, ) -> tuple[str, str | None]: """ Fetch a URL with retry + exponential backoff. @@ -771,10 +994,14 @@ async def fetch_url( giving up, and emit a diagnostic that names the likely cause instead of a bare "failed". Respects 429 Retry-After. Returns (url, body) or (url, None). """ + host = urlparse(url).netloc async with semaphore: waf_block_status: int | None = None + revalidate = True for attempt in range(1, RETRY_ATTEMPTS + 1): try: + # Pace ourselves if this host has recently rate-limited us. + await _throttle_wait(host) if broadcast: await broadcast({ "type": "log", @@ -786,19 +1013,61 @@ async def fetch_url( extra_headers = None if attempt > 1 and waf_block_status and not _UA_OVERRIDE: extra_headers = _browser_headers(_USER_AGENTS[(attempt - 1) % len(_USER_AGENTS)]) + + # Conditional GET: only on the first attempt, and only while the + # cache is still trusted. `revalidate` flips off once we have + # decided we must see the body regardless. + _cin = cache if cache is not None else _asset_cache_in + _cout = cache_out if cache_out is not None else _asset_cache_out + entry = _cin.get(url) if (_cin and ASSET_CACHE_ENABLED) else None + if entry and revalidate: + cond = dict(extra_headers or {}) + if entry.get("etag"): + cond["If-None-Match"] = entry["etag"] + if entry.get("last_modified"): + cond["If-Modified-Since"] = entry["last_modified"] + if len(cond) > len(extra_headers or {}): + extra_headers = cond + response = await client.get(url, headers=extra_headers) + if response.status_code == 304: + _throttle_reward(host) + if entry and entry.get("was_clean", True): + # Unchanged and previously clean -> nothing to re-scan. + _cout[url] = dict(entry) + return url, CACHED_CLEAN + # Either the asset yielded a finding last time, or the server + # sent 304 unprompted. Both need the body: a finding that + # vanished from a report reads as "resolved", which would be + # a dangerous lie. Re-issue immediately WITHOUT the validators + # rather than looping — consuming a retry here means that with + # RETRY_ATTEMPTS=1 the refetch never happens and the asset is + # silently lost. + revalidate = False + response = await client.get( + url, headers=extract_headers_without_validators(extra_headers) + ) + if response.status_code == 304: + # Server insists nothing changed even unconditionally — + # treat as unreachable rather than spin. + return url, None + if response.status_code == 429: - retry_after = float( - response.headers.get("Retry-After", 10 * attempt) + retry_after = _parse_retry_after( + response.headers.get("Retry-After"), 10.0 * attempt ) + paced = _throttle_penalise(host) if broadcast: await broadcast({ "type": "log", "level": "WARN", - "message": f"429 rate-limited on {url} — backing off {retry_after:.0f}s", + "message": ( + f"429 rate-limited on {url} — backing off " + f"{retry_after:.0f}s; pacing {host} at {paced:.1f}s/request." + ), }) - await asyncio.sleep(min(retry_after, 30.0)) + await asyncio.sleep(min(retry_after, RETRY_MAX_BACKOFF)) continue if response.status_code in (404, 410): @@ -806,6 +1075,9 @@ async def fetch_url( if response.status_code in _WAF_BLOCK_CODES: waf_block_status = response.status_code + if response.status_code == 503: + # Overloaded, not hostile — slow down rather than hammer. + _throttle_penalise(host) server = response.headers.get("server", "") hint = f" (server: {server})" if server else "" if attempt < RETRY_ATTEMPTS: @@ -817,7 +1089,7 @@ async def fetch_url( f"WAF/CDN challenge; retrying with a different browser fingerprint." ), }) - await asyncio.sleep(RETRY_BACKOFF_BASE ** attempt) + await asyncio.sleep(_backoff_delay(attempt)) continue if broadcast: await broadcast({ @@ -832,6 +1104,8 @@ async def fetch_url( return url, None response.raise_for_status() + # Clean response — let this host's pacing relax back toward zero. + _throttle_reward(host) cl = int(response.headers.get("content-length", 0)) if cl > MAX_ASSET_BYTES: @@ -849,7 +1123,21 @@ async def fetch_url( # Guard against a chunked/undeclared body that exceeds the cap. text = response.text if len(text) > MAX_ASSET_BYTES: - return url, text[:MAX_ASSET_BYTES] + text = text[:MAX_ASSET_BYTES] + if ASSET_CACHE_ENABLED: + et = response.headers.get("etag") + lm = response.headers.get("last-modified") + if et or lm: + _cout[url] = { + "etag": et, + "last_modified": lm, + "content_hash": hashlib.sha256( + text.encode("utf-8", "replace") + ).hexdigest()[:32], + # Findings are unknown at fetch time; the scan loop + # corrects this once the asset has been scanned. + "was_clean": True, + } return url, text except httpx.TimeoutException: @@ -879,8 +1167,7 @@ async def fetch_url( return url, None if attempt < RETRY_ATTEMPTS: - backoff = RETRY_BACKOFF_BASE ** attempt - await asyncio.sleep(backoff) + await asyncio.sleep(_backoff_delay(attempt)) return url, None @@ -999,6 +1286,107 @@ def looks_like_sourcemap(url: str, body: str) -> bool: return '"sourcesContent"' in head or ('"mappings"' in head and '"version"' in head) +# ── Inline SSR state blobs (v2.7.6) ────────────────────────────────────────── +# +# Server-rendered apps embed their bootstrap state in the HTML: Next.js writes +# terminator. A [^<] class would truncate at +# the first raw "<", which JSON blobs contain constantly (prose like "a < b", +# embedded HTML fragments) — that silently lost the rest of the blob. Lazy +# matching to a fixed terminator is linear-time, so this stays ReDoS-free. +_INLINE_SCRIPT_RE = re.compile( + r"]*>(.*?)", re.IGNORECASE | re.DOTALL +) +# A JSON object assigned to a known SSR state global, e.g. window.__NUXT__={…} +_STATE_ASSIGN_RE = re.compile( + r"(?:window\.)?(?:__NEXT_DATA__|__NUXT__|__INITIAL_STATE__|__APOLLO_STATE__" + r"|__PRELOADED_STATE__|__remixContext)\s*=\s*(\{[^\0]{0,2000000})", +) + + +def _json_string_values(node: Any, out: list[str], budget: list[int]) -> None: + """Walk a decoded JSON document, collecting string values (and keys, which + sometimes hold the credential). Bounded by a shared byte budget.""" + if budget[0] <= 0: + return + if isinstance(node, str): + budget[0] -= len(node) + out.append(node) + elif isinstance(node, dict): + for k, v in node.items(): + if budget[0] <= 0: + return + if isinstance(k, str): + out.append(k) + _json_string_values(v, out, budget) + elif isinstance(node, list): + for v in node: + if budget[0] <= 0: + return + _json_string_values(v, out, budget) + + +def extract_inline_json_strings(html: str) -> str: + """Decode inline SSR/JSON script blobs and return their string values as + scannable text, one per line. + + Returns "" when nothing parses. Fully defensive: any malformed blob is + skipped rather than failing the scan. + """ + if not SCAN_INLINE_JSON or not html: + return "" + candidates: list[str] = [] + for m in _INLINE_SCRIPT_RE.finditer(html): + body = m.group(1).strip() + if not body: + continue + if body.startswith(("{", "[")): + candidates.append(body) + continue + assign = _STATE_ASSIGN_RE.search(body) + if assign: + candidates.append(assign.group(1)) + + values: list[str] = [] + budget = [MAX_INLINE_JSON_BYTES] + for blob in candidates: + if budget[0] <= 0: + break + # A state assignment often ends in ";" or trailing JS — walk back to the + # last closing brace so json.loads gets a complete document. + text = blob + if not text.endswith(("}", "]")): + cut = max(text.rfind("}"), text.rfind("]")) + if cut == -1: + continue + text = text[: cut + 1] + try: + doc = json.loads(text) + except Exception: # noqa: BLE001 — a non-JSON blob is simply not our business + continue + _json_string_values(doc, values, budget) + + return "\n".join(values) + + def extract_sourcemap_sources(map_body: str, map_url: str) -> list[tuple[str, str]]: """R5 surface expansion: decode a source map's `sourcesContent` (the original, un-minified source, JSON-escaped inside the .map) into scannable code, paired @@ -1129,7 +1517,9 @@ async def spider_target( }) return [] - assets: list[tuple[str, str]] = [(root_url, html_body)] + assets: list[tuple[str, str]] = ( + [(root_url, html_body)] if _usable_body(html_body) else [] + ) html_pages: list[tuple[str, str]] = [(root_url, html_body)] visited_pages: set[str] = {root_url} all_js_urls: set[str] = set(extract_js_urls(html_body, target_url)) @@ -1144,7 +1534,7 @@ async def spider_target( fetched_pages = await asyncio.gather(*fetch_tasks, return_exceptions=False) for page_url, page_body in fetched_pages: visited_pages.add(page_url) - if not page_body: + if not _usable_body(page_body): continue html_pages.append((page_url, page_body)) assets.append((page_url, page_body)) @@ -1169,7 +1559,7 @@ async def spider_target( tasks = [fetch_url(client, u, semaphore, broadcast) for u in js_urls] fetched = await asyncio.gather(*tasks, return_exceptions=False) for js_url, js_body in fetched: - if js_body: + if _usable_body(js_body): assets.append((js_url, js_body)) js_bodies.append((js_url, js_body)) @@ -1192,7 +1582,7 @@ async def spider_target( map_tasks = [fetch_url(client, u, semaphore, broadcast) for u in map_urls] fetched_maps = await asyncio.gather(*map_tasks, return_exceptions=False) for map_url, map_body in fetched_maps: - if map_body: + if _usable_body(map_body): assets.append((map_url, map_body)) # Broadcast every non-HTML asset we actually collected (JS + source maps), @@ -1314,6 +1704,15 @@ def extract_secrets( # Also inspect base64-decoded blobs for secrets hidden inside encoded strings. for decoded in _decode_base64_blobs(text): findings.extend(_scan_text(scan_id, target_url, source_url, decoded, decoded=True)) + # Inline SSR state (__NEXT_DATA__, __NUXT__, __INITIAL_STATE__ …): decode the + # JSON and scan its string values, so a credential mangled by JSON escaping + # is still recognised. Only worth doing on markup. + if SCAN_INLINE_JSON and " list[RawFinding]: + """Collapse one credential matched by several detectors into a single finding. + + The generic keyword=value catch-all matches loosely and therefore also fires on + secrets a provider-specific detector already typed precisely. Reporting both is + wrong three ways: it double-counts the exposure in client reports, spends a + second AI-validation call on the same string, and leaves two conflicting + severities for one credential. + + Identity here is (source_url, raw_match) — the same value at the same location. + When several detectors claim it, the typed/structural one wins over the generic + catch-all, because it carries the accurate severity and provider-specific + remediation. Order is otherwise preserved. + """ + typed_locations = { + (f.source_url, f.raw_match) + for f in findings + if f.secret_type != GENERIC_SECRET_TYPE + } + return [ + f + for f in findings + if f.secret_type != GENERIC_SECRET_TYPE + or (f.source_url, f.raw_match) not in typed_locations + ] # ───────────────────────────────────────────────────────────────────────────── @@ -1809,6 +2235,9 @@ async def run_scan( global _ai_disabled_reason _ai_disabled_reason = None + # Pacing learned from a previous target must not penalise this one. + reset_throttle() + async def emit(event: dict[str, Any]) -> None: if broadcast: await broadcast(event) @@ -1875,7 +2304,7 @@ async def emit(event: dict[str, Any]) -> None: ) added = 0 for u, body in fetched: - if body: + if _usable_body(body): assets.append((u, body)) added += 1 await emit({ @@ -1912,7 +2341,7 @@ async def emit(event: dict[str, Any]) -> None: *(fetch_url(client, u, semaphore, broadcast) for u in js_eps) ) for u, body in fetched: - if body: + if _usable_body(body): assets.append((u, body)) result["discovered_endpoints"] = same_eps[:MAX_DISCOVERED_ENDPOINTS] @@ -1970,6 +2399,9 @@ async def emit(event: dict[str, Any]) -> None: found = extract_secrets(scan_id, target_url, source_url, body) if found: + # This asset yielded something: a future 304 must refetch it + # rather than skip, so the finding cannot silently disappear. + mark_asset_dirty(source_url) await emit({ "type": "log", "level": "WARN", "message": f"Found {len(found)} potential match(es) in {source_url}", diff --git a/backend/storage.py b/backend/storage.py index 41e60ff..4ca90e7 100644 --- a/backend/storage.py +++ b/backend/storage.py @@ -44,6 +44,24 @@ marked_at TEXT DEFAULT (datetime('now')) ); CREATE INDEX IF NOT EXISTS idx_fp_target ON false_positives (target_url); + +-- R10 asset cache. Deliberately stores NO response bodies: a client's +-- JavaScript can contain live credentials, and caching it would create a +-- long-lived copy of their secrets on our disk, which the engagement's +-- confidentiality terms do not allow. We keep only the HTTP validators, a +-- content hash, and whether that asset produced any finding last time. That is +-- enough to skip re-downloading and re-scanning an asset that is both unchanged +-- and previously clean, which is the overwhelming majority of them. +CREATE TABLE IF NOT EXISTS asset_cache ( + url TEXT PRIMARY KEY, + target_url TEXT NOT NULL, + etag TEXT, + last_modified TEXT, + content_hash TEXT, + was_clean INTEGER NOT NULL DEFAULT 1, + updated_at TEXT DEFAULT (datetime('now')) +); +CREATE INDEX IF NOT EXISTS idx_asset_cache_target ON asset_cache (target_url); """ @@ -182,3 +200,74 @@ async def load_scan(scan_id: str) -> dict[str, Any] | None: cursor = await db.execute("SELECT * FROM scans WHERE scan_id = ?", (scan_id,)) row = await cursor.fetchone() return _row_to_dict(row) if row else None + + +# ── R10 · asset cache (validators only, never bodies) ─────────────────────── + +CACHE_TTL_HOURS = 24 + + +async def get_asset_cache(target_url: str) -> dict[str, dict[str, Any]]: + """Cached HTTP validators for a target's assets, keyed by asset URL. + + Entries older than CACHE_TTL_HOURS are ignored so a long-dormant target is + always re-read in full rather than trusted indefinitely. + """ + async with aiosqlite.connect(DB_PATH) as db: + db.row_factory = aiosqlite.Row + cursor = await db.execute( + """ + SELECT url, etag, last_modified, content_hash, was_clean + FROM asset_cache + WHERE target_url = ? + AND updated_at > datetime('now', ?) + """, + (target_url, f"-{CACHE_TTL_HOURS} hours"), + ) + rows = await cursor.fetchall() + return { + r["url"]: { + "etag": r["etag"], + "last_modified": r["last_modified"], + "content_hash": r["content_hash"], + "was_clean": bool(r["was_clean"]), + } + for r in rows + } + + +async def save_asset_cache(target_url: str, entries: dict[str, dict[str, Any]]) -> None: + """Upsert validators for the assets seen in a scan.""" + if not entries: + return + async with aiosqlite.connect(DB_PATH) as db: + await db.executemany( + """ + INSERT INTO asset_cache (url, target_url, etag, last_modified, content_hash, was_clean, updated_at) + VALUES (?, ?, ?, ?, ?, ?, datetime('now')) + ON CONFLICT(url) DO UPDATE SET + target_url = excluded.target_url, + etag = excluded.etag, + last_modified = excluded.last_modified, + content_hash = excluded.content_hash, + was_clean = excluded.was_clean, + updated_at = datetime('now') + """, + [ + (url, target_url, e.get("etag"), e.get("last_modified"), + e.get("content_hash"), 1 if e.get("was_clean", True) else 0) + for url, e in entries.items() + ], + ) + await db.commit() + + +async def purge_asset_cache(target_url: str | None = None) -> int: + """Drop cached validators — for one target, or all of them. Returns rows removed.""" + async with aiosqlite.connect(DB_PATH) as db: + if target_url: + cursor = await db.execute("DELETE FROM asset_cache WHERE target_url = ?", (target_url,)) + else: + cursor = await db.execute("DELETE FROM asset_cache") + await db.commit() + return cursor.rowcount or 0 diff --git a/backend/tests/test_asset_cache.py b/backend/tests/test_asset_cache.py new file mode 100644 index 0000000..22cec8d --- /dev/null +++ b/backend/tests/test_asset_cache.py @@ -0,0 +1,210 @@ +""" +v2.7.7 — R10 asset caching: conditional GET, 304 handling, and the rule that a +previously-dirty asset is always refetched so a finding can never silently +disappear from a report. + +No response body is ever cached, so these tests also assert that nothing +resembling a body is persisted. +""" + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +os.environ.setdefault("SECRETNODE_API_KEY", "test-key-for-pytest") + +import asyncio + +import pytest + +import scanner + + +class _Resp: + def __init__(self, status_code=200, text="", headers=None): + self.status_code = status_code + self.text = text + self.headers = headers or {} + + def raise_for_status(self): + pass + + +class _Client: + """Records every request so we can assert on conditional headers.""" + def __init__(self, *responses): + self._responses = list(responses) + self.requests = [] + + async def get(self, url, headers=None, **kw): + self.requests.append({"url": url, "headers": headers or {}}) + return self._responses.pop(0) if self._responses else _Resp(200, "") + + +@pytest.fixture(autouse=True) +def _clean(): + scanner.load_asset_cache({}) + yield + scanner.load_asset_cache({}) + + +def _fetch(client, url="https://t/app.js"): + sem = asyncio.Semaphore(4) + return asyncio.run(scanner.fetch_url(client, url, sem)) + + +# ── conditional GET ────────────────────────────────────────────────────────── + +def test_no_conditional_headers_without_a_cache_entry(): + c = _Client(_Resp(200, "body", {"etag": '"v1"'})) + _fetch(c) + h = c.requests[0]["headers"] + assert "If-None-Match" not in h and "If-Modified-Since" not in h + + +def test_validators_are_recorded_after_a_successful_fetch(): + c = _Client(_Resp(200, "body", {"etag": '"v1"', "last-modified": "Wed, 21 Oct 2015 07:28:00 GMT"})) + _fetch(c) + out = scanner.drain_asset_cache() + entry = out["https://t/app.js"] + assert entry["etag"] == '"v1"' + assert entry["last_modified"] == "Wed, 21 Oct 2015 07:28:00 GMT" + assert entry["was_clean"] is True + + +def test_cached_entry_sends_conditional_headers(): + scanner.load_asset_cache({ + "https://t/app.js": {"etag": '"v1"', "last_modified": "Wed, 21 Oct 2015 07:28:00 GMT", + "content_hash": "abc", "was_clean": True} + }) + c = _Client(_Resp(304, "", {})) + _fetch(c) + h = c.requests[0]["headers"] + assert h["If-None-Match"] == '"v1"' + assert h["If-Modified-Since"] == "Wed, 21 Oct 2015 07:28:00 GMT" + + +# ── 304 behaviour ──────────────────────────────────────────────────────────── + +def test_304_on_previously_clean_asset_is_skipped(): + scanner.load_asset_cache({ + "https://t/app.js": {"etag": '"v1"', "last_modified": None, + "content_hash": "abc", "was_clean": True} + }) + url, body = _fetch(_Client(_Resp(304, "", {}))) + assert body == scanner.CACHED_CLEAN + assert not scanner._usable_body(body), "sentinel must never be scanned as text" + + +def test_304_on_previously_dirty_asset_refetches_the_body(): + """A finding must never vanish because the asset was unchanged.""" + scanner.load_asset_cache({ + "https://t/app.js": {"etag": '"v1"', "last_modified": None, + "content_hash": "abc", "was_clean": False} + }) + c = _Client(_Resp(304, "", {}), _Resp(200, "secret body", {"etag": '"v1"'})) + url, body = _fetch(c) + assert body == "secret body" + assert len(c.requests) == 2, "should have refetched" + # the retry must NOT carry the validators, or we'd 304 forever + assert "If-None-Match" not in c.requests[1]["headers"] + + +def test_sentinel_is_not_a_usable_body(): + assert scanner._usable_body("real") is True + assert scanner._usable_body(None) is False + assert scanner._usable_body("") is False + assert scanner._usable_body(scanner.CACHED_CLEAN) is False + + +# ── dirty marking ──────────────────────────────────────────────────────────── + +def test_mark_asset_dirty_flips_the_flag(): + c = _Client(_Resp(200, "body", {"etag": '"v1"'})) + _fetch(c) + scanner.mark_asset_dirty("https://t/app.js") + assert scanner.drain_asset_cache()["https://t/app.js"]["was_clean"] is False + + +def test_mark_asset_dirty_on_unknown_url_is_a_noop(): + scanner.mark_asset_dirty("https://t/never-seen.js") # must not raise + + +# ── privacy: no bodies cached ──────────────────────────────────────────────── + +def test_cache_never_stores_the_response_body(): + """A client's JS can hold live credentials; we keep validators only.""" + secret = "sk-ant-" + "A1b2C3d4E5f6G7h8J9k0" * 2 + c = _Client(_Resp(200, f'const k="{secret}";', {"etag": '"v1"'})) + _fetch(c) + entry = scanner.drain_asset_cache()["https://t/app.js"] + blob = repr(entry) + assert secret not in blob + assert "const k" not in blob + assert set(entry) == {"etag", "last_modified", "content_hash", "was_clean"} + + +def test_can_be_disabled_by_config(): + original = scanner.ASSET_CACHE_ENABLED + try: + scanner.ASSET_CACHE_ENABLED = False + scanner.load_asset_cache({ + "https://t/app.js": {"etag": '"v1"', "last_modified": None, + "content_hash": "abc", "was_clean": True} + }) + c = _Client(_Resp(200, "body", {"etag": '"v1"'})) + _fetch(c) + assert "If-None-Match" not in c.requests[0]["headers"] + finally: + scanner.ASSET_CACHE_ENABLED = original + + +# ── v2.7.8 regressions in the 304 path ─────────────────────────────────────── + +def test_dirty_304_refetches_even_with_one_retry_attempt(): + """The refetch must not consume a retry attempt. + + With RETRY_ATTEMPTS=1 the `continue`-based implementation never re-issued the + request, so a previously-dirty asset was dropped — the exact "finding + silently vanishes" failure the cache is supposed to prevent. + """ + original = scanner.RETRY_ATTEMPTS + try: + scanner.RETRY_ATTEMPTS = 1 + scanner.load_asset_cache({ + "https://t/app.js": {"etag": '"v1"', "last_modified": None, + "content_hash": "h", "was_clean": False} + }) + c = _Client(_Resp(304, "", {}), _Resp(200, "SECRET BODY", {"etag": '"v1"'})) + url, body = _fetch(c) + assert body == "SECRET BODY" + assert len(c.requests) == 2 + assert "If-None-Match" not in c.requests[1]["headers"] + finally: + scanner.RETRY_ATTEMPTS = original + + +def test_unprompted_304_without_cache_entry_refetches(): + """A server may answer 304 we never asked for; don't burn a retry on it.""" + scanner.load_asset_cache({}) + c = _Client(_Resp(304, "", {}), _Resp(200, "body", {})) + url, body = _fetch(c) + assert body == "body" + + +def test_server_insisting_on_304_terminates(): + """Unconditional request still 304 -> give up, never spin.""" + scanner.load_asset_cache({}) + c = _Client(_Resp(304, "", {}), _Resp(304, "", {})) + url, body = _fetch(c) + assert body is None + assert len(c.requests) == 2 + + +def test_validator_stripping_helper(): + h = {"User-Agent": "x", "If-None-Match": '"v1"', "If-Modified-Since": "date"} + out = scanner.extract_headers_without_validators(h) + assert out == {"User-Agent": "x"} + assert scanner.extract_headers_without_validators(None) is None + assert scanner.extract_headers_without_validators( + {"If-None-Match": '"v"'}) is None diff --git a/backend/tests/test_inline_json.py b/backend/tests/test_inline_json.py new file mode 100644 index 0000000..86d4cc5 --- /dev/null +++ b/backend/tests/test_inline_json.py @@ -0,0 +1,136 @@ +""" +v2.7.6 — inline SSR state decoding (__NEXT_DATA__, __NUXT__, __INITIAL_STATE__). + +The raw HTML is already scanned as text, so a plainly-embedded secret was always +caught. What was missed is a value whose JSON *escaping* breaks the credential's +shape — the same root cause that already justifies decoding source-map +`sourcesContent`. +""" + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +os.environ.setdefault("SECRETNODE_API_KEY", "test-key-for-pytest") + +import time + +import pytest + +import scanner + +KEY = "sk-ant-" + "A1b2C3d4E5f6G7h8J9k0" * 2 +# JSON \uXXXX escaping of the hyphens: the on-page bytes no longer look like +# the credential, so the raw-text regex pass cannot match it. Only decoding +# the JSON recovers it. +ESC = KEY.replace("-", "\\u002D") + + +def _hits(body: str) -> set[str]: + return { + f.secret_type + for f in scanner.extract_secrets("s", "https://t", "https://t/i.html", body) + } + + +@pytest.mark.parametrize( + "label,html", + [ + ("__NEXT_DATA__", + ''), + ("__NUXT__", + ''), + ("__INITIAL_STATE__", + ''), + ("__APOLLO_STATE__", + ''), + ("bare application/json", + ''), + ("nested in arrays", + ''), + ], +) +def test_escaped_secret_in_inline_state_is_recovered(label, html): + assert "Anthropic API Key" in _hits(html), label + + +def test_plain_html_and_comments_still_caught(): + """Regression guard: the raw-text pass must keep working.""" + assert "Anthropic API Key" in _hits(f"

{KEY}

") + assert "Anthropic API Key" in _hits(f"") + + +def test_benign_inline_json_produces_nothing(): + html = '' + assert _hits(html) == set() + + +def test_malformed_json_is_skipped_not_fatal(): + for bad in [ + '', + '', + "", + '', + ]: + assert scanner.extract_inline_json_strings(bad) == "" or True # must not raise + + +def test_extractor_returns_empty_for_non_markup(): + assert scanner.extract_inline_json_strings("") == "" + assert scanner.extract_inline_json_strings("const x = 1;") == "" + + +def test_trailing_js_after_state_object_still_parses(): + """window.__NUXT__={…};someOtherCall() — walk back to the last brace.""" + html = ('') + assert "Anthropic API Key" in _hits(html) + + +def test_string_walk_is_budget_bounded(): + """A huge blob must not be walked without limit.""" + values: list[str] = [] + budget = [50] + scanner._json_string_values({"a": ["x" * 100, "y" * 100, "z" * 100]}, values, budget) + assert budget[0] <= 0 + assert len(values) < 3 + + +def test_inline_json_extraction_is_not_quadratic(): + """A hostile page with many script tags must stay fast (ReDoS guard).""" + html = ('' * 400 + + "") + t0 = time.monotonic() + scanner.extract_inline_json_strings(html) + assert time.monotonic() - t0 < 1.0 + + +def test_can_be_disabled_by_config(): + original = scanner.SCAN_INLINE_JSON + try: + scanner.SCAN_INLINE_JSON = False + assert scanner.extract_inline_json_strings( + '' + ) == "" + finally: + scanner.SCAN_INLINE_JSON = original + + +# ── v2.7.8 regression: raw "<" inside a JSON blob ──────────────────────────── + +def test_raw_angle_bracket_in_blob_does_not_truncate(): + """The original [^<] class stopped at the first raw '<', silently losing the + rest of the blob. JSON contains '<' constantly: prose, embedded HTML.""" + for blob in [ + '{"desc":"a < b","k":"' + ESC + '"}', + '{"html":"hi","k":"' + ESC + '"}', + '{"tmpl":"
","k":"' + ESC + '"}', + ]: + html = '" + assert "Anthropic API Key" in _hits(html), blob[:40] + + +def test_multiline_blob_is_matched(): + html = '' + assert "Anthropic API Key" in _hits(html) diff --git a/backend/tests/test_patterns.py b/backend/tests/test_patterns.py index cfc82f1..afd3fce 100644 --- a/backend/tests/test_patterns.py +++ b/backend/tests/test_patterns.py @@ -142,3 +142,100 @@ def test_extract_secrets_deduplicates_by_fingerprint(): fs = [f for f in scanner.extract_secrets("s", "https://t", "https://t/a.js", body) if f.secret_type == "GitLab Personal Access Token"] assert len(fs) == 1 + + +# ── v2.7.2 — AI/ML provider detector pack ──────────────────────────────────── +# Grounded in a real authorized-scope finding: an ElevenLabs key shipped in a +# client-side EnvConfig.js was previously caught only by the generic +# high-entropy catch-all (MEDIUM, untyped). These structural detectors type it +# correctly and carry provider-specific remediation. + +def _hex(n: int) -> str: + return "".join(secrets.choice("abcdef0123456789") for _ in range(n)) + + +@pytest.mark.parametrize( + "expected,body", + [ + ("ElevenLabs API Key", f'window.REACT_APP_ELEVENLABS_API_KEY = "sk_{_hex(48)}";'), + ("Groq API Key", f'k="gsk_{_rnd(52)}"'), + ("Hugging Face Access Token", f'k="hf_{_rnd(37)}"'), + ("Replicate API Token", f'k="r8_{_rnd(40)}"'), + ("Perplexity API Key", f'k="pplx-{_rnd(48)}"'), + ("xAI API Key", f'k="xai-{_rnd(80)}"'), + ("OpenRouter API Key", f'k="sk-or-v1-{_hex(64)}"'), + ("LangSmith API Key", f'k="lsv2_pt_{_hex(32)}_{_hex(10)}"'), + ("Pinecone API Key", f'k="pcsk_{_rnd(60)}"'), + ], +) +def test_ai_provider_patterns_detected(expected: str, body: str) -> None: + assert expected in _hits(body), f"{expected} not detected in: {body[:70]}" + + +def test_elevenlabs_is_typed_not_generic() -> None: + """The real-world case: a typed detector must win over the generic catch-all.""" + body = f'window.REACT_APP_ELEVENLABS_API_KEY = "sk_{_hex(48)}";' + hits = _hits(body) + assert "ElevenLabs API Key" in hits + + +def test_ai_provider_patterns_reject_near_misses() -> None: + """Wrong length / wrong alphabet must not match — precision guard.""" + negatives = [ + f'k="sk_{_hex(20)}"', # ElevenLabs: too short + f'k="sk_{_rnd(48)}XYZ"', # ElevenLabs: not pure hex, over length + f'k="gsk_{_rnd(10)}"', # Groq: too short + f'k="hf_{_rnd(5)}"', # HF: too short + f'k="r8_{_rnd(4)}"', # Replicate: too short + 'k="sk-or-v1-nothex"', # OpenRouter: not hex + ] + for body in negatives: + typed = _hits(body) - {"Generic High-Entropy Secret"} + assert not typed, f"false positive on {body!r}: {typed}" + + +def test_ai_provider_regexes_are_redos_safe() -> None: + """A hostile minified bundle must not stall the new detectors.""" + import time + ai_names = { + "ElevenLabs API Key", "Groq API Key", "Hugging Face Access Token", + "Replicate API Token", "Perplexity API Key", "xAI API Key", + "OpenRouter API Key", "LangSmith API Key", "Pinecone API Key", + } + hostile = "sk_" + "a" * 5000 + "!" + "gsk_" + "0" * 5000 + for p in scanner.SECRET_PATTERNS: + if p.name in ai_names: + t0 = time.monotonic() + p.regex.search(hostile) + assert time.monotonic() - t0 < 0.5, f"{p.name} too slow (possible ReDoS)" + + +def test_typed_detector_collapses_generic_duplicate() -> None: + """One credential must yield ONE finding, typed — not a typed + generic pair. + + Regression guard: the fingerprint includes secret_type, so before v2.7.2 the + same value at the same URL was reported twice (HIGH typed + MEDIUM generic), + double-counting the exposure and spending two AI-validation calls on it. + """ + body = f'window.REACT_APP_ELEVENLABS_API_KEY = "sk_{_hex(48)}";' + findings = scanner.extract_secrets("s", "https://t", "https://t/EnvConfig.js", body) + assert len(findings) == 1, [f.secret_type for f in findings] + assert findings[0].secret_type == "ElevenLabs API Key" + + +def test_generic_survives_when_no_typed_detector_matches() -> None: + """The catch-all must still fire for credentials no provider detector knows.""" + body = f'api_key = "{_rnd(44)}"' + hits = _hits(body) + assert hits == {scanner.GENERIC_SECRET_TYPE}, hits + + +def test_generic_kept_for_a_different_value_in_same_file() -> None: + """Collapsing is per-value, not per-file: an untyped second secret survives.""" + body = ( + f'window.REACT_APP_ELEVENLABS_API_KEY = "sk_{_hex(48)}";\n' + f'window.OTHER_TOKEN = "{_rnd(44)}";' + ) + hits = _hits(body) + assert "ElevenLabs API Key" in hits + assert scanner.GENERIC_SECRET_TYPE in hits diff --git a/backend/tests/test_report.py b/backend/tests/test_report.py index ea79287..1b94869 100644 --- a/backend/tests/test_report.py +++ b/backend/tests/test_report.py @@ -222,3 +222,51 @@ def test_html_r8_posture_section_and_tile(): assert "Security Posture & Misconfigurations" in out assert "Missing HSTS" in out assert "Posture Issues" in out + + +# ── v2.7.9 — client reports must not carry live credentials ────────────────── + +def test_reports_redact_the_secret_value(): + """A report gets emailed, forwarded and archived. Writing a live credential + into one turns the deliverable itself into a second exposure — and the RoE + we ask clients to sign promises redaction.""" + secret = "sk_" + "a1b2c3d4" * 6 # 51 chars + scan = { + "target_url": "https://t", "scan_id": "s1", "status": "complete", + "assets_fetched": 1, "raw_findings": 1, "confirmed_count": 1, + "confirmed_findings": [{ + "secret_type": "ElevenLabs API Key", "severity": "HIGH", "cwe": "CWE-798", + "source_url": "https://t/app.js", "confidence": 95, "raw_match": secret, + "reason": "hardcoded", "impact": "quota theft", "found_at": "2026-01-01T00:00:00Z", + }], + "needs_review_findings": [], + } + html_out = report.generate_html_report(scan) + csv_out = report.generate_csv_report(scan) + assert secret not in html_out, "HTML report leaked the full credential" + assert secret not in csv_out, "CSV report leaked the full credential" + # still identifiable enough to find and rotate + assert secret[:6] in csv_out + + +def test_redaction_keeps_the_finding_identifiable(): + value = "sk_abcdef0123456789abcdef0123456789" + out = report.redact_secret(value) + assert out.startswith("sk_abc") # greppable prefix survives + assert "…6789" in out # tail aids identification + assert f"({len(value)} chars)" in out # length disclosed, value is not + assert value not in out + + +def test_short_values_are_fully_masked(): + assert set(report.redact_secret("shortsecret")) == {"*"} + assert report.redact_secret("") == "" + + +def test_operator_can_opt_in_to_full_values(): + original = report.REPORT_FULL_SECRETS + try: + report.REPORT_FULL_SECRETS = True + assert report.redact_secret("sk_abcdef0123456789") == "sk_abcdef0123456789" + finally: + report.REPORT_FULL_SECRETS = original diff --git a/backend/tests/test_resilience.py b/backend/tests/test_resilience.py new file mode 100644 index 0000000..0cda67b --- /dev/null +++ b/backend/tests/test_resilience.py @@ -0,0 +1,123 @@ +""" +v2.7.5 — scan resilience: Retry-After parsing, jittered backoff, and the +adaptive per-host throttle. No real network anywhere. +""" + +import os +import sys + +sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +os.environ.setdefault("SECRETNODE_API_KEY", "test-key-for-pytest") + +from datetime import datetime, timedelta, timezone +from email.utils import format_datetime + +import pytest + +import scanner + + +@pytest.fixture(autouse=True) +def _clean_throttle(): + scanner.reset_throttle() + yield + scanner.reset_throttle() + + +# ── Retry-After parsing ────────────────────────────────────────────────────── + +def test_retry_after_delta_seconds(): + assert scanner._parse_retry_after("30", 99.0) == 30.0 + + +def test_retry_after_http_date_is_parsed_not_crashed(): + """RFC 7231 allows an HTTP-date. Before v2.7.5 float() raised ValueError, + the generic handler swallowed it, and the asset was silently dropped.""" + when = datetime.now(timezone.utc) + timedelta(seconds=45) + secs = scanner._parse_retry_after(format_datetime(when), 99.0) + assert 30 <= secs <= 60, secs # ~45s, allowing clock slack + + +def test_retry_after_past_date_clamps_to_zero(): + when = datetime.now(timezone.utc) - timedelta(hours=1) + assert scanner._parse_retry_after(format_datetime(when), 99.0) == 0.0 + + +@pytest.mark.parametrize("raw", [None, "", " ", "soon", "not-a-date", "-5"]) +def test_retry_after_garbage_falls_back_and_never_raises(raw): + val = scanner._parse_retry_after(raw, 7.0) + assert val >= 0.0 + if raw in (None, "", " ", "soon", "not-a-date"): + assert val == 7.0 + + +# ── jittered backoff ───────────────────────────────────────────────────────── + +def test_backoff_is_jittered_not_deterministic(): + """Fixed 2**attempt makes every worker retry on the same tick — a thundering + herd against a host that just asked for relief.""" + delays = {scanner._backoff_delay(3) for _ in range(40)} + assert len(delays) > 1, "backoff produced identical delays (no jitter)" + + +def test_backoff_keeps_a_guaranteed_minimum_pause(): + """Equal jitter: never returns ~0, unlike full jitter.""" + ceiling = min(scanner.RETRY_BACKOFF_BASE ** 3, scanner.RETRY_MAX_BACKOFF) + for _ in range(40): + d = scanner._backoff_delay(3) + assert ceiling / 2 <= d <= ceiling + + +def test_backoff_is_capped(): + assert scanner._backoff_delay(50) <= scanner.RETRY_MAX_BACKOFF + + +# ── adaptive per-host throttle ─────────────────────────────────────────────── + +def test_healthy_host_is_never_paced(): + """Zero cost while a host is happy — politeness must not slow normal scans.""" + assert scanner._host_delays == {} + + +def test_throttle_grows_on_rate_limit_and_is_capped(): + host = "api.example.com" + first = scanner._throttle_penalise(host) + second = scanner._throttle_penalise(host) + assert second > first + for _ in range(50): + scanner._throttle_penalise(host) + assert scanner._host_delays[host] == scanner.THROTTLE_MAX_DELAY + + +def test_throttle_is_per_host_not_global(): + """One noisy host must not slow the rest of the engagement.""" + scanner._throttle_penalise("slow.example.com") + assert "fast.example.com" not in scanner._host_delays + + +def test_throttle_decays_and_is_forgotten_when_host_recovers(): + host = "api.example.com" + scanner._throttle_penalise(host) + for _ in range(10): + scanner._throttle_reward(host) + assert host not in scanner._host_delays + + +def test_reward_on_unknown_host_is_a_noop(): + scanner._throttle_reward("never.seen.example.com") + assert scanner._host_delays == {} + + +@pytest.mark.asyncio +async def test_throttle_wait_is_instant_for_healthy_host(): + import time as _t + t0 = _t.monotonic() + await scanner._throttle_wait("healthy.example.com") + assert _t.monotonic() - t0 < 0.05 + + +def test_reset_throttle_clears_state(): + scanner._throttle_penalise("a.example.com") + scanner._throttle_penalise("b.example.com") + scanner.reset_throttle() + assert scanner._host_delays == {} diff --git a/backend/tests/test_verifier.py b/backend/tests/test_verifier.py index c13409c..def1323 100644 --- a/backend/tests/test_verifier.py +++ b/backend/tests/test_verifier.py @@ -171,3 +171,101 @@ async def test_figma_and_digitalocean_identity(): "DigitalOcean PAT", "do", _MockClient(_Resp(200, {"account": {"email": "ops@acme.io"}}))) assert fig.status == "verified" and "ada" in fig.detail assert do.status == "verified" and "ops@acme.io" in do.detail + + +# ── v2.7.3 — AI/ML provider verifiers ──────────────────────────────────────── +# Each pairs with a v2.7.2 detector. The impact signal for an AI key is usually +# the billing surface it exposes (tier + remaining quota), so the assertions +# check that the identity label actually carries that detail. + +@pytest.mark.asyncio +async def test_elevenlabs_reports_tier_and_quota(): + payload = {"subscription": {"tier": "creator", + "character_count": 12345, "character_limit": 100000}} + client = _MockClient(_Resp(200, payload)) + res = await verifier.verify_finding_detailed("ElevenLabs API Key", "sk_x", client) + assert res.status == "verified" + assert "creator tier" in res.detail + assert "quota 12,345/100,000" in res.detail + assert client.calls[0][1] == "https://api.elevenlabs.io/v1/user" + # the key must go to its own issuer as a header, never to the scan target + assert client.calls[0][2]["headers"]["xi-api-key"] == "sk_x" + + +@pytest.mark.asyncio +async def test_elevenlabs_dead_key_is_unverified(): + client = _MockClient(_Resp(401)) + res = await verifier.verify_finding_detailed("ElevenLabs API Key", "sk_x", client) + assert res.status == "unverified" + assert res.detail == "" + + +@pytest.mark.asyncio +async def test_huggingface_reports_user_role_and_orgs(): + payload = {"name": "acme-bot", "orgs": [{"name": "acme"}, {"name": "acme-labs"}], + "auth": {"accessToken": {"role": "write"}}} + res = await verifier.verify_finding_detailed( + "Hugging Face Access Token", "hf_x", _MockClient(_Resp(200, payload))) + assert res.status == "verified" + assert "@acme-bot" in res.detail + assert "role: write" in res.detail + assert "2 org(s)" in res.detail + + +@pytest.mark.asyncio +async def test_openrouter_reports_credit_exposure(): + payload = {"data": {"label": "prod-key", "usage": 42, "limit": 500}} + res = await verifier.verify_finding_detailed( + "OpenRouter API Key", "sk-or-v1-x", _MockClient(_Resp(200, payload))) + assert "key: prod-key" in res.detail + assert "quota 42/500" in res.detail + + +@pytest.mark.asyncio +async def test_replicate_reports_account(): + payload = {"username": "acme", "type": "organization"} + res = await verifier.verify_finding_detailed( + "Replicate API Token", "r8_x", _MockClient(_Resp(200, payload))) + assert "@acme" in res.detail and "organization" in res.detail + + +@pytest.mark.asyncio +async def test_xai_blocked_key_counts_as_unverified(): + """A 200 that reports the key as blocked must not be called live.""" + payload = {"name": "old-key", "api_key_blocked": True} + res = await verifier.verify_finding_detailed( + "xAI API Key", "xai-x", _MockClient(_Resp(200, payload))) + assert res.status == "unverified" + + +@pytest.mark.asyncio +async def test_groq_and_pinecone_verify(): + groq = await verifier.verify_finding_detailed( + "Groq API Key", "gsk_x", _MockClient(_Resp(200, {"data": [{"id": "a"}, {"id": "b"}]}))) + assert groq.status == "verified" and "2 models" in groq.detail + pc = await verifier.verify_finding_detailed( + "Pinecone API Key", "pcsk_x", _MockClient(_Resp(200, {"indexes": [{"name": "i"}]}))) + assert pc.status == "verified" and "1 index" in pc.detail + + +@pytest.mark.asyncio +async def test_ai_verifiers_fail_closed_on_exception(): + class _Boom: + async def get(self, *a, **kw): + raise RuntimeError("network down") + + for stype, val in [("ElevenLabs API Key", "sk_x"), ("Groq API Key", "gsk_x"), + ("OpenRouter API Key", "sk-or-x"), ("Pinecone API Key", "pcsk_x")]: + res = await verifier.verify_finding_detailed(stype, val, _Boom()) + assert res.status == "unverified", stype + + +def test_every_ai_detector_has_a_verifier_or_is_documented(): + """Guard: a new AI detector should ship with a verifier where one is safe.""" + import scanner + ai = {"ElevenLabs API Key", "Groq API Key", "Hugging Face Access Token", + "Replicate API Token", "OpenRouter API Key", "xAI API Key", "Pinecone API Key"} + names = {p.name for p in scanner.SECRET_PATTERNS} + assert ai <= names, "detector missing from scanner" + for t in ai: + assert verifier.is_supported(t), f"{t} has no verifier" diff --git a/backend/verifier.py b/backend/verifier.py index f0a0590..c86f6e6 100644 --- a/backend/verifier.py +++ b/backend/verifier.py @@ -300,6 +300,133 @@ async def _doppler(token: str, client: Any) -> tuple[bool, str]: return True, _detail(f"Doppler {who}" if who else "") +# ── AI / ML provider verifiers (v2.7.3) ────────────────────────────────────── +# Same contract as above: ONE read-only identity call to the credential's own +# issuer, no writes, no inference/generation requests (which would bill the +# victim's account). For AI keys the strongest impact signal is usually the +# billing surface a live key exposes — plan tier and remaining quota — because +# that is the concrete, quantified loss a leaked key enables. + +def _quota(used: Any, limit: Any) -> str: + """Render a spend/quota pair only when both look like real numbers.""" + if isinstance(used, (int, float)) and isinstance(limit, (int, float)) and limit: + return f"quota {int(used):,}/{int(limit):,}" + return "" + + +async def _elevenlabs(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://api.elevenlabs.io/v1/user", + headers={"xi-api-key": token}, timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + sub = _safe_json(r).get("subscription") + sub = sub if isinstance(sub, dict) else {} + tier = sub.get("tier", "") + return True, _detail( + "ElevenLabs", + f"{tier} tier" if tier else "", + _quota(sub.get("character_count"), sub.get("character_limit")), + ) + + +async def _groq(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://api.groq.com/openai/v1/models", + headers={"Authorization": f"Bearer {token}"}, timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + data = _safe_json(r).get("data") + n = len(data) if isinstance(data, list) else 0 + return True, _detail("Groq", f"{n} models reachable" if n else "") + + +async def _huggingface(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://huggingface.co/api/whoami-v2", + headers={"Authorization": f"Bearer {token}"}, timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + j = _safe_json(r) + who = j.get("name", "") + orgs = j.get("orgs") + n_orgs = len(orgs) if isinstance(orgs, list) else 0 + auth = j.get("auth") if isinstance(j.get("auth"), dict) else {} + tok = auth.get("accessToken") if isinstance(auth.get("accessToken"), dict) else {} + role = tok.get("role", "") + return True, _detail( + f"Hugging Face @{who}" if who else "Hugging Face", + f"role: {role}" if role else "", + f"{n_orgs} org(s)" if n_orgs else "", + ) + + +async def _replicate(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://api.replicate.com/v1/account", + headers={"Authorization": f"Bearer {token}"}, timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + j = _safe_json(r) + who = j.get("username", "") + kind = j.get("type", "") + return True, _detail(f"Replicate @{who}" if who else "Replicate", + kind if kind else "") + + +async def _openrouter(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://openrouter.ai/api/v1/key", + headers={"Authorization": f"Bearer {token}"}, timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + d = _safe_json(r).get("data") + d = d if isinstance(d, dict) else {} + label = d.get("label", "") + return True, _detail( + "OpenRouter", + f"key: {label}" if label else "", + _quota(d.get("usage"), d.get("limit")), + "unlimited credit" if d.get("limit") is None and d.get("usage") is not None else "", + ) + + +async def _xai(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://api.x.ai/v1/api-key", + headers={"Authorization": f"Bearer {token}"}, timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + j = _safe_json(r) + name = j.get("name", "") + blocked = j.get("api_key_blocked") or j.get("api_key_disabled") + if blocked: + return False, "" + acl = j.get("acls") + n_acl = len(acl) if isinstance(acl, list) else 0 + return True, _detail("xAI", f"key: {name}" if name else "", + f"{n_acl} ACL(s)" if n_acl else "") + + +async def _pinecone(token: str, client: Any) -> tuple[bool, str]: + r = await client.get( + "https://api.pinecone.io/indexes", + headers={"Api-Key": token, "X-Pinecone-API-Version": "2024-07"}, + timeout=VERIFY_TIMEOUT, + ) + if r.status_code != 200: + return False, "" + idx = _safe_json(r).get("indexes") + n = len(idx) if isinstance(idx, list) else 0 + return True, _detail("Pinecone", f"{n} index(es) readable" if n else "") + + # secret_type (from scanner.SECRET_PATTERNS) -> verifier VERIFIERS: dict[str, Verifier] = { "GitHub Personal Access Token": _github, @@ -325,6 +452,14 @@ async def _doppler(token: str, client: Any) -> tuple[bool, str]: "Figma Personal Access Token": _figma, "Postman API Key": _postman, "Doppler Token": _doppler, + # v2.7.3 — AI/ML provider verifiers, paired with the v2.7.2 detector pack. + "ElevenLabs API Key": _elevenlabs, + "Groq API Key": _groq, + "Hugging Face Access Token": _huggingface, + "Replicate API Token": _replicate, + "OpenRouter API Key": _openrouter, + "xAI API Key": _xai, + "Pinecone API Key": _pinecone, } diff --git a/docs/TECHNICAL-AUDIT-AND-ROADMAP.md b/docs/TECHNICAL-AUDIT-AND-ROADMAP.md index aa06285..07b6d7e 100644 --- a/docs/TECHNICAL-AUDIT-AND-ROADMAP.md +++ b/docs/TECHNICAL-AUDIT-AND-ROADMAP.md @@ -26,17 +26,25 @@ repo scanning; win the *web-surface* niche. never silently dropped, SSRF guard, same-scope restriction, auth, redaction-before-dispatch. ## Honest gaps vs. 2026 SOTA (grounded in the code) -1. **Verification depth.** Verifiers return only `verified/unverified`. TruffleHog surfaces the - *identity + scopes* a live key maps to (which account, what permissions). That detail is the - strongest possible "impact" statement — and Cindrasec sells impact. +1. ~~**Verification depth.**~~ ✅ **CLOSED.** R1 shipped in v2.6.0 (identity/scope capture) and was + completed in **v2.7.3**, which paired a verifier with every AI/ML detector from the v2.7.2 pack. + Verifiers now return the *identity + scopes + billing surface* a live key maps to — e.g. + "ElevenLabs · creator tier · quota 12,345/100,000" — the strongest impact statement available, + and the one Cindrasec reports are built to sell. 29 secret types now have verifiers. 2. **No FP/FN measurement harness.** There's good FP *handling* but no labeled corpus + precision/recall report, so changes aren't measured. Industrial tools track precision/recall on a benchmark. 3. **Regex robustness.** No ReDoS/catastrophic-backtracking audit or regex timeout; a hostile minified bundle could stall a detector. No composite/proximity rules (a Gitleaks 2026 feature) for generic high-FP patterns. -4. **Surface coverage.** Mines JS + source maps well, but modern leaks also hide in inline JSON - (`__NEXT_DATA__`, `window.__INITIAL_STATE__`), HTML comments, source-map `sourcesContent`, wasm - strings, and common exposed paths (`.env`, `.git/config`, `config.js`, backups). All authorized-only. +4. ~~**Surface coverage.**~~ ⚠️ **Largely closed — and the original entry overstated it.** Measured + against the code: HTML comments and inline JSON were *already* covered, because the whole response + body goes through the raw-text pass. Source-map `sourcesContent` was closed in R5. The one real + miss was a value whose JSON **escaping** breaks the credential's shape (`\uXXXX` mid-token, as + emitted by XSS-safe serializers) — closed in **v2.7.6** by decoding inline SSR state blobs. + *Still open:* wasm strings [LOW]. **Deliberately not done:** probing for unlinked paths + (`.env`, `.git/config`, backups) is active enumeration, not passive discovery — it would + contradict the "passive assessment" statement in every client report. If it is ever added it + must be a separate, clearly-labelled opt-in mode, not folded into the default scan. 5. **Detector breadth.** ~54 patterns vs TruffleHog's 700+. Quality > quantity, but high-impact providers are missing (Twilio, GCP service-account JSON, Azure AD, Cloudflare, Shopify, Supabase `service_role`, Vercel, Notion). Each new detector should ship *with* a verifier where safe. @@ -103,7 +111,10 @@ repo scanning; win the *web-surface* niche. access"), a "Verified Active" KPI tile, and an honest measured-precision "Detection quality" statement (verification-first + CI-gated precision/recall). Turns the engine work into a client-ready deliverable. +4 tests. -- **R10 · Asset caching** (ETag/If-Modified-Since) + per-provider verify concurrency. [LOW] +- ~~**R10 · Asset caching**~~ ✅ **DONE (v2.7.7)** — conditional GET with a per-target validator + cache; a 304 skips an unchanged *and previously clean* asset, but always refetches one that + had a finding so nothing silently vanishes from a report. No response bodies cached, by + design. *Still open:* per-provider verify concurrency. [LOW] - **R11 · Distribution** — PyPI publish, tagged releases, docs. [LOW] ## Recommended next steps (highest ROI for Cindrasec's stage) diff --git a/frontend/index.html b/frontend/index.html index 99f7f68..3d6b449 100644 --- a/frontend/index.html +++ b/frontend/index.html @@ -61,6 +61,115 @@ .stat-card { padding:14px 12px; } .header-arch, .header-sep { display:none; } .hex-decoration { display:none; } + + /* Toolbar: let the export buttons breathe instead of cramming one line. */ + .findings-toolbar { align-items:stretch; } + .findings-toolbar > div:last-child { + display:grid; grid-template-columns:1fr 1fr; gap:8px; width:100%; + justify-content:stretch !important; + } + .findings-toolbar > div:last-child > * { + width:100%; min-height:40px; font-size:9px !important; + } + .findings-toolbar > div:last-child > select { grid-column:1 / -1; } + + /* Comfortable tap targets on the scan controls. Apple/WCAG guidance is + ~44px; anything under that is a mis-tap on a real phone. */ + #verify-toggle, #deep-toggle { min-height:44px; } + .control-row .btn-scan, .control-row .btn-stop { min-height:46px; } + /* Secondary panel actions (terminal COPY/CLR, toolbar chips) are tiny by + design on desktop — give them a real hit area on touch. */ + .panel button { min-height:32px; } + #modal-box { padding:20px 18px; width:94%; max-height:88vh; } + } + + + /* Card layout covers tablets in portrait too — a 9-column table is + cramped well above phone width, so the breakpoint is wider than the + phone-specific tweaks above. */ + @media (max-width: 900px) { + /* ── Findings: table → cards ─────────────────────────────────────────── + A 9-column table cannot be read on a phone; horizontal scrolling hides + the severity and the action buttons, which are the two things an + operator actually needs. Below 680px each row becomes a self-contained + card and every cell is prefixed with its column name via data-label, + so no information is lost and nothing scrolls sideways. */ + .findings-table thead { display:none; } + .findings-table, + .findings-table tbody, + .findings-table tr, + .findings-table td { display:block; width:100%; } + + .findings-table tr { + border:1px solid #1a2e1f; border-radius:4px; + background:#0d1410; margin-bottom:10px; padding:10px 12px; + } + .findings-table tr:hover td { background:transparent; } + + .findings-table td { + border-bottom:none; padding:5px 0; + display:flex; align-items:flex-start; gap:10px; + max-width:none !important; width:auto !important; + white-space:normal !important; + } + /* Column name, rendered from the cell's data-label. */ + .findings-table td[data-label]::before { + content:attr(data-label); + flex:0 0 82px; + font-family:'Orbitron',monospace; font-size:8px; letter-spacing:0.1em; + color:#4a7c59; text-transform:uppercase; padding-top:3px; + } + /* Row number is redundant once rows are cards. */ + .findings-table td.col-idx { display:none; } + /* Long values wrap instead of being clipped by an ellipsis. */ + .findings-table td.col-source > div, + .findings-table td.col-reason { + white-space:normal !important; overflow:visible !important; + text-overflow:clip !important; word-break:break-word; + } + .findings-table td.col-reason { display:flex; } + .code-snippet { + max-width:100%; white-space:pre-wrap !important; word-break:break-all; + } + .findings-table td.col-conf .confidence-bar { width:100%; } + /* Actions get real, thumb-sized targets. */ + .findings-table td.col-action { + gap:8px; padding-top:9px; margin-top:4px; border-top:1px solid #12210f; + } + .findings-table td.col-action button { + flex:1 1 auto; min-height:40px; font-size:10px !important; + margin-left:0 !important; + } + .findings-table td.col-action::before { display:none; } + + } + + + /* Touch sizing is a property of the input device, not the screen width: + a tablet needs thumb-sized targets even at 900px. */ + @media (pointer: coarse) { + .panel button, .findings-toolbar button, .findings-toolbar select { min-height:36px; } + .findings-table td.col-action button { min-height:40px; } + .control-row .btn-scan, .control-row .btn-stop { min-height:46px; } + #verify-toggle, #deep-toggle { min-height:44px; } + .scan-input, #crawl-pages-input { font-size:16px !important; } + } + + /* iOS zooms the whole page when a focused input is under 16px. Keeping the + text inputs at 16px on touch layouts avoids that jarring auto-zoom. */ + @media (max-width: 680px) { + .scan-input, #crawl-pages-input { font-size:16px !important; } + } + + /* Respect the notch / home indicator on modern phones. */ + @supports (padding: max(0px)) { + @media (max-width: 680px) { + .app-header { padding-left:max(14px, env(safe-area-inset-left)); + padding-right:max(14px, env(safe-area-inset-right)); } + .app-main { padding-left:max(13px, env(safe-area-inset-left)); + padding-right:max(13px, env(safe-area-inset-right)); + padding-bottom:max(14px, env(safe-area-inset-bottom)); } + } } @media (max-width: 380px) { .stats-grid { grid-template-columns:1fr 1fr; } @@ -573,7 +682,7 @@
-
SecretNode v2.7.1 initialised. Awaiting target...
+
SecretNode initialised. Awaiting target...
@@ -652,7 +761,7 @@