dx_evidence_graph: viz stub — coordination with dx-agent data model #62
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,12 +1,25 @@ | ||
| --- | ||
| # SCRIPT_BUNDLE_URLS is read by openresty (cloud-proxy_server_image) and | ||
| # injected into the UI's `window.__PIXIE_FLAGS__` (see | ||
| # k8s/cloud/base/proxy_nginx_config.yaml's sub_filter + set_by_lua_block). | ||
| # The UI's fetch resolves relative URLs against document.baseURI, which is | ||
| # always the cloud-proxy itself, so a relative `/bundle-oss.json` URL hits | ||
| # the bundle that the cloud-release pipeline bakes into the proxy image as | ||
| # a container layer (src/cloud/proxy/BUILD.bazel `script_bundle` -> | ||
| # /bundle/bundle-oss.json). nginx serves it at the path `/bundle-oss.json` | ||
| # from both server blocks (bare + work.* subdomain, | ||
| # k8s/cloud/base/proxy_nginx_config.yaml lines 270 + 342). | ||
| # | ||
| # Bottom line: every cloud-release tag's bundle now ships with the | ||
| # deployment, no separate update-script-bundle workflow needed. | ||
| apiVersion: v1 | ||
| kind: ConfigMap | ||
| metadata: | ||
| name: pl-script-bundles-config | ||
| data: | ||
| SCRIPT_BUNDLE_URLS: >- | ||
| [ | ||
| "/bundle-oss.json" | ||
| ] | ||
| SCRIPT_BUNDLE_DEV: "false" | ||
| PL_SCRIPT_MODIFICATION_DISABLED: "false" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| # dx_evidence_graph | ||
|
|
||
| A Pixie UI dashboard that renders one dx-agent investigation as a | ||
| **severity-weighted, all-protocol pod-to-pod attack graph**. Replaces | ||
| the latency-weighted HTTP service map in `cluster_overview` for | ||
| security work. | ||
|
|
||
| * Nodes = pods. Falls back to service → IP, mirroring `net_flow_graph`. | ||
| * Edges = the attack path emitted by dx (delivery → egress → | ||
| execution → collection → exfil → pivot). | ||
| * Display spec: `vispb.Graph`. **`edgeWeightColumn = weight`** | ||
| (open-ended UInt16 sum of CRS severity → edge thickness), | ||
| **`edgeColorColumn = max_severity`** (discrete 2-5 heat → edge | ||
| colour). | ||
| * Read source: `forensic_db.dx_attack_graph` via `px.DataFrame`'s | ||
| `clickhouse_dsn` kwarg (`src/carnot/planner/objects/dataframe.cc:43`). | ||
|
|
||
| ## Schema — `forensic_db.dx_attack_graph` | ||
|
|
||
| Locked with dx-agent in PR #62 / `entlein/dx#68`. The | ||
| `attackgraph.Edge` Go struct is the single source of truth for the | ||
| JSON wire format, the ClickHouse row, and the test fixture. | ||
|
|
||
| | Column | Type | Role | | ||
| |---|---|---| | ||
| | `investigation_id` | String | one graph per dx verdict / pivot incident (UI filter key) | | ||
| | `ts` | UInt64 | unix nanos | | ||
| | `requestor_pod` / `responder_pod` | String | the hop (`ns/pod`); `""` if only an IP is known | | ||
| | `requestor_service` / `responder_service` | String | | | ||
| | `requestor_ip` / `responder_ip` | String | peer IP when pod unresolved | | ||
| | `weight` | UInt16 | Σ CRS severity on the hop — `edgeWeightColumn` | | ||
| | `max_severity` | UInt8 | top single-criterion severity (2-5) — `edgeColorColumn` | | ||
| | `confidence` | Float32 | verdict confidence | | ||
| | `edge_kind` | String | `delivery`/`egress`/`execution`/`collection`/`exfil`/`pivot` | | ||
| | `condition` / `criteria` | String | ruled-in condition + criterion label(s) | | ||
| | `num_findings` | UInt32 | | | ||
|
|
||
| Table DDL (mirrors `kubescape_logs` partition/TTL convention): | ||
|
|
||
| ```sql | ||
| CREATE TABLE forensic_db.dx_attack_graph ( ...columns above... ) | ||
| ENGINE = MergeTree | ||
| PARTITION BY toYYYYMM(fromUnixTimestamp64Nano(ts)) | ||
| ORDER BY (investigation_id, requestor_pod, responder_pod) | ||
| TTL toDateTime(fromUnixTimestamp64Nano(ts)) + INTERVAL 30 DAY DELETE; | ||
| ``` | ||
|
|
||
| ## Per-rig ClickHouse DSN | ||
|
|
||
| The bundled `vis.json` ships with `clickhouse_dsn` **empty** — the | ||
| default is intentionally non-credentialed so the bundle stays | ||
| portable across clusters. Operators fill the DSN in via the Pixie | ||
| UI script-args panel at run time. | ||
|
|
||
| For the in-cluster soc deployment the DSN is: | ||
|
|
||
| ``` | ||
| forensic_analyst:changeme-analyst@clickhouse-forensic-soc-db.clickhouse.svc.cluster.local:9000/forensic_db | ||
| ``` | ||
|
Comment on lines
+57
to
+59
There was a problem hiding this comment. Add a language tag to the fenced DSN block. Line 57 opens a fenced code block without a language, which violates MD040 and may fail markdown lint gates. Suggested fix-```
+```text
forensic_analyst:changeme-analyst@clickhouse-forensic-soc-db.clickhouse.svc.cluster.local:9000/forensic_dbVerify each finding against current code. Fix only still-valid issues, skip the In Source: Linters/SAST tools |
||
|
|
||
| `forensic_analyst` has read-only SELECT on `forensic_db`; same | ||
| credential the existing `soc/analysis/px_clickhouse/kubescape/observe.pxl` | ||
| script uses for `kubescape_logs`. Override in the UI for other rigs. | ||
|
|
||
| ## Manual-load prototype | ||
|
|
||
| `tools/load_prototype/` is a Go helper that renders the `Edge` | ||
| schema from a JSON fixture into a standalone HTML page using | ||
| cytoscape.js. Same column→visual mapping the production | ||
| `vispb.Graph` spec uses. Useful when ClickHouse isn't reachable | ||
| from the UI (offline review, fixture validation). | ||
|
|
||
| ```bash | ||
| go run ./tools/load_prototype \ | ||
| -fixture fixtures/sample.json \ | ||
| -investigation_id log4shell-6a32ea57 \ | ||
| -out /tmp/dx_log4shell.html | ||
| ``` | ||
|
|
||
| The fixture in `fixtures/sample.json` is dx-agent's real | ||
| log4shell + argocd verdicts from the rig run that locked the | ||
| schema. `fixtures/screenshots/dx_log4shell.html` and | ||
| `fixtures/screenshots/dx_argocd.html` are the pre-rendered pages | ||
| for review without running the tool. | ||
|
|
||
| The tool retires once the AE live-write (`WriteAttackGraph` → | ||
| `forensic_db.dx_attack_graph`) is on every cluster running this | ||
| bundle. | ||
|
|
||
| ## Deploy | ||
|
|
||
| Bundle build path: | ||
|
|
||
| 1. `//src/pxl_scripts:script_bundle` walks every `*.pxl` + `vis.json` | ||
| under `src/pxl_scripts/` and emits `bundle-oss.json` | ||
| (`src/pxl_scripts/BUILD.bazel:34`). | ||
| 2. `//src/cloud/proxy:proxy_server_image` bakes the bundle in as a | ||
| container layer at `/bundle` | ||
| (`src/cloud/proxy/BUILD.bazel:36`). | ||
| 3. `skaffold run -f skaffold/skaffold_cloud.yaml` rebuilds the | ||
| cloud-proxy image and applies the Deployment. | ||
|
|
||
| Vizier / PEM / standalone-pem images are unaffected — this is a | ||
| UI-bundle-only change. | ||
|
|
||
| ## Out of scope for v1 | ||
|
|
||
| * `conn_stats` overlay (the "render the benign neighbourhood + light | ||
| up the attack path" view). Ship the attack-path-only graph first; | ||
| add the join in v2 once the visual has been used on a real | ||
| incident. | ||
| * Time anchoring relative to `ts` rather than free-form `start_time`. | ||
| Operators today use `-15m` defaults; a future widget could centre | ||
| the window on the investigation's first `ts`. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # Copyright 2018- The Pixie Authors. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| ''' DX Attack Graph: pod-to-pod hops weighted by dx evidence severity. ''' | ||
| import px | ||
|
|
||
|
|
||
| def dx_attack_graph(start_time: str, clickhouse_dsn: str): | ||
| ''' Read forensic_db.dx_attack_graph and return the edge columns. | ||
| Args: | ||
| @start_time: e.g. "-15m". | ||
| @clickhouse_dsn: user:pass@host:port/db. | ||
| ''' | ||
| df = px.DataFrame('dx_attack_graph', | ||
| clickhouse_dsn=clickhouse_dsn, | ||
| start_time=start_time) | ||
| return df[['requestor_pod', 'responder_pod', | ||
| 'requestor_service', 'responder_service', | ||
| 'requestor_ip', 'responder_ip', | ||
| 'weight', 'max_severity', 'confidence', | ||
| 'edge_kind', 'condition', 'criteria', 'num_findings']] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove password-bearing DSN examples from documentation.
Line 58 publishes a credential-bearing DSN pattern (
user:pass@...) with a concrete password token. Even in docs, this tends to get copied into runtime configs and weakens the security baseline.Suggested doc change
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 57-57: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents