Preflight

Everything that reads Solana reads confirmed blocks, so it tells you what happened about a second after it did. This reads the shreds a leader is still propagating, decodes the swaps inside them, and reconciles them against the chain afterwards.

Two things fall out of that. You see a swap before any RPC will admit it exists, including the ones that are about to revert, and over five consecutive mainnet slots that was 172 of 307. And you see the ones the chain never confirms at all, because the slot carrying them didn't survive. Those are in no confirmed-block dataset anywhere: once the fork is gone, nothing records that they were ever attempted.

The three buckets

Each swap gets reconciled against the chain and lands in one of:

landed	in the block, no error
failed_onchain	in the block, reverted
never_landed	seen propagating, never confirmed

The second is the volume. The third is the one nothing else can show you.

The part you have to get right first

never_landed is an argument from absence, and there are two reasons a swap can be absent: its slot never made it, or we dropped the shred. Those are indistinguishable unless you know how completely you saw the slot.

So every row carries the coverage of its slot, and anything that reasons about absence filters on it. Rows from a slot we only partly reconstructed don't get to vote. Landed rows survive any coverage, because a block is positive proof and our own packet loss can't fake an inclusion.

Two independent estimates say how good the view was: shred completeness, out of the deshredder's own FEC accounting, and confirmed swaps that arrive with no attempt behind them. The second needs no cooperation from the ingest, which makes it the harder evidence. They should agree. If they don't, believe the second.

The rest is in solana-flow/README.md.

Running it on the simulator

The simulator produces attempts and confirmations with known drop rates, injected contention spikes and injected coordinated buys, so detectors can be evaluated rather than merely executed. It's what the page runs on by default, which is why a fresh clone works with no Solana access at all:

pip install -r api/requirements.txt -r dashboard/requirements.txt

cd api && FLOW_SOURCE=demo uvicorn main:app --port 8000   # then, in another shell
cd dashboard && streamlit run app.py

Dashboard on 8501, API on 8000, /docs for Swagger.

Running it on the chain

Three processes. The ingest and the confirmation feed write NDJSON; everything else reads it.

   Jito proxy --> shred-ingest --> flow.ndjson ------+
   (UDP shreds)   (rebuild slots,                    |
                   decode swaps)                     +--> tap --> API --> dashboard
                                                     |             |
   RPC endpoint <-- confirm.py --> confirmations.ndjson             +--> daily_report
   (what landed)   (statuses, plus                                       (EN and FR)
                    whole blocks now
                    and then)

# 1. the ingest, fed by your own jito-shredstream-proxy on 127.0.0.1:8002
shred-ingest --bind 127.0.0.1:8002 --out /var/lib/preflight/flow.ndjson

# 2. the confirmation feed, against whatever RPC you have
python solana-flow/confirm.py --rpc "$RPC_URL" \
    --tap /var/lib/preflight/flow.ndjson \
    --out /var/lib/preflight/confirmations.ndjson

# 3. the page, reading both
cd api && FLOW_SOURCE=tap uvicorn main:app --port 8000

deploy/ has systemd units for the first two. Setup for the ingest, including the net.core.rmem_max sysctl to set before anything else, is in shred-ingest/README.md.

The report needs no cluster and no Kafka:

python solana-flow/daily_report.py --source tap \
    --path /var/lib/preflight/flow.ndjson \
    --confirmations /var/lib/preflight/confirmations.ndjson

Kafka and Spark are still there for a backlog of many days, which is the one thing a single reconciler in one process is bad at. producer.py --source tap forwards both files into topics and spark_job.py does the same join at scale.

What is verified, and what is not

The confirmation feed was checked against mainnet: 282 swaps out of one slot, resolved by getSignatureStatuses and then compared with the block itself, zero disagreements on either the outcome or the error code. Over five consecutive slots, all 307 swaps reconciled and all 307 matched what the chain said. tests/test_confirm.py runs against a real block kept in tests/fixtures/.

The reconciliation, the metrics and the report are covered by the suite over simulated flow whose answers are known. tests/roundtrip.rs builds real shreds, drops some, and demands the original blob back byte for byte. tests/test_flow_spark_parity.py runs the Spark job and the reference implementation over the same input and fails on any disagreement.

What none of that covers is the ingest itself: it has never seen a real shred, only ones it built. If Agave's layout has moved, that is where you find out, and a wrong offset in a swap layout doesn't fail loudly, it produces plausible numbers. Read shred-ingest/README.md and decode a handful of known transactions against an explorer before trusting anything that came off a real link.

The lead time is measured against our own confirmation feed, so its floor is the status poll interval. Sub-100ms claims would need a Geyser subscription rather than an RPC.

Tests

pip install -r tests/requirements.txt
pytest

cd shred-ingest && cargo test --all-targets

No Kafka, no Spark, no network, no Solana access. If testing something needs infrastructure, that's usually a sign the logic wants pulling out into a function that takes plain data.

Layout

solana-flow/   schema, reconciler, metrics, simulator, the tap, the confirmation feed, the report
shred-ingest/  the Rust front: UDP in, rebuilt slots, decoded swaps, NDJSON out
api/           FastAPI over the reconciled flow
dashboard/     the Streamlit page
reports/       one bilingual report per day, written here. Empty until it has run on real data
deploy/        systemd units for the ingest and the confirmation feed