Sentinel

Streaming anomaly detection on crypto markets. Kafka carries the trades, Spark turns them into rolling features, an Isolation Forest decides what looks wrong, and a Streamlit dashboard shows you what it flagged.

It runs on either a simulator with anomalies injected on purpose (useful because you know the ground truth) or on live Binance trades over their public WebSocket.

Getting it up

Docker and Docker Compose, that's it.

docker compose up --build

For real market data instead of the simulator:

DATA_SOURCE=binance docker compose up --build

Give it two or three minutes on first start. Nothing is broken during that time: Spark has to accumulate enough windows to write features, the training job waits for those files to show up, and the API only picks up a model once training finishes. docker compose logs -f if you want to watch.

Then: dashboard on 8501, API on 8000 (/docs for Swagger), Spark UI on 4040. docker compose down -v when you're done and want the volumes gone too.

The pipeline

   generator --> Kafka --> Spark Streaming --> Parquet --> training --> API --> dashboard
  (sim or                  rolling stats,     partitioned  Isolation    /predict   Streamlit
   Binance)                z-scores           by symbol    Forest

Generating. The simulator produces BTC/ETH/BNB against USDT with configurable rates of price spikes, volume spikes and flash crashes, labelling each event so you can measure yourself afterwards. The Binance connector streams real trades and needs no API key. Both emit the same shape:

{
  "timestamp": 1710000000,
  "symbol": "BTC-USDT",
  "price": 43150.50,
  "volume": 12.534210,
  "log_return": 0.003521,
  "is_anomaly": false,
  "anomaly_type": null
}

Spark. Structured Streaming reads the topic and computes rolling mean and standard deviation for price, log return and volume over one-minute tumbling windows, then z-scores off those. Where the standard deviation is zero the z-score returns 0 rather than a NaN: a flat minute isn't an anomaly, and a NaN poisons everything downstream. Output is Parquet partitioned by symbol.

Training. Isolation Forest, 200 estimators, 1% contamination, over the five z-score and volatility features with a StandardScaler in front. On simulated data there's an 80/20 split and a classification report, since the labels exist. On real data it's fully unsupervised, because there's nothing to score against.

Serving. FastAPI comes up immediately and loads the model lazily, so the API is reachable while training is still running and reports model: not loaded rather than refusing to start.

curl http://localhost:8000/health
curl http://localhost:8000/stats
curl http://localhost:8000/model-info
curl "http://localhost:8000/latest-predictions?limit=50&symbol=BTC-USDT"

curl -X POST http://localhost:8000/predict \
  -H "Content-Type: application/json" \
  -d '{"symbol":"BTC-USDT","z_score_price":4.5,"z_score_log_return":3.8,
       "z_score_volume":1.5,"rolling_price_std":0.008,"rolling_volume_std":25}'

/system-status reports on the other services too, which is what the dashboard's status page is built on.

Dashboard

Four pages in the sidebar.

System Status shows whether the API, Spark, Kafka and Zookeeper are up, with latency and the current model parameters. Start here if something looks wrong.

Live Feed auto-refreshes. Pick a symbol, a refresh interval and how far back to look, and it shows the score timeline and recent alerts.

Analytics is the retrospective view: per-symbol breakdown, feature correlations, score trend, and a filterable table you can export to CSV.

Manual Test lets you push feature values in by hand. The presets (Normal, Price Spike, Volume Spike, Flash Crash) fill the sliders with something representative, which is the quickest way to get a feel for where the decision boundary sits.

Configuration

Environment variables, all with defaults that work under Compose:

Variable	Default
DATA_SOURCE	simulated	or binance
KAFKA_BOOTSTRAP_SERVERS	localhost:9092
KAFKA_TOPIC	crypto-market
EVENT_FREQUENCY_SECONDS	1	simulator only
ANOMALY_PROBABILITY	0.01	simulator only
MIN_PARQUET_FILES	3	how much data before training starts
MAX_WAIT_SECONDS	600	give up waiting for it
MODEL_PATH / FEATURES_PATH		where the model and features live
API_BASE_URL	http://localhost:8000	used by the dashboard

Seven services come up: Zookeeper and Kafka, the generator, Spark, the training job (which exits once it's done), the API and the dashboard. They share three volumes: features written by Spark and read by training, Spark's checkpoints, and the model written by training and read by the API.

Tests

pip install -r tests/requirements.txt
pytest

Covers the simulator's event structure and log returns, the Binance connector's symbol mapping and message parsing, preprocessing (NaN handling, labelled and unlabelled paths), the API schemas and endpoints, and config defaults and overrides.

Layout

data-generator/ produces events, spark-java/ is the Maven-built streaming job, ml-python/ trains and evaluates, api/ serves predictions, dashboard/ is the Streamlit app, docker/ holds the Dockerfiles and tests/ the pytest suite.

There's more detail in docs/report.md, and docs/choices-en.md explains why things were built the way they were. Worth reading if you're wondering why Isolation Forest rather than something supervised.