Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Sliding Window Euclidean Distance Node

## Overview

`SlidingEuclideanDistanceNode` is a gQuant transform node that compares the **shape** of a short query signal against **every position** in a longer time series stream, using Euclidean distance. It produces a *distance profile* — one distance value per possible alignment of the query within the stream — which is the core primitive behind pattern search, motif discovery, and anomaly detection on time series (e.g. price, volume, or tick data).

## Problem

There was previously no gQuant node for computing the Euclidean distance between a query signal and every sliding window of a stream. Doing this with a naive loop is `O(n·m)` (`n` = stream length, `m` = query length), which does not scale for large or frequently-updated streams, and any hand-rolled version would sit outside the TaskGraph, losing gQuant's column/type validation, GPU dataframe support, and gQuantLab UI integration.

## Solution

This node computes the distance profile in `O(n log n)` using the FFT-based sliding dot-product trick (Mueen's Algorithm for Similarity Search, MASS), instead of the naive `O(n·m)` loop:

```
dist(i)^2 = Σ T[i+j]^2 − 2·Σ T[i+j]·Q[j] + Σ Q[j]^2
```

- `Σ Q[j]^2` — constant, computed once
- `Σ T[i+j]^2` — moving sum of squares over the stream, via cumulative sums, `O(n)`
- `Σ T[i+j]·Q[j]` — a sliding dot product, computed via FFT cross-correlation, `O(n log n)`

An optional `normalize` flag applies z-normalization to each window, making the comparison invariant to amplitude and offset — useful when you care about matching the *shape* of the query rather than its absolute scale.

## Usage

### Node configuration

| Key | Type | Description |
|---|---|---|
| `stream_col` | `str` | Column name of the longer time series to search |
| `query_col` | `str` | Column name of the short query/reference signal |
| `normalize` | `bool` (optional, default `False`) | Z-normalize each window for shape-only comparison |

### Output

Adds a new column, `distance_profile`, of the same length as the stream, front-padded with `NaN` for the first `m - 1` positions (where a full-length window isn't yet available), so it aligns with the original stream index.

### Example

```python
from greenflow_gquant_plugin.transform import SlidingEuclideanDistanceNode

node = SlidingEuclideanDistanceNode(
"distance_node",
{"stream_col": "price", "query_col": "pattern", "normalize": True}
)
result_df = node.process([df])

# position of the best (closest) match:
best_match_index = result_df["distance_profile"].idxmin()
```

## File location

```
gQuant/plugins/gquant_plugin/greenflow_gquant_plugin/transform/slidingDistanceNode.py
```

Registered in `transform/__init__.py`:

```python
try:
from .slidingDistanceNode import SlidingEuclideanDistanceNode # noqa: F401
except ImportError:
from slidingDistanceNode import SlidingEuclideanDistanceNode # noqa: F401
```

and added to that file's `__all__` list.

## Alternatives considered

1. **Naive `O(n·m)` sliding loop** — simple, useful as a correctness baseline for tests, but too slow at scale.
2. **`numpy.correlate` / `scipy.signal.correlate` alone** — computes the cross-correlation term but not the moving sum-of-squares term needed for true Euclidean distance; would still need to be paired with the cumulative-sum trick above.
3. **Skipping normalization** — simpler, but loses invariance to amplitude/offset differences, which matters when comparing shapes at different price levels.

## Testing

- Unit test: compare FFT-based output against the naive `O(n·m)` loop on random data (`atol=1e-6`).
- Benchmark: measure speedup vs. the naive loop at realistic stream sizes (10k–1M points).
- GPU/CPU dispatch: verify consistent results whether inputs come from `cudf`/`cupy` or plain `numpy`.

## Reference

Mueen's Algorithm for Similarity Search (MASS) — the standard method for computing an all-subsequence Euclidean distance profile efficiently, and the basis for Matrix Profile motif discovery.
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import numpy as np

def sliding_euclidean_distance(T: np.ndarray, Q: np.ndarray) -> np.ndarray:
n, m = len(T), len(Q)
if m > n:
raise ValueError("Query longer than stream")

q_rev = Q[::-1]
fft_size = 1
while fft_size < n + m - 1:
fft_size *= 2

T_fft = np.fft.rfft(T, fft_size)
Q_fft = np.fft.rfft(q_rev, fft_size)
cross_corr = np.fft.irfft(T_fft * Q_fft, fft_size)[m - 1:n]

T_sq = T ** 2
cumsum = np.concatenate(([0], np.cumsum(T_sq)))
T_sq_win = cumsum[m:] - cumsum[:-m]

q_sq_sum = np.sum(Q ** 2)
dist_sq = T_sq_win - 2 * cross_corr + q_sq_sum
return np.sqrt(np.maximum(dist_sq, 0))