F.E.W. — Find Exuberance Warning

Spark-enpowered pipeline for large-scale financial bubble detection using GSADF test on AWS.

Distributed financial bubble detection pipeline built with Apache Spark, AWS Glue, and Python, implementing GSADF/SADF econometric tests on large lists of securities.

---

Project Overview

F.E.W. (Find Exuberance Warning) is a financial data engineering pipeline designed to look for signals of asset price bubbles across large universes of securities.

The project complements P.E.W. (Price Exuberance Warning).

• PEW performs different econometric analysis on a small set of securities, producing metrics, charts, and reports intended to monitor an identified potential bubble.
• FEW performs large-scale screening across hundreds of securities to identify candidates potentially exhibiting bubble-like price dynamics.

Once potential candidates are detected with FEW, the user can switch to PEW for deeper monitoring and analysis: the idea is that PEW is for performing different analysis on a small range of securities, while FEW is for performing the same analysis on a large range of securities.

The core idea

Studying financial bubbles requires first finding potential bubbles. FEW addresses this challenge by running GSADF-test and its variations (GSADF/ BSAF / SADF / rolling) across large sets of securities using Apache Spark parallelism.

The pipeline workflow can be summarized in the following steps:

1. Download historical price data
2. Compute bubble detection metrics
3. Store results in Parquet format on S3
4. Register datasets in the AWS Glue Data Catalog

Once registerd in the data catalog, datasets can be analyzed through Amazon Athena, Power BI, or other analytics tools

This project combines:

• Data Engineering
• Distributed Computing
• Econometrics
• Quantitative Finance

Architecture

The architecture is intentionally simple and modular.

Unlike PEW, which focuses on visualization and reporting, FEW is designed as a data production pipeline meant for advanced analysis by data engineers or quantitative researchers. The outpu of the pipeline is a number of parquet stored on an S3 bucket intended registered in glue catalog for further analysis.

The architecture follows a Medallion Data Lake pattern:

Layer	Implementation	Description
Bronze	AWS Lambda	Raw historical data ingestion from external API
Silver	AWS Lambda	Data cleaning, alignment of time series, and schema enforcement.
Gold	AWS Glue / Spark	Distributed computation of GSADF, BSADF, and SADF metrics.

• Bronze Layer, for raw historical data downloads
• Silver Layer, for aligned time series ready for processing
• Gold Layer for computed econometric metrics.

Infrastructure overview:

• AWS Lambda

  • Data ingestion
  • Bronze → Silver transformation

• AWS Parameter Store

  • Centralized configuration and execution parameters

• AWS Glue (Spark)

  • Distributed computation of econometric tests

• AWS S3

  • Data lake storage

• AWS Step Functions

  • Pipeline orchestration

• AWS Glue Catalog

  • Metadata management

Architecture diagram:

Workflow

The workflow orchestration is managed through AWS Step Functions, and divides operations in two parts, ingestion and computing:

The workflow can optionally execute the ingestion stage, the Spark processing stage, or both, depending on the parameters provided at runtime. This makes the pipeline flexible to support experimentation, for example when re-execution of calculation is needed without re-downloading data or when processing externally sourced datasets is wanted.

Why Spark

Detecting bubbles requires running a regression on every possible sub-window of a price series, leading to explosive computational complexity, that can be handled using spark. To undestand how parallelism was used we need a little dive into the math of the problem. Consider a time series of prices:

$$ (t_i,p_i), \quad i=1,\ldots,N $$

To examine long time series (more than a year) adjusted close prices $p_i$ are needed (to keep track of dividends). The Dickey-Fuller (DF) test is a unit-root test, so is better to work with natural logarithms of prices:

$$ (t_i,y_i), \quad y_i = \log(p_i) $$

The DF test requires to do a regression:

$$ \Delta y_i = (\alpha - 1) y_{i-1} + u_i = \beta y_{i-1}+u_i $$

(see Dickey-Fuller test) on wikipedia for example). The test checks the hypothesis:

$$ H_0:\beta=0; \quad H_1:\beta<0 $$

so if $\alpha$ is equal or less than 1 or againg, if there is or there is not a unit root: in more simple terms, when a unit root is detected prices follow a geometric brownian motion. The augmented Dickey-Fuller test simply adds further lags ( $y_{i-k}$ , $k&gt;1$ ) to the regression.

SADF (Supremum Augmented Dickey Fuller), BSADF (Backward Supremum Augmented Dickey Fuller) and GSADF (Generalized Supremum Augmented Dickey Fuller) tests are obtained performing the ADF on different subseries of the original time series and taking the supremum value: critical values are different from ADF and can be obtained by simulation (the tests were introduced by Phillips, Shi and Yu in different papers, see Phillips, P. C. B., Wu, Y., & Yu, J. (2011). "Explosive Behavior in the 1990s Nasdaq: When Did Exuberance Escalate Asset Values?". International Economic Review, 52(1), 201-226 and Phillips, P. C. B., Shi, S., & Yu, J. (2015). "Testing for Multiple Bubbles: Historical Episodes of Exuberance and Collapse in the S&P 500". International Economic Review, 56(4), 1043-1078). Unlike the standard ADF test, which looks for stationarity represented by the alternative $\beta &lt; 0$ vs $\beta=0$, these tests are looking for the right tail behaviour, focusing on the case $\beta&gt;0$ that would imply an explosive acceleration of prices, signaling a possible bubble.

In symbols the tests are:

$$ SADF = \sup { ADF(y_1,\ldots,y_N; 1+w\leq m \leq N } $$ $$ BSADF = \sup { ADF(y_m,\ldots,y_N; 1\leq m \leq N-w } $$ $$ GSADF = \sup { ADF(y_a,\ldots,y_b); 1 \leq a,b \leq N, b-a \geq w } $$

where $w$ is a minimun legth for the ADF test on sub-timeseries to be meaningful. By a computational point of view GSADF test is the worst since it requires to calculate the ADF on any subseries: once all the ADF tests have been calculated one has the data to extract also the other tests, since if we define:

$$ Q={ q_{a,b}=ADF(y_a,\ldots,y_b); b-a \geq w } $$

then:

$$ SADF=\sup { q_{a,b} \in Q, a=1,b \geq 1+w }; \quad BSADF=\sup { q_{a,b} \in Q, a \leq T-w,b = T }; \quad GSADF=\sup Q $$

Using spark we can populate the set $Q$ as a Dataframe and calculate the metrics as operations on the DataFrame. Consider a time series of three years, 750 data points, and a window of 2 months, 40 data points. You then have to perform ADF regression on 202,405 intervals. Repeat that for 100 securities (for example nasdaq 100) and you will have to deal with 20,240,500 regressions. Spark allows you to do that in five minutes leveraging pandas UDFs.ù, while a serialized execution in python would have taken hours.

Installation and Running

Installation

In order to install the project it is necessary to have:

• AWS CLI installed and configured
• administrative permissions on an AWS account
• a Marketstack commercial API account

The Marketstack API is required because the project needs adjusted closing prices, and because the data download performed by the pipeline would exceed the limits of most free data providers.

The deployment is based on CloudFormation and can be executed with a single command.

1. clone the repository
2. from the root folder of the repository run

./deploy.sh

The setup of a secret to store the API key to download data is required. If you already have a MarketStack key stored you can modify the relavant parameter (see section Configuration and Running), if not:

1. setup the secret creating a file named secret.json in the root directory of the project, following the structure provided in secret.json.example
2. from the root folder of the repository run

./setup_secret.sh

Configuration and Running

In the parameters store you will find the parameters that control pipeline execution, among with:

Parameter	Description
/few/base-date	Reference date used internally for calculations
/few/date-from	Start date for historical data download
/few/date-to	End date for historical data download
/few/ticker-list-obj-name	JSON file containing the list of tickers to download
/few/ticker-filter-obj-name	Text file containing the list of tickers to process
/few/list-bucket-name	Name of the bucket where ticker lists are stored
/few/secret-api-name	Name of the secret where API key is stored

To run the pipeline go to the console of AWS Step Functions and run the state machine called few-machine. The machine needs an input, in json, as:

{
    "ExecuteIngestion":true,
    "ExecuteSpark":true
}

Setting to true of false the two entries you can decide to execute the entire pipeline or the ingestion steps (donwload and transform) or the caculation step separately.

If you want to run the pipeline on a sample list of securities, you can use the lists prepared in UTILS folder (that replicates the nasdaq100 list of june 2025 as reported on Italian wikipedia). Go to that directory and run:

make upload

to upload necessary files to list bucket.

DeInstallation

Empty all bucket that have been created (their names are all NUMACCOUNT-few-something) and then go to cloudformation console. You need to delete the stacks called few-assets-stack and few-project-stack.

Disclaimer

F.E.W. is an econometric pipeline for educational and research purposes only. The econometric models and metrics generated by this pipeline are provided for research and illustrative purposes and do not constitute financial advice, investment recommendations, or trading signals nor are they intended for use in real-world trading or investment decision-making.

Please note:

• Model limitations. Econometric models are simplified representations of complex financial systems and are subject to assumptions, estimation errors, structural uncertainty, and potential implementation errors.
• No Liability. The author assumes no responsibility for financial losses, data inaccuracies, errors, or any decisions made based on the project or on the results of the project. Any use of the project or its results for real investment purposes is strictly at the user's own risk.
• Cloud Costs. Deploying this project provisions AWS resources that incur costs. Users are responsible for monitoring their AWS billing.
• Use of Data Provider API. The user is responsible for obtaining and maintaining their own API licenses. Downloading large universes of historical data can rapidly consume your API provider's request quota: it is recommended to start with a small ticker list to monitor usage. This project is not affiliated with Marketstack.

THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

License

This project is licensed under the MIT License - see the LICENSE file. The graphical assets and the documentation are included under the same license. The symbols in diagrams that represent services in AWS are property of Amazon Web Services Inc. The banner has been generated using Gemini.