P.E.W. (Price Exuberance Warning)

Project overview and use case

P.E.W. is a data engineering project implementing an end-to-end, cloud-based, serverless data pipeline for financial time series analysis.

The project originates from the intent to recreate analytical models and workflows similar to those developed in institutional risk management contexts.

The project is named after Alan Greenspan’s famous 1996 speech, "The Challenge of Central Banking in a Democratic Society" in which he questioned the "irrational exuberance" of asset prices during the dot-com era. In that spirit, the initial aim of P.E.W. is to serve as a tool for studying and monitoring potential bubble dynamics, with a particular focus on the current AI-related equity sector.

Technically, P.E.W. is designed to decouple data ingestion from specific analytical logic. By using a cloud-native Python environment it bypasses the constraints of proprietary terminals and their limited scripting languages. This architecture ensures full transparency of the underlying algorithms and data and allows for smooth extensibility: users can implement custom models enabling different types of analysis without the overhead of complex wrappers or the rigidity of over-engineered frameworks.

Architecture and Design Decisions

The project follows modern serverless data engineering best practices, in particular cost efficiency, security, and operational robustness. The architecture is intentionally optimized for personal research and analysis, prioritizing reliability and simplicity over multi-user scalability.

Key technical aspects are:

• Workflow orchestration via AWS Step Functions for traceable execution logic.
• Compute layer implemented in AWS Fargate (ECS) to containerize econometric workloads in a hardened environment.
• Data ingestion separated from computation and implemented in AWS lambda to leverage Lambda’s native internet connectivity for API calls without compromising the security of computation core.
• Data persistence: Amazon DynamoDB serves as the primary store for metadata and analytical results.
• Infrastructure as Code (IaC) managed via Cloudformation with:
  • 9 persistent stacks
  • 1 ephemeral stack dynamically provisioned and destroyed during execution
• Security and Networking:
  • Zero-Trust Networking: Only the ingestion layer has internet access. Computation occurs within a private VPC with no direct internet egress.
  • Cost-Optimized Connectivity Utilizes VPC Endpoints (Interface & Gateway) instead of expensive NAT Gateways. Interface Endpoints are provisioned within the ephemeral stack only when needed to minimize idle costs.
  • Strict IAM policies following the Principle of Least Privilege
• Reporting and delivery:
  • Reports are served through an AWS Amplify application
  • Access protected via Basic Authentication (single-user setup)

Getting started

Follow these steps to replicate the environment and deploy the pipeline:

1. Prerequisites: ensure your AWS CLI is configured and your Docker engine is running. You will also need a MarketStack API key (commercial license required).
2. Secrets: prepare secret.json and website.json with your sensitive information (API keys and website protection settings). Refer to the secret.json.example and website.json.example files provided to facilitate setup.
3. Deploy: run the main deployment script

./deploy.sh

4. Setup models: Configure your models in DynamoDB. You can find a reference implementation in UTILS/sample_setup.py.
5. Execute: log-in in AWS console, go to Step Functions and run PEW_machine (or schedule running through EventBridge). You can retrieve the URL for the reporting website from the Amplify console.

Further documentation

Further details can be found in:

• Folders_stacks_installation.md: description of project folders, stacks and available installation commands.
• Models_and_database.md: brief descrption of the models interface and of the database setup.
• architecture_and_workflow.md: description of the workflow of the project implemented in the state machine.

Disclaimer

P.E.W. is an analytical tool intended for educational and research purposes only. The econometric models and reports generated by this pipeline are provided for research and illustrative purposes and do not constitute financial advice, investment recommendations, or trading signals nor are they intended for use in real-world trading or investment decision-making.

Please note:

• Model limitations. Econometric models are simplified representations of complex financial systems and are subject to assumptions, estimation errors, structural uncertainty, and potential implementation errors.
• No Liability. The author assumes no responsibility for financial losses, data inaccuracies, errors, or any decisions made based on the project or on the results of the project. Any use of the project or its results for real investment purposes is strictly at the user's own risk.
• Cloud Costs. Deploying this project provisions AWS resources that incur costs. Users are responsible for monitoring their AWS billing.

THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.

License

This project is licensed under the MIT License - see the LICENSE file. The graphical assets and the documentation are included under the same license. The symbols in diagrams that represent services in AWS are property of Amazon Web Services Inc. The banner has been generated using Gemini and post-edited using gimp.