fix: Aurora Serverless v2 bootstrap for free-tier AWS accounts (Guide 5)

[prodm93]

Summary

On March 25, 2026, AWS launched Aurora PostgreSQL Express Configuration and simultaneously restricted free-tier accounts to that creation path only. Guide 5's aws_rds_cluster resource calls CreateDBCluster without WithExpressConfiguration=True; free-tier accounts get back InvalidParameterCombination: Free Tier accounts must use Express Configuration. The hashicorp/aws provider does not support the flag (tracking issue filed March 26, 2026). The blocker is architectural: --with-express-configuration creates a cluster and a writer instance in a single API call, which does not fit Terraform's one-resource-per-state model.

This PR replaces the three incomplete shell scripts in bootstrap/ with two idempotent Python scripts that handle the full cluster lifecycle via boto3, then write the cluster ARN back into Terraform via bootstrap.auto.tfvars.json (auto-loaded by Terraform; no manual editing). All downstream modules are unchanged: Guide 6 and Guide 7 read the same terraform/5_database/ outputs as before.

Design tension: cluster state lives outside Terraform

The Aurora cluster is not in Terraform state. That is the necessary consequence of the provider limitation. The mitigation is aurora_cluster_arn with default = "" in variables.tf, which lets the first terraform apply run before the cluster exists to create the Secrets Manager secret and IAM role. setup_aurora_express.py then creates the cluster, writes bootstrap.auto.tfvars.json, and runs a second terraform apply to record the ARN in outputs. Downstream modules pick it up from terraform output exactly as before.

One IAM scope change to disclose: the rds-data actions in aws_iam_role_policy were previously scoped to the cluster ARN. Since the ARN does not exist at first-apply time, the policy now uses "*" for rds-data actions. The secretsmanager:GetSecretValue action remains scoped to the specific secret ARN, so the Lambda role can only authenticate against the correct credentials.

terraform destroy in 5_database/ deletes the secret and IAM role but not the cluster, which is not in state. The destroy script handles this; the student instructions call it out explicitly.

Notable design decisions

enable_http_endpoint() vs modify_db_cluster(enable_http_endpoint=True). The EnableHttpEndpoint parameter on modify_db_cluster applies only to Serverless v1. On Serverless v2 it silently does nothing. The setup script uses the dedicated enable_http_endpoint() operation, which is the correct call for Serverless v2 and Express clusters.

Typed exception on modify_db_cluster retry. After enable_http_endpoint(), the cluster can briefly reject modify_db_cluster with InvalidDBClusterStateFault even after the availability waiter returns. The retry uses tenacity Retrying on rds.exceptions.InvalidDBClusterStateFault (the typed boto3 exception) with exponential backoff. Each retry re-runs the availability waiter before attempting the modify.

DatabaseErrorException guard in rds_execute. After modify_db_cluster(MasterUserPassword=...), PostgreSQL can take 30 to 60 seconds to accept the new credential after the cluster reports "available". The Data API returns DatabaseErrorException with "password authentication failed" during that window. DatabaseErrorException also wraps genuine SQL errors, so the retry predicate checks the error message against a known-transient phrase list before retrying. Genuine SQL errors still raise immediately.

Secret ARN is not touched by the destroy script. Guide 6 Lambda functions reference the secret ARN in their environment variables. Destroying and recreating the secret would change the ARN and break those functions without a re-deploy. The destroy/setup cycle rebuilds only the cluster.

What is unchanged

DataAPIClient in backend/database/src/client.py already carried tenacity retry on execute() and begin_transaction() for auto-pause resume (DatabaseResumingException, BadRequestException: Communications link failure). This PR adds tenacity to backend/database/pyproject.toml to make that dependency explicit; the retry logic itself is not new.

Guide 6 and Guide 7 Terraform modules, all Lambda function code, the database schema, and the Secrets Manager secret ARN are untouched.

Known limitations

terraform destroy does not destroy the cluster. Students must run destroy_aurora_express.py before terraform destroy to avoid an orphaned cluster.

Scaling (min_capacity, max_capacity) is no longer a Terraform variable. It is configured as constants at the top of setup_aurora_express.py.

Future path

When hashicorp/aws adds WithExpressConfiguration support: reintroduce aws_rds_cluster, restore min_capacity and max_capacity as Terraform variables, delete bootstrap.auto.tfvars.json, and retire the bootstrap scripts. No downstream migration required.

Files changed

bootstrap/setup_aurora_express.py (new): 7-step idempotent cluster setup.
bootstrap/destroy_aurora_express.py (new): cluster teardown with confirmation prompt.
bootstrap/pyproject.toml (new): uv project; boto3, tenacity.
bootstrap/create_express_cluster.sh, destroy_express_cluster.sh, wait_for_cluster.sh: deleted.
terraform/5_database/main.tf: removed aws_rds_cluster; rds-data policy scoped to "*".
terraform/5_database/variables.tf: removed min_capacity, max_capacity; aurora_cluster_arn gets default = "".
terraform/5_database/outputs.tf: outputs read from var.aurora_cluster_arn.
terraform/5_database/terraform.tfvars.example: simplified.
backend/database/pyproject.toml: added tenacity>=9.1.2.
backend/database/src/client.py: tenacity dependency now explicit; retry logic unchanged.

Test plan

• Fresh free-tier account: cd bootstrap && uv run setup_aurora_express.py completes all 7 steps cleanly
• Re-run setup_aurora_express.py on an existing cluster: idempotent, no errors, no duplicate resources created
• cd terraform/5_database && terraform output returns correct aurora_cluster_arn, aurora_secret_arn, lambda_role_arn after setup
• cd backend/database && uv run test_data_api.py passes against the bootstrapped cluster
• uv run run_migrations.py && uv run seed_data.py && uv run verify_database.py all pass
• Guide 6 local tests (MOCK_LAMBDAS=true) pass unchanged
• cd bootstrap && uv run destroy_aurora_express.py: cluster deleted, bootstrap.auto.tfvars.json removed, terraform output aurora_cluster_arn returns ""
• Re-run setup_aurora_express.py after destroy: cluster recreated cleanly, same secret ARN preserved