perf(composer): upgrade to Airflow 3 and eliminate redundant post-cre…

[jeffonelson]

…ation package update

[gemini-code-assist]

Code Review

This pull request updates the setup_composer.sh script by changing the default Airflow image version to composer-3-airflow-3, removing the step that installs required PyPI packages (dbt-bigquery and orchestration-pipelines), and optimizing the environment metadata retrieval into a single gcloud command. Feedback points out that composer-3-airflow-3 is an invalid image version since Airflow 3 is not yet supported in Cloud Composer 3, and warns that removing the PyPI package installation step will cause DAG import errors. A suggestion is provided to use composer-3-airflow-2 and restore the package installation as a separate update step.

[gemini-code-assist]

There are two critical issues with the updated environment creation command:

1. Invalid Image Version: Cloud Composer 3 does not currently support Airflow 3 (which is still in pre-release/development). Specifying composer-3-airflow-3 will cause the gcloud composer environments create command to fail with an INVALID_ARGUMENT error. To use the latest supported Airflow 2.x version in Cloud Composer 3, you should use composer-3-airflow-2.
2. Missing Required PyPI Packages: The post-creation package update step was removed, but the required packages (dbt-bigquery and orchestration-pipelines) were completely omitted. This will cause the DAGs to fail with ImportError when they run. When provisioning Cloud Composer environments, verify if installing PyPI packages during the environment creation command is supported by the specific CLI version being used; if not, perform the package installation as a separate update step.

We can address both issues by updating the image version and installing the packages as a separate update step to ensure compatibility across CLI versions.

Suggested change

	--image-version="${IMAGE_VERSION:-composer-3-airflow-3}"
	--environment-size=small
	--service-account="${COMPOSER_SA_EMAIL}"
	--quiet
	--image-version="${IMAGE_VERSION:-composer-3-airflow-2}"
	--environment-size=small
	--service-account="${COMPOSER_SA_EMAIL}"
	--quiet
	gcloud composer environments update "${COMPOSER_ENV_NAME}" \
	--location="${LOCATION}" \
	--update-pypi-packages="dbt-bigquery,orchestration-pipelines"

References

1. When provisioning Cloud Composer environments, verify if installing PyPI packages during the environment creation command is supported by the specific CLI version being used; if not, perform the package installation as a separate update step.