[DENG-11197] Migrate orphaning_dashboard job from spark#538
Conversation
| python -m update_orphaning_dashboard.main --run-date 2026-06-08 --dry-run | ||
| ``` | ||
|
|
||
| Authenticate with `gcloud auth application-default login`. `--dry-run` writes |
There was a problem hiding this comment.
I guess we have to install the Google CLI to use this command: https://cloud.google.com/cli
@BenWu Could you please confirm?
There was a problem hiding this comment.
Yes some instructions on setup and accessing bigquery can be found here https://docs.telemetry.mozilla.org/cookbooks/bigquery/access.html#bigquery-access-request. You should already have the necessary access by default
antonioborondo
left a comment
antonioborondo
left a comment
There was a problem hiding this comment.
I was able to run the script locally and compare the data generated locally with the data generated on the website, and they match.
|
|
||
| ## Running locally | ||
|
|
||
| This job was written for Python 3.14. Compatibility with other versions isn't guaranteed. |
There was a problem hiding this comment.
Why this version in particular? Is there any incompatibility with any of the required libraries?
I was able to run the script without any problems using Python 3.12.10.
There was a problem hiding this comment.
I went with the latest version in the docker image so it has the longest time before it reaches end-of-life. The job probably works with much older versions and it should be fine to use whatever you have installed but I wrote that just in case there are issues. Generally, I would recommend using pyenv and using the same version as the image
3 participants
https://mozilla-hub.atlassian.net/browse/DENG-11197
Migration of the spark job in https://github.com/mozilla/telemetry-airflow/blob/main/jobs/update_orphaning_dashboard_etl.py. This pushes the computations into bigquery so spark isn't needed. The output is the same as the existing job and testable locally by running the frontend with:
# run date should line up with what's in https://telemetry.mozilla.org/update-orphaning/ python -m update_orphaning_dashboard.main --run-date 2026-05-31 --dry-run python serve_frontend.pySwitching to glean was not part of this.
Checklist for reviewer:
Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title)
Scan the PR and verify that no changes (particularly to
.circleci/config.yml) will cause environment variables (particularly credentials) to be exposed in test logsEnsure the container image will be using permissions granted to telemetry-airflow responsibly.