Skip to content

chore: migrate gsutil usage to gcloud storage#15845

Open
gurusai-voleti wants to merge 1 commit into
google:masterfrom
gurusai-voleti:gsutil_migration_infra_2
Open

chore: migrate gsutil usage to gcloud storage#15845
gurusai-voleti wants to merge 1 commit into
google:masterfrom
gurusai-voleti:gsutil_migration_infra_2

Conversation

@gurusai-voleti

Copy link
Copy Markdown
Contributor

No description provided.

@DavidKorczynski

Copy link
Copy Markdown
Collaborator

/gcbrun trial_build.py htslib json-c javacpp jansson janet --fuzzing-engines libfuzzer --sanitizers address

@DavidKorczynski

Copy link
Copy Markdown
Collaborator
Step #1 - "Legacy": INFO:root:================================================================
Step #1 - "Legacy": INFO:root:               PHASE 2: STARTED BUILDS
Step #1 - "Legacy": INFO:root:================================================================
Step #1 - "Legacy": INFO:root:Total projects with builds: 5
Step #1 - "Legacy": INFO:root:--- STARTED BUILDS ---
Step #1 - "Legacy": INFO:root:  - htslib:
Step #1 - "Legacy": INFO:root:    - Build Type: fuzzing
Step #1 - "Legacy": INFO:root:      GCB Build ID: e01fcf21-7ac7-4c12-a448-5b6d87019ba5
Step #1 - "Legacy": INFO:root:      GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/e01fcf21-7ac7-4c12-a448-5b6d87019ba5?project=oss-fuzz
Step #1 - "Legacy": INFO:root:      Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e01fcf21-7ac7-4c12-a448-5b6d87019ba5.txt
Step #1 - "Legacy": INFO:root:  - janet:
Step #1 - "Legacy": INFO:root:    - Build Type: fuzzing
Step #1 - "Legacy": INFO:root:      GCB Build ID: 22591772-ecbf-4094-be2b-0594001d5556
Step #1 - "Legacy": INFO:root:      GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/22591772-ecbf-4094-be2b-0594001d5556?project=oss-fuzz
Step #1 - "Legacy": INFO:root:      Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-22591772-ecbf-4094-be2b-0594001d5556.txt
Step #1 - "Legacy": INFO:root:  - jansson:
Step #1 - "Legacy": INFO:root:    - Build Type: fuzzing
Step #1 - "Legacy": INFO:root:      GCB Build ID: 84abad86-a509-40e5-9bdc-6919464b2437
Step #1 - "Legacy": INFO:root:      GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/84abad86-a509-40e5-9bdc-6919464b2437?project=oss-fuzz
Step #1 - "Legacy": INFO:root:      Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-84abad86-a509-40e5-9bdc-6919464b2437.txt
Step #1 - "Legacy": INFO:root:  - javacpp:
Step #1 - "Legacy": INFO:root:    - Build Type: fuzzing
Step #1 - "Legacy": INFO:root:      GCB Build ID: cea33b00-060a-4ebb-af58-cf0384b935f6
Step #1 - "Legacy": INFO:root:      GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/cea33b00-060a-4ebb-af58-cf0384b935f6?project=oss-fuzz
Step #1 - "Legacy": INFO:root:      Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-cea33b00-060a-4ebb-af58-cf0384b935f6.txt
Step #1 - "Legacy": INFO:root:  - json-c:
Step #1 - "Legacy": INFO:root:    - Build Type: fuzzing
Step #1 - "Legacy": INFO:root:      GCB Build ID: bd86e69c-8378-4a84-b695-e7ea28dbdd9f
Step #1 - "Legacy": INFO:root:      GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/bd86e69c-8378-4a84-b695-e7ea28dbdd9f?project=oss-fuzz
Step #1 - "Legacy": INFO:root:      Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-bd86e69c-8378-4a84-b695-e7ea28dbdd9f.txt
Step #1 - "Legacy": INFO:root:-----------------------
Step #1 - "Legacy": INFO:root:Waiting for 5 builds from 5 projects to complete...
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:            PHASE 1: BASE IMAGE BUILD REPORT
Step #3 - "Ubuntu 24.04": INFO:root:----------------------------------------------------------------
Step #3 - "Ubuntu 24.04": INFO:root:GCB Build ID: 87e23cb6-3106-4e79-8269-0bce07c016ec
Step #3 - "Ubuntu 24.04": INFO:root:GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/87e23cb6-3106-4e79-8269-0bce07c016ec?project=oss-fuzz-base
Step #3 - "Ubuntu 24.04": INFO:root:Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-87e23cb6-3106-4e79-8269-0bce07c016ec.txt
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-image-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-clang-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-clang-full-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-indexer-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-go-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-javascript-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-jvm-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-python-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-ruby-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-rust-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-builder-swift-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-runner-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:  - build-base-runner-debug-ubuntu-24-04: SUCCESS
Step #3 - "Ubuntu 24.04": INFO:root:----------------------------------------------------------------
Step #3 - "Ubuntu 24.04": INFO:root:Summary: 14 succeeded, 0 failed.
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:All base images built successfully.
Step #3 - "Ubuntu 24.04": INFO:root:"Build and Push Images" phase completed.
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:            PHASE 2: STARTING TEST BUILDS
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:Build type: fuzzing
Step #3 - "Ubuntu 24.04": INFO:root:  - Selected projects: 6 / 6 (due to successful production builds)
Step #3 - "Ubuntu 24.04": INFO:root:  - To build all projects, use the --force-build flag.
Step #3 - "Ubuntu 24.04": INFO:root:Starting to create and trigger builds for build type: fuzzing
Step #3 - "Ubuntu 24.04": INFO:root:Triggered all builds.
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:               PHASE 2: SKIPPED BUILDS
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:Total skipped builds: 6
Step #3 - "Ubuntu 24.04": INFO:root:--- SKIPPED BUILDS ---
Step #3 - "Ubuntu 24.04": INFO:root:  - fuzzing:
Step #3 - "Ubuntu 24.04": INFO:root:    - : Missing Dockerfile
Step #3 - "Ubuntu 24.04": INFO:root:    - htslib: Project requires legacy, but build version is ubuntu-24-04
Step #3 - "Ubuntu 24.04": INFO:root:    - janet: Project requires legacy, but build version is ubuntu-24-04
Step #3 - "Ubuntu 24.04": INFO:root:    - jansson: Project requires legacy, but build version is ubuntu-24-04
Step #3 - "Ubuntu 24.04": INFO:root:    - javacpp: Project requires legacy, but build version is ubuntu-24-04
Step #3 - "Ubuntu 24.04": INFO:root:    - json-c: Project requires legacy, but build version is ubuntu-24-04
Step #3 - "Ubuntu 24.04": INFO:root:-----------------------
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:               PHASE 2: STARTED BUILDS
Step #3 - "Ubuntu 24.04": INFO:root:================================================================
Step #3 - "Ubuntu 24.04": INFO:root:Total projects with builds: 0
Step #3 - "Ubuntu 24.04": INFO:root:--- STARTED BUILDS ---
Step #3 - "Ubuntu 24.04": INFO:root:-----------------------
Step #3 - "Ubuntu 24.04": INFO:root:Waiting for 0 builds from 0 projects to complete...
Step #3 - "Ubuntu 24.04": INFO:root:
Step #3 - "Ubuntu 24.04": All builds passed successfully!
Step #3 - "Ubuntu 24.04": INFO:root:------------------------
Step #3 - "Ubuntu 24.04": +------------------------------------------------------------------------------+
Step #3 - "Ubuntu 24.04": |                          BUILD REPORT: UBUNTU-24-04                          |
Step #3 - "Ubuntu 24.04": +------------------------------------------------------------------------------+
Step #3 - "Ubuntu 24.04": |  Total projects analyzed: 6                                                  |
Step #3 - "Ubuntu 24.04": |  [PASSED]  Successful builds: 0                                              |
Step #3 - "Ubuntu 24.04": |  [FAILED]  Failed builds: 0                                                  |
Step #3 - "Ubuntu 24.04": |  [SKIPPED] Skipped builds: 6                                                 |
Step #3 - "Ubuntu 24.04": +------------------------------------------------------------------------------+
Finished Step #3 - "Ubuntu 24.04"
Step #1 - "Legacy": INFO:root:
Step #1 - "Legacy": All builds passed successfully!
Step #1 - "Legacy": INFO:root:------------------------
Step #1 - "Legacy": +------------------------------------------------------------------------------+
Step #1 - "Legacy": |                             BUILD REPORT: LEGACY                             |
Step #1 - "Legacy": +------------------------------------------------------------------------------+
Step #1 - "Legacy": |  Total projects analyzed: 6                                                  |
Step #1 - "Legacy": |  [PASSED]  Successful builds: 5                                              |
Step #1 - "Legacy": |  [FAILED]  Failed builds: 0                                                  |
Step #1 - "Legacy": |  [SKIPPED] Skipped builds: 1                                                 |
Step #1 - "Legacy": +------------------------------------------------------------------------------+
Finished Step #1 - "Legacy"
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:            PHASE 1: BASE IMAGE BUILD REPORT
Step #2 - "Ubuntu 20.04": INFO:root:----------------------------------------------------------------
Step #2 - "Ubuntu 20.04": INFO:root:GCB Build ID: de143471-b10a-4151-a37e-b512975ff3a2
Step #2 - "Ubuntu 20.04": INFO:root:GCB Build URL: https://console.cloud.google.com/cloud-build/builds;region=us-central1/de143471-b10a-4151-a37e-b512975ff3a2?project=oss-fuzz-base
Step #2 - "Ubuntu 20.04": INFO:root:Log URL: https://oss-fuzz-gcb-logs.storage.googleapis.com/log-de143471-b10a-4151-a37e-b512975ff3a2.txt
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-image-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-clang-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-clang-full-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-indexer-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-go-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-javascript-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-jvm-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-python-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-ruby-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-rust-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-builder-swift-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-runner-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:  - build-base-runner-debug-ubuntu-20-04: SUCCESS
Step #2 - "Ubuntu 20.04": INFO:root:----------------------------------------------------------------
Step #2 - "Ubuntu 20.04": INFO:root:Summary: 14 succeeded, 0 failed.
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:All base images built successfully.
Step #2 - "Ubuntu 20.04": INFO:root:"Build and Push Images" phase completed.
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:            PHASE 2: STARTING TEST BUILDS
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:Build type: fuzzing
Step #2 - "Ubuntu 20.04": INFO:root:  - Selected projects: 6 / 6 (due to successful production builds)
Step #2 - "Ubuntu 20.04": INFO:root:  - To build all projects, use the --force-build flag.
Step #2 - "Ubuntu 20.04": INFO:root:Starting to create and trigger builds for build type: fuzzing
Step #2 - "Ubuntu 20.04": INFO:root:Triggered all builds.
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:               PHASE 2: SKIPPED BUILDS
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:Total skipped builds: 6
Step #2 - "Ubuntu 20.04": INFO:root:--- SKIPPED BUILDS ---
Step #2 - "Ubuntu 20.04": INFO:root:  - fuzzing:
Step #2 - "Ubuntu 20.04": INFO:root:    - : Missing Dockerfile
Step #2 - "Ubuntu 20.04": INFO:root:    - htslib: Project requires legacy, but build version is ubuntu-20-04
Step #2 - "Ubuntu 20.04": INFO:root:    - janet: Project requires legacy, but build version is ubuntu-20-04
Step #2 - "Ubuntu 20.04": INFO:root:    - jansson: Project requires legacy, but build version is ubuntu-20-04
Step #2 - "Ubuntu 20.04": INFO:root:    - javacpp: Project requires legacy, but build version is ubuntu-20-04
Step #2 - "Ubuntu 20.04": INFO:root:    - json-c: Project requires legacy, but build version is ubuntu-20-04
Step #2 - "Ubuntu 20.04": INFO:root:-----------------------
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:               PHASE 2: STARTED BUILDS
Step #2 - "Ubuntu 20.04": INFO:root:================================================================
Step #2 - "Ubuntu 20.04": INFO:root:Total projects with builds: 0
Step #2 - "Ubuntu 20.04": INFO:root:--- STARTED BUILDS ---
Step #2 - "Ubuntu 20.04": INFO:root:-----------------------
Step #2 - "Ubuntu 20.04": INFO:root:Waiting for 0 builds from 0 projects to complete...
Step #2 - "Ubuntu 20.04": INFO:root:
Step #2 - "Ubuntu 20.04": All builds passed successfully!
Step #2 - "Ubuntu 20.04": INFO:root:------------------------
Step #2 - "Ubuntu 20.04": +------------------------------------------------------------------------------+
Step #2 - "Ubuntu 20.04": |                          BUILD REPORT: UBUNTU-20-04                          |
Step #2 - "Ubuntu 20.04": +------------------------------------------------------------------------------+
Step #2 - "Ubuntu 20.04": |  Total projects analyzed: 6                                                  |
Step #2 - "Ubuntu 20.04": |  [PASSED]  Successful builds: 0                                              |
Step #2 - "Ubuntu 20.04": |  [FAILED]  Failed builds: 0                                                  |
Step #2 - "Ubuntu 20.04": |  [SKIPPED] Skipped builds: 6                                                 |
Step #2 - "Ubuntu 20.04": +------------------------------------------------------------------------------+
Finished Step #2 - "Ubuntu 20.04"
Starting Step #4 - "Generate Report"
Step #4 - "Generate Report": Already have image: gcr.io/oss-fuzz-base/trial-build
Step #4 - "Generate Report": Generating final build report...
Step #4 - "Generate Report": +------------------------------------------------------------------------------------------+
Step #4 - "Generate Report": |                                 FAILURE COMPARISON TABLE                                 |
Step #4 - "Generate Report": +------------------------------------------------------------------------------------------+
Step #4 - "Generate Report": |  Project                       |      Legacy      |   Ubuntu 20.04   |   Ubuntu 24.04    |
Step #4 - "Generate Report": | -------------------------------+------------------+------------------+------------------ |
Step #4 - "Generate Report": |                                |       SKIP      |       SKIP      |       SKIP          |
Step #4 - "Generate Report": |  htslib                        |       PASS      |       SKIP      |       SKIP          |
Step #4 - "Generate Report": |  janet                         |       PASS      |       SKIP      |   

@DavidKorczynski

Copy link
Copy Markdown
Collaborator

This is likely affecting OSS-Fuzz-Gen. However, that project is stale for now so I don't think we should be concerned about that.

},
{
'name': 'gcr.io/cloud-builders/gsutil',
'name': 'gcr.io/cloud-builders/gcloud',

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't you missing the "storage" for gcloud storage cp in the args list?

@DavidKorczynski

DavidKorczynski commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

The only usage of project_experiment.py I can find is https://github.com/google/oss-fuzz-gen/blob/c0982c5d40a7e93ce70fd319705804b9a29954d0/data_prep/project_src.py#L157

I don't think this is really used to be honest

################################################################################

gsutil -h "Cache-Control:no-cache,max-age=0" -m cp -r bower_components index.html src manifest.json gs://oss-fuzz-build-logs
gcloud storage cp --recursive bower_components index.html src manifest.json gs://oss-fuzz-build-logs --cache-control="no-cache,max-age=0"-build-logs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not correct due to that last -build-logs in
gcloud storage cp --recursive bower_components index.html src manifest.json gs://oss-fuzz-build-logs --cache-control="no-cache,max-age=0"-build-logs. Can you double-check?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants