Skip to content

Move waiting logic for host orchestrator settlement to backend#521

Open
k311093 wants to merge 1 commit into
google:mainfrom
k311093:wait
Open

Move waiting logic for host orchestrator settlement to backend#521
k311093 wants to merge 1 commit into
google:mainfrom
k311093:wait

Conversation

@k311093

@k311093 k311093 commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator

While interacting with host orchestrator right after creating host
API is returned, there is a possibility of getting 503 service
temporarily unavailable error because host it created and booted but
host orchestrator is not executed / ready yet.

Handling logic for waiting host orchestrator readiness is currently
in the client logic. Moving this logic into cloud orchestrator API
endpoint would make the client that uses host chestrator API simpler.

Context: b/501288123

@k311093 k311093 force-pushed the wait branch 5 times, most recently from deec9a3 to 2091716 Compare April 15, 2026 04:34
@k311093 k311093 marked this pull request as ready for review April 15, 2026 04:45
@k311093 k311093 requested review from 0405ysj and ikicha April 15, 2026 04:45
Comment thread pkg/app/instances/gce.go Outdated
if err != nil {
return nil, err
}
client, err := g.InstanceManager.GetHostClient(g.Zone, host.Name)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From GceIM, GetHostClient internally invokes getHostInstance, which seems to be calling GCE API to retrieve the host. However, this method already has ins which is typed as *compute.Instance.

I think it's better not to get *compute.Instance twice, also not to bring new fields(InstanceManager, Zone) into opResultGetter.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed logic to pass *compute.Instance and reuse it if specified.

@k311093 k311093 force-pushed the wait branch 2 times, most recently from 493fffb to b5e8c57 Compare April 15, 2026 05:59
Comment thread pkg/app/instances/gce.go Outdated

@ser-io ser-io left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.

CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with. To do this properly, you'd need to add and implement a new http endpoint that tests whether the host is ready, still clients would need to make these calls after CreateHost returned in order to determine whether the host is ready to interact with. However, it's up to every client implementation how they want to handle this, they can always keep doing what the current client is doing.

Comment thread pkg/app/instances/gce.go Outdated
@jemoreira

Copy link
Copy Markdown
Member

Adding an inside logic to wait for when the host is ready to interact with shouldn't be part of the CreateHost request logic on the backend side.

CreateHost returns success when the host is created and host resources are allocated, not when the host is ready to interact with.

What use is a host that doesn't have a host orchestrator? If the answer is "none", then it would make sense to only mark the operation to create a host as completed once the host orchestrator is ready. It simplifies the api and puts the need for waiting in a single place (the cloud orchestrator) rather than in every client. The CO is already waiting for the host to be created, it might as well wait a bit longer.

@jemoreira

Copy link
Copy Markdown
Member

Context: b/501288123

Please add more details to the description and the title of the PR.

What is "host orchestrator settlement"?

Which backend? The HO or the CO or something else?

Why is this change useful/needed? What are the pros and cons? Even if you write a description in the bug that's not publicly available like this PR is.

@k311093

k311093 commented Apr 16, 2026

Copy link
Copy Markdown
Collaborator Author

Context: b/501288123

Please add more details to the description and the title of the PR.

What is "host orchestrator settlement"?

Which backend? The HO or the CO or something else?

Why is this change useful/needed? What are the pros and cons? Even if you write a description in the bug that's not publicly available like this PR is.

Updated commit message with descriptions.

@0405ysj

0405ysj commented Apr 16, 2026

Copy link
Copy Markdown
Collaborator

What use is a host that doesn't have a host orchestrator? If the answer is "none"

+1. I expect users are understanding the meaning of host as a host environment to launch Cuttlefish instances, and the route of orchestrating CF instances is Host Orchestrator. I think health of Host Orchestrator is more representative than health of computing unit(e.g. GCE or docker instance).

From API view, I expect users create a host and verify with 3 steps; POST /v1/zones/{zone}/hosts, POST /v1/zones/{zone}/operations/{operation}/:wait, GET /v1/zones/{zone}/hosts/{host}/. Invoking those endpoints should be responsible by clients though. But having another round of wait by setting retry count for requesting GET /v1/zones/{zone}/hosts/{host}/ would be unexpected behavior in users' perspective, I think.

@k311093 k311093 force-pushed the wait branch 2 times, most recently from b3d13e7 to 8d2831f Compare April 16, 2026 03:55
@k311093 k311093 requested review from 0405ysj and ser-io April 16, 2026 04:47
Comment thread pkg/app/instances/gce.go Outdated
Comment thread pkg/app/instances/gce.go Outdated
@k311093

k311093 commented Apr 21, 2026

Copy link
Copy Markdown
Collaborator Author

Let's do #521 (comment)

Got it. Thanks!

@k311093 k311093 force-pushed the wait branch 4 times, most recently from f8c9f9f to ef76e5e Compare May 6, 2026 06:32
@k311093

k311093 commented May 6, 2026

Copy link
Copy Markdown
Collaborator Author

Added endpoint :wait-host-availability endpoint and changed cvdr to use it. When :wait-host-availability fails, fallback to the logic used before.

@k311093 k311093 requested review from jemoreira and ser-io May 6, 2026 06:51
Comment thread pkg/app/app.go Outdated
@k311093 k311093 force-pushed the wait branch 6 times, most recently from 1a22a6e to ec38115 Compare May 26, 2026 06:25
@k311093 k311093 requested a review from ser-io May 27, 2026 05:18
Comment thread pkg/app/app.go Outdated
@k311093 k311093 force-pushed the wait branch 5 times, most recently from c4d69b0 to ce31cf8 Compare June 4, 2026 06:07
@k311093 k311093 force-pushed the wait branch 5 times, most recently from 0d668a2 to 427a0a1 Compare June 5, 2026 07:11
While interacting with host orchestrator right after creating host API is returned, there is a possibility of getting 503 service temporarily unavailable error because host is created and booted but host orchestrator is not executed / ready yet.

Handling logic for waiting host orchestrator readiness is currently in the client logic. Moving this logic into cloud orchestrator API endpoint (specifically inside WaitOperation) makes the client simpler and reduces redundant API endpoints.

Context: b/501288123

TAG=agy
CONV=f5f788b6-7aa2-4614-b441-87f3ad9fed4e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants