Skip to content

Serialize long-running webhook actions - #37

Open
Lun Yue (lunyue-ms) wants to merge 9 commits into
Azure:mainfrom
lunyue-ms:fix/serialize-webhook-actions
Open

Serialize long-running webhook actions#37
Lun Yue (lunyue-ms) wants to merge 9 commits into
Azure:mainfrom
lunyue-ms:fix/serialize-webhook-actions

Conversation

@lunyue-ms

@lunyue-ms Lun Yue (lunyue-ms) commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Why I did it

Concurrent webhook deliveries can start multiple long-running GitHub and Azure DevOps actions at the same time. Synchronous child processes block the Node.js event loop, while overlapping handlers can lose or corrupt work.

ADO: https://msazure.visualstudio.com/One/_workitems/edit/39378257

How I did it

  • Run long-lived automation scripts asynchronously through a shared single-concurrency queue.
  • Serialize /azpw retry requests and replace blocking ADO curl calls with asynchronous HTTPS requests.
  • Refresh credentials when queued work starts instead of when it is received.
  • Bound pending work and report queue saturation without posting a false retry-success message.
  • Keep Event Hub telemetry failures from blocking the webhook action.

How to verify it

npx jest test/action_queue.test.js test/issue_comment.test.js --runInBand
node --check action_queue.js
node --check auto_cherrypick.js
node --check conflict_detect.js
node --check issue_comment.js

Result: 2 test suites and 5 tests passed.

Lun Yue (lunyue-ms) and others added 3 commits August 25, 2026 08:18
@lunyue-ms
fix(webhook): serialize long-running actions
a14bef5 
Keep the Node.js event loop responsive while processing concurrent GitHub webhook deliveries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@lunyue-ms
fix(webhook): harden queued action handling
ec0bcd2 
Refresh credentials at execution time, bound pending work, and keep retry and telemetry paths from blocking webhook processing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@lunyue-ms
fix(webhook): handle queue saturation
79482b7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

/azp run

@lunyue-ms
fix(webhook): delay Azure PR triggers
3c85e81 
Azure Pipelines can reject /azp run while the latest PR revision is still synchronizing. Match the other automation paths with a 10-second delay.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

/azp run

@lunyue-ms
fix(webhook): guard delayed PR triggers
b495b66 
Keep webhook responses below GitHub timeout while delaying Azure triggers. Skip stale or closed PR revisions and deduplicate pending comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

/azp run

@lunyue-ms
fix(checks): mark conflicts action required
dbc80ae 
Report detected and existing sync conflicts as action-required checks so GitHub shows an orange intervention state while required checks remain blocking. Keep unexpected errors as failures.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/azp run

@lunyue-ms
revert: keep conflict checks unchanged
6769e7c 
Revert the global action-required mapping because it affects every ms_conflict check, not only the code-sync pipeline.

This reverts commit dbc80ae.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/azp run

@lunyue-ms
fix(webhook): retry stale Azure triggers
8cd2ca4 
Retry an automatically issued /azp run once when Azure has not yet registered the unchanged PR revision. Skip retries after revision changes, merge conflicts, or a validation start.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/azp run

@lunyue-ms
fix(webhook): use 15s trigger delay
628c168 
Remove the stale-comment retry flow and give Azure more time to register the pull request revision before issuing /azp run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Lun Yue <17232861+lunyue-ms@users.noreply.github.com>
@mssonicbld

mssonicbld commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

/azp run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lunyue-ms @mssonicbld