fix(foreman): report gate Job DeadlineExceeded as GATE-ERROR and make the gate deadline settable (#1748)

[Defilan]

What

Two changes to the verify gate's run_gate_job tool. A gate Job killed by its activeDeadlineSeconds now maps to GATE-ERROR instead of GATE-FAIL, with a summary that names the deadline and the phase the Job was in (gate Job exceeded its 1800s deadline during per-hunk mutation coverage (base=...)). And the deadline is settable: the verifying Agent's spec.execution.activeDeadlineSeconds is threaded through ToolDeps.GateActiveDeadlineSeconds into RunGateJobToolConfig.ActiveDeadlineSeconds, with unset keeping the 1800 s default.

Why

Refs #1748

wl-1725-agw-pool-refs passed its bite check, entered the per-hunk mutation pass on internal/controller (39 hunks at about five minutes of envtest each), and was killed at 1800 s. pollForTerminal read Job.Status.Failed >= 1 and reported GATE-FAIL, so a branch whose checks were passing was recorded as a failing gate and its review cascade-failed. Nothing could raise the deadline without a code change. A timeout is a gate infrastructure problem, not a failing test, and the verdict decides whether the Workload treats the branch as bad.

How

gateJobDeadlineExceeded walks the Job's conditions for JobFailed with reason DeadlineExceeded, the shape the Kubernetes Job controller sets on a deadline kill; every other failure reason keeps GATE-FAIL. pollForTerminal returns a deadlineHit flag, and Execute appends the last === <phase> === banner found in the log tail (lastGatePhase) so the operator can see whether the standard checks survived and where the Job stalled. Defaults are applied before polling, so the summary always names the effective deadline.

The deadline wiring mirrors what run_coder_job already does for the coder Job: cmd/foreman-agent/main.go reads agent.Spec.Execution.ActiveDeadlineSeconds (*int64), clamps it to int32 via gateActiveDeadlineSeconds (nil, zero and negative keep the default; values above MaxInt32 saturate), and passes it through ToolDeps. PollTimeout keeps deriving as twice the deadline.

Tests: TestRunGateJob_DeadlineExceededProducesGATEERROR drives the fake apiserver through a DeadlineExceeded terminal state and asserts GATE-ERROR plus the phase suffix; TestRunGateJob_FailedProducesGATEFAIL still holds for a plain failure; TestBuildAll_ThreadsGateActiveDeadlineSeconds and TestGateActiveDeadlineSeconds cover the wiring and the clamp.

Not in this PR: bounding the mutation pass itself, which is #1749. To use the new knob on a fleet, roll the agent image and set spec.execution.activeDeadlineSeconds on the verifier Agent.

Checklist

• Tests added/updated
• make test passes locally (verify gate GATE-PASS on the branch; CI green)
• make lint passes locally (CI)
• Commit messages follow conventional commits
• All commits are signed off (git commit -s) per DCO (Foreman-bot sign-off under my address; squash-merge attributes the commit to me, as with fix(foreman): clear Dispatched condition on terminal Workloads #1741)
• AI assistance (if any) is disclosed above, per CONTRIBUTING.md
• Documentation updated (if user-facing change): not user-facing; the Agent field already exists and is documented

Assisted-by: Foreman (coder dsv4-flash-coder, DeepSeek-V4-Flash-Vision-Exp on vLLM) generated the code and tests from the decided design in #1748; the verify gate ran the full check suite. I reviewed the diff by hand, ran the two packages' tests, vet and gofmt on the branch, and own the sign-off.

[codecov]

Codecov Report

❌ Patch coverage is 66.66667% with 16 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
cmd/foreman-agent/main.go	35.29%	11 Missing ⚠️
pkg/foreman/agent/tools/run_gate_job.go	80.76%	4 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Defilan]

Human review of the Foreman branch. Approve, pending finalize.

Read the whole diff against the decided design in #1748 and ran go test, go vet and gofmt -l for pkg/foreman/agent/tools and cmd/foreman-agent on the branch: green. Points checked:

• gateJobDeadlineExceeded keys on JobFailed + DeadlineExceeded, the exact condition the wl-1725 Job carried; any other failure reason still returns GATE-FAIL, and TestRunGateJob_FailedProducesGATEFAIL proves it.
• The phase suffix comes from the last === ... === banner in the tail. On the wl-1725 tail that is per-hunk mutation coverage (base=9786684f6...), which is what an operator needs to see.
• Execute applies applyConfigDefaults before polling, so the summary names the effective deadline (1800 when unset), not a zero.
• The *int64 to int32 clamp mirrors clampInt32 next to it and the table test covers nil, zero, negative, exact max and overflow.
• PollTimeout is untouched and keeps deriving as twice the deadline, so a raised deadline cannot outlive the poller.

Two notes, neither blocking:

1. Downstream, a GATE-ERROR still cascades the review to INCOMPLETE and the Workload ends Failed. What changes is the failure reason (FailureGateError, infrastructure) instead of a bad-branch signal, and the operator now has the knob to re-run with more time. That is what the issue asked for; the cheaper pass itself is [FEATURE] Bound the per-hunk mutation pass: run only the changed specs, honour a time budget, report partial coverage #1749.
2. The in-cluster reviewer's approval listed api/foreman/v1alpha1/agent_types.go in filesTouchedClaimed; that file is not in the diff. The claim is harmless here, but it is the kind of drift the scope rail should catch, and it did not (scopeDriftDetected: false).

To put it to use on the lab fleet after merge: roll the foreman-agent image, then give the verifier gate Agent an execution: {activeDeadlineSeconds: 7200} block (it has none today).

Finalization: the commit is bot-signed. scripts/foreman-finalize.sh --branch foreman/wl-1748-gate-deadline/issue-1748 squashes to one human-signed commit.