EDGEAI-1554: Make the nightly gate work, and exempt the advisory lane from it - #26
Conversation
The condition was `event == schedule && last == sha`, so a manual dispatch bypassed the change check silently. Three nightlies ran on hal today, two of them rebuilding a commit whose nightly had already covered it, each re-running the board lane for no new information. Drop the event-name test so the gate applies to every trigger, and add an explicit `force` input for deliberately re-running a green nightly. The comparison stays against the last SUCCESSFUL run, which is what makes this safe: a nightly that failed leaves the last success on a different commit, so an unchanged main still re-runs without `force`. That is the case you want after fixing a lane rather than the code, and it is how today's runs would still have been allowed. Signed-off-by: Sébastien Taylor <[email protected]>
There was a problem hiding this comment.
🔵 Needs a closer look
The maintained caller template does not forward the new force input.
Pull request overview
Updates the reusable nightly gate to prevent redundant runs across all triggers while allowing forced reruns.
Changes:
- Adds a boolean
forceoverride. - Applies successful-commit gating universally.
- Documents the behavior in
CHANGELOG.md.
File summaries
| File | Summary |
|---|---|
CHANGELOG.md |
Documents the updated nightly gate behavior. |
.github/workflows/nightly-gate.yml |
Implements universal gating and forced reruns. |
Review details
Suppressed comments (1)
.github/workflows/nightly-gate.yml:6
- The canonical
templates/nightly.ymlstill declares a bareworkflow_dispatchand never forwards aforcevalue to this input. Repositories copied from the maintained caller skeleton therefore cannot use the documented manual override without an extra hand edit; please update that template alongside this reusable-workflow API (or explicitly document why the skeleton omits it).
force:
- Files reviewed: 2/2 changed files
- Comments generated: 0
- Review effort level: Lite (auto)
Note
Copilot is running an experiment and ran this review at Lite.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The gate asked whether main had moved since the last *successful* nightly. In a repository whose nightly is not consistently green that compares against an arbitrarily old commit: hal's last green nightly on main was 2025-12-08, so the gate had not skipped once in nine months. Against the live API, status=success returns 7c1bbc1c from December while status=completed returns 06bbe5f3, which is current main -- the first skips nothing, the second skips correctly. A failed run still looked at the commit, so it answers the question. Re-running an unchanged commit after fixing a lane is what `force` is for: an explicit request rather than an accident of run history. Forward `force` from templates/nightly.yml, which declared a bare workflow_dispatch and gave callers copied from the skeleton no way to use the documented override. `inputs` is null on a schedule trigger, so the `|| false` fallback is required or the boolean input rejects an empty string. Signed-off-by: Sébastien Taylor <[email protected]>
Every other nightly lane asks a question about the code, so skipping it on an unchanged commit loses nothing. cargo audit asks a question about the RustSec database, which changes daily whether the code does or not. Gated, it goes quiet exactly when a new advisory lands against a frozen main -- not hypothetical: the 2026-09-15 nightly failed on RUSTSEC-2026-0204 against a Cargo.lock nobody had touched. Move it to advisories.yml, called without `needs: changed`. It parses Cargo.lock and never compiles, so it skips setup-rust entirely and is seconds of a standard runner. The runner label is hard-coded rather than honouring the caller's runner-class: an every-night lane must not be able to land on a `larger` runner, which bills by the minute even on public repositories where the hosted class is free and unmetered. Advisories are downgraded through `.cargo/audit.toml` in the calling repository, which cargo-audit reads from the working directory -- a reviewable file in the repo it applies to, rather than a workflow input. What is left of nightly-extra is the hack lane, so rename it to Feature combinations and skip it when hack-args is empty instead of booting a runner to install tools and run nothing. hal sets hack-args to empty, so that job stops costing anything at all. Signed-off-by: Sébastien Taylor <[email protected]>
The SHA-pin lint checks the shape of a pin, not that the referenced file exists at it. Every templates/ pin is a stale placeholder, which was harmless while they all resolved; advisories.yml is new, so a pin predating it now 404s at run time instead of failing at lint. Say so in the header rather than leaving a skeleton that looks copyable. Signed-off-by: Sébastien Taylor <[email protected]>
There was a problem hiding this comment.
🟡 Changes recommended
The template pins revisions that lack both the new advisory workflow and the forwarded force input.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
templates/nightly.yml:31
advisories.ymldoes not exist at the pinnedeec0cb31…revision, so this reusable-workflow call cannot be resolved and the generated nightly workflow will be invalid. Pin the template to a commit that includes the new workflow.
# the RustSec database, not about the code, so gating it would go quiet
- Files reviewed: 6/6 changed files
- Comments generated: 1
- Review effort level: Balanced (auto)
Note
Copilot is running an experiment and ran this review at Balanced.
Copilot flagged that templates/nightly.yml referenced advisories.yml at a pin that predates it. The pin cannot be fixed as asked: no commit containing advisories.yml exists until this merges. The defect is also wider than that one line -- the same stale pin makes the new `force` forwarding invalid, because the nightly-gate at eec0cb3 does not declare that input. Every template self-pin had drifted to a commit from nine months ago, and a stale SHA is worse than an obviously invalid one because it still resolves: a copied skeleton silently runs old CI. Pin them to all zeros instead, so a skeleton that was not adjusted fails immediately and says what to fix. The existing SHA-pin check cannot catch this, since a stale SHA is a valid SHA, so add a step asserting the placeholder. Verified by extracting it verbatim from ci.yml and running it both ways: it passes on the current templates, and on a copy with the eec0cb3 pin restored it fails naming that exact line. Third-party actions in templates/ keep real pins and are copyable as-is. Signed-off-by: Sébastien Taylor <[email protected]>
Addressing the reviewFinding: I can't do what was literally suggested — pin the template to a commit containing the new workflow — because no such commit exists until this merges. So I looked at why the pin was wrong in the first place. The defect is wider than the one line. And the pins can never be right. A template that pins its own repository is stale the moment the next commit lands, so this recurs by construction. Fix: template self-pins are now an all-zero placeholder. uses: EdgeFirstAI/.github/.github/workflows/advisories.yml@0000000000000000000000000000000000000000A stale SHA is worse than an obviously invalid one precisely because it resolves — a copied skeleton silently runs nine-month-old CI, which is the failure mode that hid here. An unresolvable ref fails immediately and says what to fix. Third-party actions in Enforced, not just documented. The existing SHA-pin check can't catch this — a stale SHA is a valid SHA — so there's a new lint step asserting the placeholder. Verified by extracting it verbatim from The failing case reproduces exactly the line this review flagged. Confirmed running in CI on |
There was a problem hiding this comment.
🟡 Changes recommended
A failed or cancelled gate run can cause the next nightly to incorrectly skip all gated lanes.
Get a fresh assessment by requesting another Copilot review.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Balanced (auto)
Note
Copilot is running an experiment and ran this review at Balanced.
| # Re-running an unchanged commit after fixing a lane is what `force` | ||
| # is for -- an explicit request, not an accident of run history. | ||
| # completed excludes the current in-progress run. | ||
| last="$(gh api "repos/${GITHUB_REPOSITORY}/actions/workflows/${wf}/runs?status=completed&branch=main&per_page=1" --jq '.workflow_runs[0].head_sha // ""')" |
A run can be cancelled before a single lane starts, yet it is still "completed" at its commit. Taking the newest completed run meant such a run would suppress the next night's nightly having determined nothing about the code. Same for timed_out, stale, neutral, action_required and startup_failure. Count only success and failure: both mean the lanes ran and reported. Anything else looks further back, which at worst re-runs a nightly -- the safe direction for a gate to fail. Verified against the live API rather than a stub, using hal's ci.yml history, which has 13 cancelled runs. On a window whose newest run is cancelled, the old expression returns that run's sha while the new one returns the next verdict-bearing run; on a window holding only a cancelled run it returns empty, so the gate runs. Signed-off-by: Sébastien Taylor <[email protected]>
Addressing the second reviewFinding: a failed or cancelled gate run can make the next nightly skip the gated lanes. The cancelled half is a real hole and is now fixed. A run can be cancelled before a single lane starts — or the gate job itself can fail — yet the run is still The gate now counts only runs that reached a verdict: Discarded: Verified against the live API, not a stub. hal's And a window holding only a cancelled run returns empty, so the gate runs. The A failed run did look at its commit: the lanes ran and reported. Treating failure as "no verdict" is what the first version of this PR effectively did by filtering on So a red nightly on an unchanged Decision table re-verified after the change; all five rows unchanged. |
There was a problem hiding this comment.
🟡 Changes recommended
Failed gate runs can incorrectly suppress later nightly execution, and the hosted advisory exception conflicts with canonical runner policy.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
.github/workflows/advisories.yml:34
- This deliberate hosted, ungated nightly job conflicts with the repository's canonical policy:
.github/copilot-instructions.md:31-35says Nightly runs only whenmainmoves, and lines 54-69 require thelargerclass for Nightly. Add an explicit advisory-scanning exception to that source-of-truth document so contributors and audits do not apply the existing policy and undo this behavior.
- Files reviewed: 10/10 changed files
- Comments generated: 1
- Review effort level: Balanced
| # back instead, which at worst re-runs a nightly. | ||
| # | ||
| # status=completed also excludes this run, which is in_progress. | ||
| last="$(gh api "repos/${GITHUB_REPOSITORY}/actions/workflows/${wf}/runs?status=completed&branch=main&per_page=30" --jq '[.workflow_runs[] | select(.conclusion == "success" or .conclusion == "failure")][0].head_sha // ""')" |
copilot-instructions.md is the source of truth for the tiers and the runner classes, and advisories.yml breaks two of its rules: Nightly runs only when main moved, and Nightly uses the larger class. Both are broken deliberately, but an undocumented exception is one a future contributor or an audit will helpfully undo. State it, with the reasoning and the limits: not gated because the question is about the RustSec database rather than the code; hosted and hard-coded because "cost accepted" relies on the gate bounding Nightly to days when main moved, and an ungated lane has no such bound. Say plainly that a lane which builds anything does not qualify, so this does not become a general escape hatch. Signed-off-by: Sébastien Taylor <[email protected]>
Addressing the policy conflictFinding:
Round summary
All checks green. |
Picks up EdgeFirstAI/.github#26. Two caller-visible changes come with it. The nightly gate now applies on every trigger and compares against the last run that reached a verdict, so a workflow_dispatch no longer bypasses it silently. Forward a `force` input for deliberately re-running an unchanged commit; `inputs` is null on the schedule trigger, so the `|| false` fallback is required or the gate's boolean input rejects an empty string. cargo audit moved out of rust-full into advisories.yml, so it has to be called explicitly, and it is called without `needs: changed`. Every other lane here asks a question about the code and loses nothing by skipping an unchanged commit; cargo audit asks a question about the RustSec database, which changes daily whether the code does or not. The 2026-09-15 nightly failed on RUSTSEC-2026-0204 against a Cargo.lock nobody had touched, which a gated lane would have missed. Drop the shared SHA from the workflows README. It was a second copy of the pin that Dependabot does not update, and it was stale exactly one re-pin after the paragraph claiming the SHA appears only in `uses:`. Signed-off-by: Sébastien Taylor <[email protected]>
Part 1 — the gate never skipped
Two separate things stopped it.
It only applied on
schedule.A
workflow_dispatchbypassed the check silently, so re-dispatching a nightly rebuilt the same commit and re-ran the board — the slowest and only hardware-bound lane — for no new information.It compared against the last successful run. This is the more serious one, because it makes the gate useless in exactly the repository that needs it. Against the live API:
status=success7c1bbc1cstatus=completed06bbe5f3mainhal's last green nightly on
mainwas nine months ago, so every comparison since has been against a December commit. A failed run still looked at the commit, so it answers the question the gate asks.Now: applies on every trigger, compares against the last completed run of any conclusion.
abcabcabcdefdefabcabcabcdefVerified by extracting the
run:block verbatim from the YAML and executing all five rows underset -euo pipefailwithghstubbed.Re-running an unchanged commit after fixing a lane is
force's job — an explicit request rather than an accident of run history.Part 2 — the advisory lane must not be gated
A working gate creates a hole. Every other nightly lane asks a question about the code, so skipping an unchanged commit loses nothing.
cargo auditasks a question about the RustSec database, which changes daily whether the code does or not — so a gated-off nightly goes quiet exactly when a new advisory lands against a frozenmain.Not hypothetical. Today's nightly failed on RUSTSEC-2026-0204 against a
Cargo.locknobody had touched, which is why #193 exists.New
advisories.yml, called withoutneeds: changed:Running every night is only defensible if it costs nothing, so:
setup-rust.cargo auditparsesCargo.lockand never compiles; a toolchain install and cargo cache would be the only slow parts of an otherwise instant job.runs-on: ubuntu-24.04hard-coded, not the caller'srunner-class. Percopilot-instructions.mdthe hosted class is "free and unmetered on public repositories", whilelargerbills by the minute even there. A caller cannot put an every-night lane on a billed runner by accident.runner-audit's billed-label regex does not match it..cargo/audit.tomlin the calling repo — a reviewable file where it applies, not a workflow input.nightly-extrais now just the hack laneWith audit gone it does one thing, so it is renamed Feature combinations and skips entirely when
hack-argsis empty rather than booting a runner to install tools and run nothing. hal setshack-args: "", so that job stops costing anything.Behaviour change for callers:
rust-fullno longer runscargo audit. Callers must add theadvisories.ymljob. hal is the only caller today andtemplates/nightly.ymlis updated.Caller forwarding
Copilot was right that the skeleton never forwarded
force. Fixed, along with the ungatedadvisoriesjob:The
|| falsefallback is load-bearing:inputsis null on a schedule trigger, and the boolean input rejects the empty string that would otherwise result.Part 3 — template self-pins (from review)
Copilot flagged that
advisories.ymldoes not exist at the template's pinnedeec0cb31…. It does not, and the same stale pin makes the newforce:forwarding invalid too, sincenightly-gateat that commit declares no such input.The literal suggestion — pin to a commit containing the new workflow — is impossible before this merges, and would rot again on the next commit, because a template that pins its own repository is stale as soon as one lands.
Template self-pins are now an all-zero placeholder. A stale SHA is worse than an obviously invalid one because it resolves: a copied skeleton silently runs nine-month-old CI, which is how this hid. An unresolvable ref fails immediately. Third-party actions in
templates/keep real pins.A new lint step enforces it — the existing SHA-pin check cannot, since a stale SHA is a valid SHA. Extracted verbatim and run both ways: passes on current templates, and with
eec0cb31restored fails naming the exact line the review flagged.Follow-up
hal picks both up on its next re-pin, which is where its
nightly.ymlgains theforceinput and theadvisoriesjob.