Skip to content

EDGEAI-1554: Make the nightly gate work, and exempt the advisory lane from it - #26

Merged
sebastient merged 7 commits into
mainfrom
feature/EDGEAI-1554-gate-all-triggers
Sep 16, 2026
Merged

sebastient merged 7 commits into
mainfrom
feature/EDGEAI-1554-gate-all-triggers

Conversation

@sebastient

@sebastient sebastient commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Part 1 — the gate never skipped

Two separate things stopped it.

It only applied on schedule.

if [[ "${GITHUB_EVENT_NAME}" == schedule && "$last" == "$GITHUB_SHA" ]]; then

A workflow_dispatch bypassed the check silently, so re-dispatching a nightly rebuilt the same commit and re-ran the board — the slowest and only hardware-bound lane — for no new information.

It compared against the last successful run. This is the more serious one, because it makes the gate useless in exactly the repository that needs it. Against the live API:

filter head_sha date
status=success 7c1bbc1c 2025-12-08
status=completed 06bbe5f3 today — is current main

hal's last green nightly on main was nine months ago, so every comparison since has been against a December commit. A failed run still looked at the commit, so it answers the question the gate asks.

Now: applies on every trigger, compares against the last completed run of any conclusion.

force last completed HEAD result
false abc abc skip
false abc def run — main moved
false (none) def run — no prior run
true abc abc run — forced
true abc def run — forced

Verified by extracting the run: block verbatim from the YAML and executing all five rows under set -euo pipefail with gh stubbed.

Re-running an unchanged commit after fixing a lane is force's job — an explicit request rather than an accident of run history.

Part 2 — the advisory lane must not be gated

A working gate creates a hole. Every other nightly lane asks a question about the code, so skipping an unchanged commit loses nothing. cargo audit asks a question about the RustSec database, which changes daily whether the code does or not — so a gated-off nightly goes quiet exactly when a new advisory lands against a frozen main.

Not hypothetical. Today's nightly failed on RUSTSEC-2026-0204 against a Cargo.lock nobody had touched, which is why #193 exists.

New advisories.yml, called without needs: changed:

  advisories:
    uses: EdgeFirstAI/.github/.github/workflows/advisories.yml@<sha>

Running every night is only defensible if it costs nothing, so:

  • No setup-rust. cargo audit parses Cargo.lock and never compiles; a toolchain install and cargo cache would be the only slow parts of an otherwise instant job.
  • runs-on: ubuntu-24.04 hard-coded, not the caller's runner-class. Per copilot-instructions.md the hosted class is "free and unmetered on public repositories", while larger bills by the minute even there. A caller cannot put an every-night lane on a billed runner by accident. runner-audit's billed-label regex does not match it.
  • Advisories are downgraded via .cargo/audit.toml in the calling repo — a reviewable file where it applies, not a workflow input.

nightly-extra is now just the hack lane

With audit gone it does one thing, so it is renamed Feature combinations and skips entirely when hack-args is empty rather than booting a runner to install tools and run nothing. hal sets hack-args: "", so that job stops costing anything.

Behaviour change for callers: rust-full no longer runs cargo audit. Callers must add the advisories.yml job. hal is the only caller today and templates/nightly.yml is updated.

Caller forwarding

Copilot was right that the skeleton never forwarded force. Fixed, along with the ungated advisories job:

on:
  workflow_dispatch:
    inputs:
      force: { type: boolean, default: false }

jobs:
  changed:
    uses: .../nightly-gate.yml@<sha>
    with:
      force: ${{ inputs.force || false }}

The || false fallback is load-bearing: inputs is null on a schedule trigger, and the boolean input rejects the empty string that would otherwise result.

Part 3 — template self-pins (from review)

Copilot flagged that advisories.yml does not exist at the template's pinned eec0cb31…. It does not, and the same stale pin makes the new force: forwarding invalid too, since nightly-gate at that commit declares no such input.

The literal suggestion — pin to a commit containing the new workflow — is impossible before this merges, and would rot again on the next commit, because a template that pins its own repository is stale as soon as one lands.

Template self-pins are now an all-zero placeholder. A stale SHA is worse than an obviously invalid one because it resolves: a copied skeleton silently runs nine-month-old CI, which is how this hid. An unresolvable ref fails immediately. Third-party actions in templates/ keep real pins.

A new lint step enforces it — the existing SHA-pin check cannot, since a stale SHA is a valid SHA. Extracted verbatim and run both ways: passes on current templates, and with eec0cb31 restored fails naming the exact line the review flagged.

Follow-up

hal picks both up on its next re-pin, which is where its nightly.yml gains the force input and the advisories job.

The condition was `event == schedule && last == sha`, so a manual
dispatch bypassed the change check silently. Three nightlies ran on
hal today, two of them rebuilding a commit whose nightly had already
covered it, each re-running the board lane for no new information.

Drop the event-name test so the gate applies to every trigger, and add
an explicit `force` input for deliberately re-running a green nightly.

The comparison stays against the last SUCCESSFUL run, which is what
makes this safe: a nightly that failed leaves the last success on a
different commit, so an unchanged main still re-runs without `force`.
That is the case you want after fixing a lane rather than the code,
and it is how today's runs would still have been allowed.

Signed-off-by: Sébastien Taylor <[email protected]>
Copilot AI balanced review requested due to automatic review settings September 15, 2026 22:43

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The maintained caller template does not forward the new force input.

Pull request overview

Updates the reusable nightly gate to prevent redundant runs across all triggers while allowing forced reruns.

Changes:

  • Adds a boolean force override.
  • Applies successful-commit gating universally.
  • Documents the behavior in CHANGELOG.md.
File summaries
File Summary
CHANGELOG.md Documents the updated nightly gate behavior.
.github/workflows/nightly-gate.yml Implements universal gating and forced reruns.
Review details

Suppressed comments (1)

.github/workflows/nightly-gate.yml:6

  • The canonical templates/nightly.yml still declares a bare workflow_dispatch and never forwards a force value to this input. Repositories copied from the maintained caller skeleton therefore cannot use the documented manual override without an extra hand edit; please update that template alongside this reusable-workflow API (or explicitly document why the skeleton omits it).
      force:
  • Files reviewed: 2/2 changed files
  • Comments generated: 0
  • Review effort level: Lite (auto)

Note

Copilot is running an experiment and ran this review at Lite.


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

The gate asked whether main had moved since the last *successful* nightly.
In a repository whose nightly is not consistently green that compares
against an arbitrarily old commit: hal's last green nightly on main was
2025-12-08, so the gate had not skipped once in nine months. Against the
live API, status=success returns 7c1bbc1c from December while
status=completed returns 06bbe5f3, which is current main -- the first
skips nothing, the second skips correctly.

A failed run still looked at the commit, so it answers the question.
Re-running an unchanged commit after fixing a lane is what `force` is
for: an explicit request rather than an accident of run history.

Forward `force` from templates/nightly.yml, which declared a bare
workflow_dispatch and gave callers copied from the skeleton no way to use
the documented override. `inputs` is null on a schedule trigger, so the
`|| false` fallback is required or the boolean input rejects an empty
string.

Signed-off-by: Sébastien Taylor <[email protected]>
Every other nightly lane asks a question about the code, so skipping it
on an unchanged commit loses nothing. cargo audit asks a question about
the RustSec database, which changes daily whether the code does or not.
Gated, it goes quiet exactly when a new advisory lands against a frozen
main -- not hypothetical: the 2026-09-15 nightly failed on
RUSTSEC-2026-0204 against a Cargo.lock nobody had touched.

Move it to advisories.yml, called without `needs: changed`. It parses
Cargo.lock and never compiles, so it skips setup-rust entirely and is
seconds of a standard runner. The runner label is hard-coded rather than
honouring the caller's runner-class: an every-night lane must not be
able to land on a `larger` runner, which bills by the minute even on
public repositories where the hosted class is free and unmetered.

Advisories are downgraded through `.cargo/audit.toml` in the calling
repository, which cargo-audit reads from the working directory -- a
reviewable file in the repo it applies to, rather than a workflow input.

What is left of nightly-extra is the hack lane, so rename it to Feature
combinations and skip it when hack-args is empty instead of booting a
runner to install tools and run nothing. hal sets hack-args to empty, so
that job stops costing anything at all.

Signed-off-by: Sébastien Taylor <[email protected]>
@sebastient sebastient changed the title EDGEAI-1554: Gate the nightly on every trigger, not only schedule EDGEAI-1554: Make the nightly gate work, and exempt the advisory lane from it Sep 16, 2026
@sebastient
sebastient requested a balanced review from Copilot September 16, 2026 00:29
The SHA-pin lint checks the shape of a pin, not that the referenced file
exists at it. Every templates/ pin is a stale placeholder, which was
harmless while they all resolved; advisories.yml is new, so a pin
predating it now 404s at run time instead of failing at lint. Say so in
the header rather than leaving a skeleton that looks copyable.

Signed-off-by: Sébastien Taylor <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The template pins revisions that lack both the new advisory workflow and the forwarded force input.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

templates/nightly.yml:31

  • advisories.yml does not exist at the pinned eec0cb31… revision, so this reusable-workflow call cannot be resolved and the generated nightly workflow will be invalid. Pin the template to a commit that includes the new workflow.
  # the RustSec database, not about the code, so gating it would go quiet
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Balanced (auto)

Note

Copilot is running an experiment and ran this review at Balanced.

Comment thread templates/nightly.yml
Copilot flagged that templates/nightly.yml referenced advisories.yml at
a pin that predates it. The pin cannot be fixed as asked: no commit
containing advisories.yml exists until this merges. The defect is also
wider than that one line -- the same stale pin makes the new `force`
forwarding invalid, because the nightly-gate at eec0cb3 does not
declare that input.

Every template self-pin had drifted to a commit from nine months ago,
and a stale SHA is worse than an obviously invalid one because it still
resolves: a copied skeleton silently runs old CI. Pin them to all zeros
instead, so a skeleton that was not adjusted fails immediately and says
what to fix.

The existing SHA-pin check cannot catch this, since a stale SHA is a
valid SHA, so add a step asserting the placeholder. Verified by
extracting it verbatim from ci.yml and running it both ways: it passes
on the current templates, and on a copy with the eec0cb3 pin restored
it fails naming that exact line.

Third-party actions in templates/ keep real pins and are copyable as-is.

Signed-off-by: Sébastien Taylor <[email protected]>
@sebastient

Copy link
Copy Markdown
Contributor Author

Addressing the review

Finding: advisories.yml does not exist at the pinned eec0cb31…, so the skeleton's call cannot resolve. Correct, and worth more than a one-line fix.

I can't do what was literally suggested — pin the template to a commit containing the new workflow — because no such commit exists until this merges. So I looked at why the pin was wrong in the first place.

The defect is wider than the one line. eec0cb31 is nine months stale, and it's not only advisories.yml that breaks at it: the force: forwarding added in this PR is also invalid there, because nightly-gate at that commit doesn't declare a force input. Two unresolvable references, one root cause.

And the pins can never be right. A template that pins its own repository is stale the moment the next commit lands, so this recurs by construction.

Fix: template self-pins are now an all-zero placeholder.

uses: EdgeFirstAI/.github/.github/workflows/advisories.yml@0000000000000000000000000000000000000000

A stale SHA is worse than an obviously invalid one precisely because it resolves — a copied skeleton silently runs nine-month-old CI, which is the failure mode that hid here. An unresolvable ref fails immediately and says what to fix. Third-party actions in templates/ keep real pins and are copyable as-is.

Enforced, not just documented. The existing SHA-pin check can't catch this — a stale SHA is a valid SHA — so there's a new lint step asserting the placeholder. Verified by extracting it verbatim from ci.yml and running it both ways:

PASS (current templates):  all template self-pins use the placeholder          exit=0
FAIL (eec0cb31 restored):  templates/nightly.yml:40: uses: ...advisories.yml@eec0cb31…  exit=1

The failing case reproduces exactly the line this review flagged. Confirmed running in CI on 35040550733.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A failed or cancelled gate run can cause the next nightly to incorrectly skip all gated lanes.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Balanced (auto)

Note

Copilot is running an experiment and ran this review at Balanced.

Comment thread .github/workflows/nightly-gate.yml Outdated
# Re-running an unchanged commit after fixing a lane is what `force`
# is for -- an explicit request, not an accident of run history.
# completed excludes the current in-progress run.
last="$(gh api "repos/${GITHUB_REPOSITORY}/actions/workflows/${wf}/runs?status=completed&branch=main&per_page=1" --jq '.workflow_runs[0].head_sha // ""')"
A run can be cancelled before a single lane starts, yet it is still
"completed" at its commit. Taking the newest completed run meant such a
run would suppress the next night's nightly having determined nothing
about the code. Same for timed_out, stale, neutral, action_required and
startup_failure.

Count only success and failure: both mean the lanes ran and reported.
Anything else looks further back, which at worst re-runs a nightly --
the safe direction for a gate to fail.

Verified against the live API rather than a stub, using hal's ci.yml
history, which has 13 cancelled runs. On a window whose newest run is
cancelled, the old expression returns that run's sha while the new one
returns the next verdict-bearing run; on a window holding only a
cancelled run it returns empty, so the gate runs.

Signed-off-by: Sébastien Taylor <[email protected]>
@sebastient

Copy link
Copy Markdown
Contributor Author

Addressing the second review

Finding: a failed or cancelled gate run can make the next nightly skip the gated lanes.

The cancelled half is a real hole and is now fixed. A run can be cancelled before a single lane starts — or the gate job itself can fail — yet the run is still completed at that SHA. Taking the newest completed run meant such a run would suppress the next night having determined nothing about the code.

The gate now counts only runs that reached a verdict:

success | failure   -> counts; the lanes ran and reported
everything else     -> discarded; look further back

Discarded: cancelled, timed_out, stale, neutral, action_required, startup_failure. Looking further back at worst re-runs a nightly, which is the safe direction for a gate to fail.

Verified against the live API, not a stub. hal's ci.yml has 13 genuinely cancelled runs, so I paged to a window whose newest run is cancelled:

window:  cancelled 91a42561 / success b295de43 / success 93d97157
new:     b295de43   <- skips the cancelled run
old:     91a42561   <- would have suppressed the next nightly

And a window holding only a cancelled run returns empty, so the gate runs.


The failure half I've kept deliberately, and it's worth being explicit since it's the opposite call.

A failed run did look at its commit: the lanes ran and reported. Treating failure as "no verdict" is what the first version of this PR effectively did by filtering on status=success, and it makes the gate inert — hal's last green nightly on main was 2025-12-08, so every comparison would be against a December commit and the gate would never skip. That is the bug this PR exists to fix.

So a red nightly on an unchanged main does not automatically re-run. That is intentional: re-running the same commit after fixing a lane rather than the code is what force is for — an explicit request rather than an accident of run history. Pushing the fix to main moves the SHA and re-runs it anyway, which is the common case.

Decision table re-verified after the change; all five rows unchanged.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Failed gate runs can incorrectly suppress later nightly execution, and the hosted advisory exception conflicts with canonical runner policy.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

.github/workflows/advisories.yml:34

  • This deliberate hosted, ungated nightly job conflicts with the repository's canonical policy: .github/copilot-instructions.md:31-35 says Nightly runs only when main moves, and lines 54-69 require the larger class for Nightly. Add an explicit advisory-scanning exception to that source-of-truth document so contributors and audits do not apply the existing policy and undo this behavior.
  • Files reviewed: 10/10 changed files
  • Comments generated: 1
  • Review effort level: Balanced

# back instead, which at worst re-runs a nightly.
#
# status=completed also excludes this run, which is in_progress.
last="$(gh api "repos/${GITHUB_REPOSITORY}/actions/workflows/${wf}/runs?status=completed&branch=main&per_page=30" --jq '[.workflow_runs[] | select(.conclusion == "success" or .conclusion == "failure")][0].head_sha // ""')"
copilot-instructions.md is the source of truth for the tiers and the
runner classes, and advisories.yml breaks two of its rules: Nightly runs
only when main moved, and Nightly uses the larger class. Both are broken
deliberately, but an undocumented exception is one a future contributor
or an audit will helpfully undo.

State it, with the reasoning and the limits: not gated because the
question is about the RustSec database rather than the code; hosted and
hard-coded because "cost accepted" relies on the gate bounding Nightly
to days when main moved, and an ungated lane has no such bound. Say
plainly that a lane which builds anything does not qualify, so this does
not become a general escape hatch.

Signed-off-by: Sébastien Taylor <[email protected]>
@sebastient

Copy link
Copy Markdown
Contributor Author

Addressing the policy conflict

Finding: advisories.yml contradicts copilot-instructions.md — Nightly runs only when main moves, and Nightly uses the larger class. Both true, and the right call: an undocumented exception is one a future contributor or an audit will helpfully undo.

copilot-instructions.md now records it in both tables plus a section stating the reasoning and the limits, so it doesn't become a general escape hatch:

  • Not gated — the question is about the RustSec database, not the code.
  • hosted, hard-coded — "cost accepted" for Nightly relies on the gate bounding it to days when main moved. An ungated lane has no such bound, so it only stays affordable on a free class, and hard-coding stops a caller putting an every-night lane on a billed runner via runner-class-linux: larger.
  • Explicit limit — the exception holds only because cargo audit never compiles. A lane that builds anything does not qualify: gate it, or leave it out of the nightly.
  • Noted that runner-audit doesn't flag it, since ubuntu-24.04 isn't a billed label, so no # runner-class: larger marker is needed.

Round summary

Review Finding Outcome
1 Template doesn't forward force Fixed
2 Template pin lacks advisories.yml Fixed at the class level — placeholder pins + enforcing lint
3 Cancelled run suppresses next nightly Fixed — verdict-bearing runs only
3 Failed run suppresses next nightly Kept deliberately, reasoning above
4 Conflicts with canonical runner policy Fixed — exception documented with limits

All checks green.

@sebastient
sebastient merged commit 797eea7 into main Sep 16, 2026
4 checks passed
@sebastient
sebastient deleted the feature/EDGEAI-1554-gate-all-triggers branch September 16, 2026 00:54
sebastient added a commit to EdgeFirstAI/hal that referenced this pull request Sep 16, 2026
Picks up EdgeFirstAI/.github#26. Two caller-visible changes come with it.

The nightly gate now applies on every trigger and compares against the
last run that reached a verdict, so a workflow_dispatch no longer
bypasses it silently. Forward a `force` input for deliberately re-running
an unchanged commit; `inputs` is null on the schedule trigger, so the
`|| false` fallback is required or the gate's boolean input rejects an
empty string.

cargo audit moved out of rust-full into advisories.yml, so it has to be
called explicitly, and it is called without `needs: changed`. Every other
lane here asks a question about the code and loses nothing by skipping an
unchanged commit; cargo audit asks a question about the RustSec database,
which changes daily whether the code does or not. The 2026-09-15 nightly
failed on RUSTSEC-2026-0204 against a Cargo.lock nobody had touched,
which a gated lane would have missed.

Drop the shared SHA from the workflows README. It was a second copy of
the pin that Dependabot does not update, and it was stale exactly one
re-pin after the paragraph claiming the SHA appears only in `uses:`.

Signed-off-by: Sébastien Taylor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants