cd /home/david/projects/stirlingbridge/machine gh run list --workflow e2e-test.yml --branch dboreham/improve-auth-fail-behavior --limit 3 --json databaseId,status,conclusion,createdAt,event 2>&1
[{"conclusion":"failure","createdAt":"2026-07-07T14:52:08Z","databaseId":28875670291,"event":"pull_request","status":"completed"}]cd /home/david/projects/stirlingbridge/machine
gh run view 28875670291 --json status,conclusion,jobs --jq '{status,conclusion,jobs:[.jobs[]|{name,status,conclusion,startedAt}]}' 2>&1{"conclusion":"","jobs":[{"conclusion":"","name":"E2E Tests (digital-ocean)","startedAt":"2026-07-07T15:18:26Z","status":"in_progress"},{"conclusion":"success","name":"E2E Tests (gcp)","startedAt":"2026-07-07T14:52:11Z","status":"completed"}],"status":"in_progress"}The re-run picked up — the DigitalOcean job is now in_progress (started 15:18 UTC), and the GCP job already passed. Let me wait for it to finish and report back.
cd /home/david/projects/stirlingbridge/machine
gh run watch 28875670291 --exit-status --interval 30 >/dev/null 2>&1
echo "watch exited: $?"
gh run view 28875670291 --json status,conclusion,jobs --jq '{status,conclusion,jobs:[.jobs[]|{name,conclusion}]}' 2>&1watch exited: 0
{"conclusion":"success","jobs":[{"conclusion":"success","name":"E2E Tests (digital-ocean)"},{"conclusion":"success","name":"E2E Tests (gcp)"},{"conclusion":"success","name":"E2E Tests (vultr)"}],"status":"completed"}The e2e run passed — all three jobs green:
So the failure was indeed just the expired CI token, unrelated to this PR. The whole run is now completed / success. PR #96 should be showing all checks green.
Worth remembering that the weekly scheduled run will keep passing until this token expires too — if you want to avoid the silent-failure surprise next time, the failure-notification step I mentioned would catch it. Let me know if you'd like me to add that.
cd /home/david/projects/stirlingbridge/machine gh issue create --title "Scheduled e2e workflow failures are silent (no notification)" --body "$(cat <<'EOF' ## Problem The `e2e-test.yml` workflow runs on a weekly schedule (`cron: "0 6 * * 1"`, Mondays 06:00 UTC) in addition to PR/merge triggers. When the scheduled run fails, no failure notification is delivered. This was discovered while working on #95: the DigitalOcean e2e job had been failing for some time (an expired `E2E_DO_TOKEN`), but no failure emails were received, so the breakage went unnoticed until it surfaced on an unrelated PR's checks. ## Why it happens GitHub only emails scheduled-workflow failures to the account that *last modified the workflow file*, and it stops sending after repeated failures on the same schedule. So the missing emails are expected GitHub behavior, not a separate bug — but it means a broken weekly run can silently rot. ## Suggested fix Add an explicit failure-notification step to the scheduled path of `e2e-test.yml` so breakage is surfaced reliably. Options: - A notification step gated on `if: failure() && github.event_name == 'schedule'` (Slack webhook, email, or `gh issue create` to auto-open a tracking issue). - Alternatively/additionally, a lightweight monitor that alerts if the weekly run hasn't succeeded recently. ## Related - Discovered during #95 (auth-failure handling PR #96). - The immediate cause (expired `E2E_DO_TOKEN`) has been resolved by rotating the token; this issue tracks the missing-notification gap so the next expiry/failure doesn't go unnoticed. ## Notes Credentials rotate/expire periodically (DO token, Vultr API key, GCP creds), so this will recur without proactive alerting. EOF )" 2>&1
Filed: https://github.com/stirlingbridge/machine/issues/97 — "Scheduled e2e workflow failures are silent (no notification)".
It captures the problem (silent weekly-run failures), why it happens (GitHub only emails the last workflow-file editor and stops after repeated failures), suggested fixes (a if: failure() && github.event_name == 'schedule' notification step or an auto-opened tracking issue), and links back to #95 / PR #96 with a note that the immediate expired-token cause is already resolved.
You're welcome — that was a good session. Quick recap of where things landed:
Anytime you need a hand — cheers! 👋