11 prompts · 137 messages · 38 tool calls · 0 commits · 3 pages
Please take a look at issue #95
Vultr's per-method handling already catches its exceptions (that 502 is a transient Vultr maintenance error, handled by the existing fatal_error). The DigitalOcean read paths were the real gap, and the top-level wrapper is a safety net for those and GCP.
Now let me add hermetic tests for the new behavior.
Done. Here's what I did for issue #95.
An invalid/expired provider API token caused an uncaught digitalocean.DataReadError to bubble up as a raw Python traceback. The DigitalOcean provider only wrapped a few calls (create_vm, destroy_vm), so read paths like list_vms → get_all_droplets crashed ungracefully.
Added a top-level entry-point wrapper cli() in machine/main.py that runs the Click group and catches the provider SDKs' base API exceptions, converting them into a clear message instead of a traceback:
digitalocean.Error, vultr.VultrException, and (lazily) the Google Cloud auth/API exception bases — so it's a safety net across all subcommands and providers, not just list.--debug re-raises the original exception so the full traceback is still available for troubleshooting.Updated [project.scripts] in pyproject.toml to point machine at machine.main:cli (the shiv build uses this console script automatically).
Now:
$ machine --config-file … list
Error: the cloud provider rejected the request as unauthenticated or unauthorized.
Check that the API token/key in your config file is correct and has not expired.
--debug still shows the traceback.version, types), Click usage errors, and help are unaffected.tests/test_error_handling.py (message formatting, wrapper catches provider errors, --debug re-raises, non-provider exceptions pass through).I haven't committed — let me know if you'd like me to.
Ok thanks. Looking at the CI test results on the PR I noticed that the Digitalocean e2e test failed. The log shows that it failed with API auth errors so I initially suspected something related to the current code changes. However manual testing (using uv run on this machine) showed that a valid DO token worked. So then I checked the run history for that job (in addition to PR and main branch merge triggers, there is a scheduled weekly run) which showed that it had been failing for some time(!) somehow GitHub is not sending me the run failure notification emails. Anyway, this likely implies that the API token used by the CI environment has expired. I will issue a new one and re-run the failing job.
That diagnosis makes sense — the e2e job runs against real DO resources, so an expired CI token would fail with exactly those auth errors regardless of this PR. And notably, my change only wraps the CLI entry point; it doesn't touch the create/destroy API paths the e2e tests exercise, so it wouldn't cause (or fix) that failure.
Let me point you to exactly where the CI token is configured so the rotation is quick:
Here's what you'll need:
E2E_DO_TOKEN (referenced at .github/workflows/e2e-test.yml:45). The DNS zone is a repo variable (vars.E2E_DO_DNS_ZONE), so only the token needs updating.cron: "0 6 * * 1" — weekly, Mondays 06:00 UTC (line 11), which matches the recurring failures you found.Once you've issued the new DO token, you can rotate the secret and re-run without leaving the terminal:
# Set the new token (prompts for the value, doesn't echo it to history)
gh secret set E2E_DO_TOKEN --repo stirlingbridge/machine
# Re-run just the failed e2e job on this PR's latest run
gh run list --workflow e2e-test.yml --branch dboreham/improve-auth-fail-behavior --limit 1
gh run rerun <run-id> --failed
One heads-up on notifications: GitHub only emails scheduled-workflow failures to the user whose account last modified the workflow file, and it silently stops after ~repeated failures on a schedule. So the missing emails are expected behavior rather than a second bug — worth adding an explicit failure notification (Slack/email step) to the workflow if you want reliable alerts on the weekly run. Happy to draft that if useful.
Want me to do anything here, or are you set to handle the token rotation and re-run yourself?
I've rotated the token and re-started the job.
The e2e run passed — all three jobs green:
So the failure was indeed just the expired CI token, unrelated to this PR. The whole run is now completed / success. PR #96 should be showing all checks green.
Worth remembering that the weekly scheduled run will keep passing until this token expires too — if you want to avoid the silent-failure surprise next time, the failure-notification step I mentioned would catch it. Let me know if you'd like me to add that.
Ok, perhaps you could file a bug for that so we can address it in the future?
Filed: https://github.com/stirlingbridge/machine/issues/97 — "Scheduled e2e workflow failures are silent (no notification)".
It captures the problem (silent weekly-run failures), why it happens (GitHub only emails the last workflow-file editor and stops after repeated failures), suggested fixes (a if: failure() && github.event_name == 'schedule' notification step or an auto-opened tracking issue), and links back to #95 / PR #96 with a note that the immediate expired-token cause is already resolved.
Thanks! Nice doing business with you.
You're welcome — that was a good session. Quick recap of where things landed:
Anytime you need a hand — cheers! 👋