I tried running the recently added e2e test suite but it fails with an exception parsing the config file. I wondered if you could see if you can diagnose the error. This session already has the required three environment variables set.
{
"subagent_type": "Explore",
"prompt": "Find all e2e test files and the config.py file in this project. Read the e2e test files and machine/config.py to understand how config is loaded and what the tests expect. Also read machine/di.py and any fixture files. Report back the full contents of the e2e test files, config.py, and di.py."
}Perfect! I have found and read all the requested files. Here's the complete content:
## E2E Test File
**File**: `/home/david/projects/stirlingbridge/machine/tests/test_e2e.py`
This is the complete e2e test file. It contains:
1. **Module documentation** - explains that these tests require real DigitalOcean credentials and must be run explicitly with environment variables:
- E2E_DO_TOKEN
- E2E_SSH_KEY
- E2E_DNS_ZONE
- Optional: E2E_REGION, E2E_IMAGE, E2E_SIZE, E2E_PROJECT
2. **Helper functions**:
- `_unique_name()` - generates unique droplet names
- `_write_config()` - writes a temporary config file for testing with the e2e environment variables
- `run_machine()` - runs the machine CLI as a subprocess
3. **Fixtures**:
- `config_file` (module scope) - creates a temporary config.yml with e2e credentials
- `session_id` (module scope) - generates a unique session ID
- `droplet_cleanup` (function scope) - tracks created droplet IDs and destroys them after tests
4. **Test classes**:
- `TestDropletLifecycle` - tests create → list → destroy workflow
- `TestDNSLifecycle` - tests create with DNS → verify DNS → destroy with DNS cleanup
- `TestCreateWithInitialize` - tests creating droplets with cloud-init (using machine types)
- `TestCustomTag` - tests custom tag application
## Config File
**File**: `/home/david/projects/stirlingbridge/machine/machine/config.py`
The config module handles YAML configuration file parsing with:
1. **Environment variable expansion** (`_expand_env_vars()`) - supports `${VAR_NAME}` and `${VAR_NAME:-default}` syntax
2. **Config loading** (`_load_config_data()`) - loads YAML from the config file path, expands env vars, and caches it
3. **Config retrieval** (`get()`) - returns a `Config` object with these required fields from the `digital-ocean` section:
- access-token
- ssh-key
- machine-size
- image
- region
- project (required)
- dns-zone (optional)
4. **Machine config retrieval** (`get_machine()`, `get_machines()`) - loads machine type definitions from the `machines` section
- Each machine requires a `new-user-name`
- Optional fields: script-url, script-dir, script-path, script-args
## Dependency Injection Module
**File**: `/home/david/projects/stirlingbridge/machine/machine/di.py`
This is a minimal DI module that exposes global state:
```python
class d:
opt: CliOptions = None
```
It stores a single `CliOptions` object that contains CLI command-line options.
## Key Integration Points
The e2e tests expect:
- Config file at a custom path (`--config-file` argument)
- Session ID support (`--session-id` argument)
- YAML config format with `digital-ocean` and `machines` sections
- Droplet creation to output the droplet ID
- JSON output format for `list` and `list-domain` commandsagentId: a633a85d0c952048d (for resuming to continue this agent's work if needed) <usage>total_tokens: 22185 tool_uses: 10 duration_ms: 16291</usage>
uv run pytest tests/test_e2e.py -x --tb=short 2>&1 | head -80
============================= test session starts ============================== platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 rootdir: /home/david/projects/stirlingbridge/machine configfile: pyproject.toml collected 4 items / 4 deselected / 0 selected ============================ 4 deselected in 0.01s =============================
The tests are being deselected - likely due to missing environment variable markers. Let me check how to enable them.
1→"""End-to-end tests that create real VMs on DigitalOcean.
2→
3→These tests require a real DigitalOcean environment and are NOT run as part of
4→the normal test suite. They must be invoked explicitly:
5→
6→ uv run pytest tests/test_e2e.py -v
7→
8→Required environment variables:
9→ E2E_DO_TOKEN - DigitalOcean API token
10→ E2E_SSH_KEY - Name of an SSH key already registered in DO
11→ E2E_DNS_ZONE - DNS zone managed by DO (e.g. "test.example.com")
12→
13→Optional environment variables:
14→ E2E_REGION - Region slug (default: nyc1)
15→ E2E_IMAGE - Image slug (default: ubuntu-24-04-x64)
16→ E2E_SIZE - Machine size slug (default: s-1vcpu-512mb-10gb)
17→ E2E_PROJECT - DO project name to assign droplets to
18→"""
19→
20→import json
21→import os
22→import subprocess
23→import textwrap
24→import uuid
25→
26→import pytest
27→
28→
29→# ---------------------------------------------------------------------------
30→# Skip the entire module if credentials are not provided
31→# ---------------------------------------------------------------------------
32→
33→E2E_DO_TOKEN = os.environ.get("E2E_DO_TOKEN")
34→E2E_SSH_KEY = os.environ.get("E2E_SSH_KEY")
35→E2E_DNS_ZONE = os.environ.get("E2E_DNS_ZONE")
36→E2E_REGION = os.environ.get("E2E_REGION", "nyc1")
37→E2E_IMAGE = os.environ.get("E2E_IMAGE", "ubuntu-24-04-x64")
38→E2E_SIZE = os.environ.get("E2E_SIZE", "s-1vcpu-512mb-10gb")
39→E2E_PROJECT = os.environ.get("E2E_PROJECT", "")
40→
41→pytestmark = pytest.mark.e2e
42→
43→_MISSING = []
44→if not E2E_DO_TOKEN:
45→ _MISSING.append("E2E_DO_TOKEN")
46→if not E2E_SSH_KEY:
47→ _MISSING.append("E2E_SSH_KEY")
48→if not E2E_DNS_ZONE:
49→ _MISSING.append("E2E_DNS_ZONE")
50→
51→if _MISSING:
52→ pytestmark = [
53→ pytest.mark.e2e,
54→ pytest.mark.skip(reason=f"E2E env vars not set: {', '.join(_MISSING)}"),
55→ ]
56→
57→
58→# ---------------------------------------------------------------------------
59→# Helpers
60→# ---------------------------------------------------------------------------
61→
62→
63→def _unique_name(prefix="e2etest"):
64→ """Generate a short unique droplet name safe for DNS."""
65→ return f"{prefix}-{uuid.uuid4().hex[:8]}"
66→
67→
68→def _write_config(path, **overrides):
69→ """Write a minimal config file for the e2e test run."""
70→ cfg = {
71→ "access-token": E2E_DO_TOKEN,
72→ "ssh-key": E2E_SSH_KEY,
73→ "dns-zone": E2E_DNS_ZONE,
74→ "machine-size": E2E_SIZE,
75→ "image": E2E_IMAGE,
76→ "region": E2E_REGION,
77→ }
78→ if E2E_PROJECT:
79→ cfg["project"] = E2E_PROJECT
80→ cfg.update(overrides)
81→
82→ do_lines = "\n".join(f" {k}: {v}" for k, v in cfg.items())
83→ content = textwrap.dedent(
84→ f"""\
85→ digital-ocean:
86→ {do_lines}
87→ machines:
88→ e2e-basic:
89→ new-user-name: e2euser
90→ """
91→ )
92→ with open(path, "w") as f:
93→ f.write(content)
94→
95→
96→def run_machine(*args, config_file=None, session_id=None):
97→ """Run the machine CLI as a subprocess with the given arguments."""
98→ cmd = ["uv", "run", "machine"]
99→ if config_file:
100→ cmd += ["--config-file", str(config_file)]
101→ if session_id:
102→ cmd += ["--session-id", session_id]
103→ cmd += list(args)
104→ result = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
105→ return result
106→
107→
108→# ---------------------------------------------------------------------------
109→# Fixtures
110→# ---------------------------------------------------------------------------
111→
112→
113→@pytest.fixture(scope="module")
114→def config_file(tmp_path_factory):
115→ """Write a config file that lives for the whole test module."""
116→ path = tmp_path_factory.mktemp("e2e") / "config.yml"
117→ _write_config(path)
118→ return path
119→
120→
121→@pytest.fixture(scope="module")
122→def session_id():
123→ """A unique session id shared across all tests in this module."""
124→ return uuid.uuid4().hex[:8]
125→
126→
127→@pytest.fixture()
128→def droplet_cleanup(config_file, session_id):
129→ """Fixture that tracks created droplet IDs and destroys them after the test."""
130→ created_ids = []
131→ yield created_ids
132→ for did in created_ids:
133→ run_machine(
134→ "--verbose",
135→ "destroy",
136→ "--no-confirm",
137→ str(did),
138→ config_file=config_file,
139→ session_id=session_id,
140→ )
141→
142→
143→# ---------------------------------------------------------------------------
144→# Tests
145→# ---------------------------------------------------------------------------
146→
147→
148→class TestDropletLifecycle:
149→ """Create a droplet, verify it, then destroy it."""
150→
151→ def test_create_and_list_and_destroy(self, config_file, session_id, droplet_cleanup):
152→ name = _unique_name()
153→
154→ # ---- CREATE --------------------------------------------------------
155→ result = run_machine(
156→ "create",
157→ "--name",
158→ name,
159→ "--no-initialize",
160→ "--wait-for-ip",
161→ config_file=config_file,
162→ session_id=session_id,
163→ )
164→ assert result.returncode == 0, f"create failed: {result.stderr}"
165→ # Extract the droplet id from output like "New droplet created with id: 12345"
166→ create_out = result.stdout + result.stderr
167→ droplet_id = None
168→ for line in create_out.splitlines():
169→ if "id:" in line.lower():
170→ # grab the number after "id:"
171→ parts = line.split("id:")
172→ if len(parts) >= 2:
173→ candidate = parts[-1].strip()
174→ if candidate.isdigit():
175→ droplet_id = candidate
176→ break
177→ assert droplet_id, f"Could not find droplet id in output:\n{create_out}"
178→ droplet_cleanup.append(droplet_id)
179→
180→ # ---- LIST ----------------------------------------------------------
181→ result = run_machine(
182→ "list",
183→ "--output",
184→ "json",
185→ config_file=config_file,
186→ session_id=session_id,
187→ )
188→ assert result.returncode == 0, f"list failed: {result.stderr}"
189→ droplets = json.loads(result.stdout)
190→ matched = [d for d in droplets if str(d["id"]) == droplet_id]
191→ assert len(matched) == 1, f"Expected 1 droplet with id {droplet_id}, got {len(matched)}"
192→ assert matched[0]["name"] == name
193→ assert matched[0]["ip"] is not None
194→
195→ # ---- DESTROY -------------------------------------------------------
196→ result = run_machine(
197→ "destroy",
198→ "--no-confirm",
199→ droplet_id,
200→ config_file=config_file,
201→ session_id=session_id,
202→ )
203→ assert result.returncode == 0, f"destroy failed: {result.stderr}"
204→ # Remove from cleanup list since we already destroyed it
205→ droplet_cleanup.remove(droplet_id)
206→
207→ # ---- VERIFY GONE ---------------------------------------------------
208→ result = run_machine(
209→ "list",
210→ "--output",
211→ "json",
212→ config_file=config_file,
213→ session_id=session_id,
214→ )
215→ assert result.returncode == 0
216→ droplets = json.loads(result.stdout)
217→ matched = [d for d in droplets if str(d["id"]) == droplet_id]
218→ assert len(matched) == 0, "Droplet still exists after destroy"
219→
220→
221→class TestDNSLifecycle:
222→ """Create a droplet with DNS, verify the record, then destroy and verify cleanup."""
223→
224→ def test_create_with_dns_and_destroy(self, config_file, session_id, droplet_cleanup):
225→ name = _unique_name()
226→
227→ # ---- CREATE with DNS -----------------------------------------------
228→ result = run_machine(
229→ "create",
230→ "--name",
231→ name,
232→ "--no-initialize",
233→ "--update-dns",
234→ config_file=config_file,
235→ session_id=session_id,
236→ )
237→ assert result.returncode == 0, f"create failed: {result.stderr}"
238→ create_out = result.stdout + result.stderr
239→ droplet_id = None
240→ for line in create_out.splitlines():
241→ if "id:" in line.lower():
242→ parts = line.split("id:")
243→ if len(parts) >= 2:
244→ candidate = parts[-1].strip()
245→ if candidate.isdigit():
246→ droplet_id = candidate
247→ break
248→ assert droplet_id, f"Could not find droplet id in output:\n{create_out}"
249→ droplet_cleanup.append(droplet_id)
250→
251→ # Verify DNS was mentioned in output
252→ assert E2E_DNS_ZONE in create_out, f"DNS zone not mentioned in output:\n{create_out}"
253→
254→ # ---- LIST DOMAIN ---------------------------------------------------
255→ result = run_machine(
256→ "list-domain",
257→ "--name",
258→ name,
259→ "--output",
260→ "json",
261→ E2E_DNS_ZONE,
262→ config_file=config_file,
263→ session_id=session_id,
264→ )
265→ assert result.returncode == 0, f"list-domain failed: {result.stderr}"
266→ records = json.loads(result.stdout)
267→ a_records = [r for r in records if r.get("name") == name and r.get("type") == "A"]
268→ assert len(a_records) >= 1, f"No A record found for {name}.{E2E_DNS_ZONE}"
269→
270→ # ---- DESTROY with DNS cleanup --------------------------------------
271→ result = run_machine(
272→ "destroy",
273→ "--no-confirm",
274→ "--delete-dns",
275→ droplet_id,
276→ config_file=config_file,
277→ session_id=session_id,
278→ )
279→ assert result.returncode == 0, f"destroy failed: {result.stderr}"
280→ droplet_cleanup.remove(droplet_id)
281→
282→ # ---- VERIFY DNS RECORD REMOVED -------------------------------------
283→ result = run_machine(
284→ "list-domain",
285→ "--name",
286→ name,
287→ "--all",
288→ "--output",
289→ "json",
290→ E2E_DNS_ZONE,
291→ config_file=config_file,
292→ session_id=session_id,
293→ )
294→ assert result.returncode == 0
295→ records = json.loads(result.stdout)
296→ a_records = [r for r in records if r.get("name") == name and r.get("type") == "A"]
297→ assert len(a_records) == 0, f"DNS A record still exists for {name}.{E2E_DNS_ZONE}"
298→
299→
300→class TestCreateWithInitialize:
301→ """Create a droplet with cloud-init and verify it was initialized."""
302→
303→ def test_create_with_type(self, config_file, session_id, droplet_cleanup):
304→ name = _unique_name()
305→
306→ # ---- CREATE with initialization ------------------------------------
307→ result = run_machine(
308→ "create",
309→ "--name",
310→ name,
311→ "--type",
312→ "e2e-basic",
313→ "--wait-for-ip",
314→ config_file=config_file,
315→ session_id=session_id,
316→ )
317→ assert result.returncode == 0, f"create failed: {result.stderr}"
318→ create_out = result.stdout + result.stderr
319→ droplet_id = None
320→ for line in create_out.splitlines():
321→ if "id:" in line.lower():
322→ parts = line.split("id:")
323→ if len(parts) >= 2:
324→ candidate = parts[-1].strip()
325→ if candidate.isdigit():
326→ droplet_id = candidate
327→ break
328→ assert droplet_id, f"Could not find droplet id in output:\n{create_out}"
329→ droplet_cleanup.append(droplet_id)
330→
331→ # ---- VERIFY TYPE TAG -----------------------------------------------
332→ result = run_machine(
333→ "list",
334→ "--type",
335→ "e2e-basic",
336→ "--output",
337→ "json",
338→ config_file=config_file,
339→ session_id=session_id,
340→ )
341→ assert result.returncode == 0
342→ droplets = json.loads(result.stdout)
343→ matched = [d for d in droplets if str(d["id"]) == droplet_id]
344→ assert len(matched) == 1
345→ assert matched[0]["type"] == "e2e-basic"
346→
347→ # ---- CLEANUP -------------------------------------------------------
348→ result = run_machine(
349→ "destroy",
350→ "--no-confirm",
351→ droplet_id,
352→ config_file=config_file,
353→ session_id=session_id,
354→ )
355→ assert result.returncode == 0, f"destroy failed: {result.stderr}"
356→ droplet_cleanup.remove(droplet_id)
357→
358→
359→class TestCustomTag:
360→ """Verify that custom tags are applied to created droplets."""
361→
362→ def test_custom_tag(self, config_file, session_id, droplet_cleanup):
363→ name = _unique_name()
364→ custom_tag = f"e2e-tag-{uuid.uuid4().hex[:6]}"
365→
366→ result = run_machine(
367→ "create",
368→ "--name",
369→ name,
370→ "--no-initialize",
371→ "--tag",
372→ custom_tag,
373→ "--wait-for-ip",
374→ config_file=config_file,
375→ session_id=session_id,
376→ )
377→ assert result.returncode == 0, f"create failed: {result.stderr}"
378→ create_out = result.stdout + result.stderr
379→ droplet_id = None
380→ for line in create_out.splitlines():
381→ if "id:" in line.lower():
382→ parts = line.split("id:")
383→ if len(parts) >= 2:
384→ candidate = parts[-1].strip()
385→ if candidate.isdigit():
386→ droplet_id = candidate
387→ break
388→ assert droplet_id
389→ droplet_cleanup.append(droplet_id)
390→
391→ # Verify tag via list --tag filter
392→ result = run_machine(
393→ "list",
394→ "--tag",
395→ custom_tag,
396→ "--output",
397→ "json",
398→ config_file=config_file,
399→ session_id=session_id,
400→ )
401→ assert result.returncode == 0
402→ droplets = json.loads(result.stdout)
403→ matched = [d for d in droplets if str(d["id"]) == droplet_id]
404→ assert len(matched) == 1, f"Droplet not found with tag {custom_tag}"
405→
406→ # Cleanup
407→ result = run_machine(
408→ "destroy",
409→ "--no-confirm",
410→ droplet_id,
411→ config_file=config_file,
412→ session_id=session_id,
413→ )
414→ assert result.returncode == 0
415→ droplet_cleanup.remove(droplet_id)
416→{
"pattern": "e2e|markers|filterwarnings|addopts",
"path": "/home/david/projects/stirlingbridge/machine/pyproject.toml",
"output_mode": "content"
}34:markers = [ 35: "e2e: end-to-end tests that create real resources on DigitalOcean (require E2E_* env vars)", 37:addopts = "-m 'not e2e'"
uv run pytest tests/test_e2e.py -m e2e -x --tb=short 2>&1 | head -60
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/david/projects/stirlingbridge/machine
configfile: pyproject.toml
collected 4 items
tests/test_e2e.py F
=================================== FAILURES ===================================
____________ TestDropletLifecycle.test_create_and_list_and_destroy _____________
tests/test_e2e.py:164: in test_create_and_list_and_destroy
assert result.returncode == 0, f"create failed: {result.stderr}"
E AssertionError: create failed: Traceback (most recent call last):
E File "/home/david/projects/stirlingbridge/machine/.venv/bin/machine", line 10, in <module>
E sys.exit(main())
E ^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
E return self.main(*args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
E rv = self.invoke(ctx)
E ^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1685, in invoke
E super().invoke(ctx)
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
E return ctx.invoke(self.callback, **ctx.params)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
E return __callback(*args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
E return f(get_current_context(), *args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/machine/main.py", line 33, in main
E main_context = MainCmdCtx(config.get(config_file), session_id)
E ^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/machine/config.py", line 59, in get
E config = _load_config_data(config_file_name)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/machine/config.py", line 44, in _load_config_data
E config = _expand_env_vars(yaml().load(open(config_path, "r")))
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/ruamel/yaml/main.py", line 426, in load
E return constructor.get_single_data()
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/ruamel/yaml/constructor.py", line 111, in get_single_data
E node = self.composer.get_single_node()
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/ruamel/yaml/composer.py", line 73, in get_single_node
E if not self.parser.check_event(StreamEndEvent):
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/ruamel/yaml/parser.py", line 139, in check_event
E self.current_event = self.state()
E ^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/ruamel/yaml/parser.py", line 209, in parse_document_start
E raise ParserError(
E ruamel.yaml.parser.ParserError: expected '<document start>', but found ('<block mapping start>',)
E in "/tmp/pytest-of-david/pytest-2/e2e0/config.yml", line 3, column 1
E
E assert 1 == 0YAML parsing error on line 3 of the generated config. The issue is in _write_config - the indentation is wrong. Let me look at what the config file actually looks like:
python3 -c "
import textwrap
cfg = {'access-token': 'tok', 'ssh-key': 'key', 'dns-zone': 'zone', 'machine-size': 's', 'image': 'img', 'region': 'nyc1'}
do_lines = '\n'.join(f' {k}: {v}' for k, v in cfg.items())
content = textwrap.dedent(
f'''\
digital-ocean:
{do_lines}
machines:
e2e-basic:
new-user-name: e2euser
'''
)
print(repr(content))
print('---')
print(content)
"' digital-ocean:\n access-token: tok\nssh-key: key\ndns-zone: zone\nmachine-size: s\nimage: img\nregion: nyc1\n machines:\n e2e-basic:\n new-user-name: e2euser\n'
---
digital-ocean:
access-token: tok
ssh-key: key
dns-zone: zone
machine-size: s
image: img
region: nyc1
machines:
e2e-basic:
new-user-name: e2euserFound it. The textwrap.dedent combined with the f-string interpolation of do_lines breaks the indentation. The first line of do_lines gets the f-string's indentation (4 spaces from dedent context), but subsequent lines have only their own 2-space indent, losing the nesting under digital-ocean:.
The fix is to not use textwrap.dedent with multi-line f-string interpolation, or to properly indent do_lines:
do_lines = "\n".join(f" {k}: {v}" for k, v in cfg.items())
content = textwrap.dedent(
f"""\
digital-ocean:
{do_lines}
machines:
e2e-basic:
new-user-name: e2euser
"""
) do_lines = "\n".join(f" {k}: {v}" for k, v in cfg.items())
content = f"digital-ocean:\n{do_lines}\nmachines:\n e2e-basic:\n new-user-name: e2euser\n"The file /home/david/projects/stirlingbridge/machine/tests/test_e2e.py has been updated successfully.
python3 -c "
cfg = {'access-token': 'tok', 'ssh-key': 'key', 'dns-zone': 'zone', 'machine-size': 's', 'image': 'img', 'region': 'nyc1'}
do_lines = '\n'.join(f' {k}: {v}' for k, v in cfg.items())
content = f'digital-ocean:\n{do_lines}\nmachines:\n e2e-basic:\n new-user-name: e2euser\n'
print(content)
"digital-ocean:
access-token: tok
ssh-key: key
dns-zone: zone
machine-size: s
image: img
region: nyc1
machines:
e2e-basic:
new-user-name: e2euserimport textwrap import uuid
import uuid
The file /home/david/projects/stirlingbridge/machine/tests/test_e2e.py has been updated successfully.
The bug was in _write_config() at line 82-91. When textwrap.dedent processes the f-string, only the first line of the interpolated {do_lines} gets the proper indentation from the surrounding template. The remaining lines of do_lines retain only their own 2-space prefix and end up at the root level of the YAML, which makes ruamel.yaml see multiple document-level mappings and throw expected '<document start>'.
The fix replaces the textwrap.dedent f-string with a simple f-string that produces correctly-indented YAML directly.
Exit code 1
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /home/david/projects/stirlingbridge/machine/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/david/projects/stirlingbridge/machine
configfile: pyproject.toml
collecting ... collected 4 items
tests/test_e2e.py::TestDropletLifecycle::test_create_and_list_and_destroy FAILED [ 25%]
=================================== FAILURES ===================================
____________ TestDropletLifecycle.test_create_and_list_and_destroy _____________
tests/test_e2e.py:155: in test_create_and_list_and_destroy
assert result.returncode == 0, f"create failed: {result.stderr}"
E AssertionError: create failed: Required key 'project' not found in 'digital-ocean' section of config file
E
E assert 1 == 0
E + where 1 = CompletedProcess(args=['uv', 'run', 'machine', '--config-file', '/tmp/pytest-of-david/pytest-3/e2e0/config.yml', '--session-id', '0c430dec', 'create', '--name', 'e2etest-e58da065', '--no-initialize', '--wait-for-ip'], returncode=1, stdout='', stderr="Required key 'project' not found in 'digital-ocean' section of config file\n").returncode
=========================== short test summary info ============================
FAILED tests/test_e2e.py::TestDropletLifecycle::test_create_and_list_and_destroy
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================== 1 failed in 0.37s ===============================The YAML parsing is fixed. Now it's failing because the project key is required by config but the test only includes it when E2E_PROJECT is set. Let me check the config validation.
1→import os
2→import re
3→from pathlib import Path
4→from machine.di import d
5→from machine.factory import yaml
6→from machine.log import fatal_error, debug
7→from machine.types import Config, MachineConfig
8→from machine import constants
9→
10→_env_var_pattern = re.compile(r"\$\{([^}]+)\}")
11→
12→
13→def _expand_env_vars(value):
14→ if isinstance(value, str):
15→
16→ def _replace(match):
17→ expr = match.group(1)
18→ if ":-" in expr:
19→ var_name, default = expr.split(":-", 1)
20→ return os.environ.get(var_name, default)
21→ else:
22→ if expr not in os.environ:
23→ fatal_error(f"Environment variable '{expr}' referenced in config is not set")
24→ return os.environ[expr]
25→
26→ return _env_var_pattern.sub(_replace, value)
27→ elif isinstance(value, dict):
28→ return {k: _expand_env_vars(v) for k, v in value.items()}
29→ elif isinstance(value, list):
30→ return [_expand_env_vars(item) for item in value]
31→ return value
32→
33→
34→class _loaded_config:
35→ c: any = None
36→
37→
38→def _load_config_data(config_file_name: str):
39→ if not config_file_name:
40→ config_file_name = constants.default_config_file_path
41→ config_path = Path(os.path.expanduser(config_file_name))
42→ if not config_path.exists():
43→ fatal_error(f"Error: Config file: {config_path} not found")
44→ config = _expand_env_vars(yaml().load(open(config_path, "r")))
45→ if d.opt.debug:
46→ debug(f"Loaded config file: {config_path}")
47→ debug(f"Parsed config: {config}")
48→ _loaded_config.c = config
49→ return config
50→
51→
52→def _require_key(d, key, section_name):
53→ if key not in d:
54→ fatal_error(f"Required key '{key}' not found in '{section_name}' section of config file")
55→ return d[key]
56→
57→
58→def get(config_file_name: str) -> Config:
59→ config = _load_config_data(config_file_name)
60→ if "digital-ocean" not in config:
61→ fatal_error("Required 'digital-ocean' section not found in config file")
62→ config_do = config["digital-ocean"]
63→ return Config(
64→ _require_key(config_do, "access-token", "digital-ocean"),
65→ _require_key(config_do, "ssh-key", "digital-ocean"),
66→ config_do.get("dns-zone"),
67→ _require_key(config_do, "machine-size", "digital-ocean"),
68→ _require_key(config_do, "image", "digital-ocean"),
69→ _require_key(config_do, "region", "digital-ocean"),
70→ _require_key(config_do, "project", "digital-ocean"),
71→ )
72→
73→
74→def get_machine(name: str) -> MachineConfig:
75→ if not _loaded_config.c:
76→ fatal_error("Attempt to fetch machine data before config loaded")
77→ config = _loaded_config.c
78→ if "machines" not in config:
79→ fatal_error("Required 'machines' section not found in config file")
80→ config_machines = config["machines"]
81→ if name not in config_machines:
82→ fatal_error(f"Machine type '{name}' not found in config file. Available types: {', '.join(config_machines.keys())}")
83→ target_config = config_machines[name]
84→ return MachineConfig(
85→ _require_key(target_config, "new-user-name", f"machines.{name}"),
86→ target_config.get("script-url"),
87→ target_config.get("script-dir"),
88→ target_config.get("script-path"),
89→ target_config.get("script-args"),
90→ )
91→
92→
93→def get_machines():
94→ if not _loaded_config.c:
95→ fatal_error("Attempt to fetch machine data before config loaded")
96→ config = _loaded_config.c
97→
98→ if "machines" not in config:
99→ fatal_error("Required 'machines' section not found in config file")
100→ ret = {}
101→ for name in config["machines"]:
102→ ret[name] = get_machine(name)
103→ return ret
104→The project field is required by config.py (line 70), but the test only includes it when E2E_PROJECT is set. The test needs to always provide a project value. Let me check if E2E_PROJECT is set in this session:
It's not set. The test treats project as optional but the config requires it. The fix: always include project in the config, defaulting to an empty string or a sensible default.
Looking at the test code, line 78-79 only adds project if E2E_PROJECT is truthy. Since config requires it, it should always be included:
if E2E_PROJECT:
cfg["project"] = E2E_PROJECTcfg["project"] = E2E_PROJECT
The file /home/david/projects/stirlingbridge/machine/tests/test_e2e.py has been updated successfully.
Wait — an empty string might also cause issues downstream. Let me check what E2E_PROJECT defaults to and how project is used.
{
"pattern": "\\.project",
"path": "/home/david/projects/stirlingbridge/machine/machine",
"output_mode": "content"
}machine/subcommands/create.py:110: if config.project: machine/subcommands/create.py:111: project_name = config.project
Good — config.project is only used if truthy (line 110), so an empty string is fine. Now let's re-run:
The user doesn't want to proceed with this tool use. The tool use was rejected (eg. if it was a file edit, the new_string was NOT written to the file). STOP what you are doing and wait for the user to tell you how to proceed. Note: The user's next message may contain a correction or preference. Pay close attention — if they explain what went wrong or how they'd prefer you to work, consider saving that to memory for future sessions.
Can you update the documentation to reflect that E2E_PROJECT must be set? Also you can set it to "Testing" to try the tests.
Required environment variables:
E2E_DO_TOKEN - DigitalOcean API token
E2E_SSH_KEY - Name of an SSH key already registered in DO
E2E_DNS_ZONE - DNS zone managed by DO (e.g. "test.example.com")
Optional environment variables:
E2E_REGION - Region slug (default: nyc1)
E2E_IMAGE - Image slug (default: ubuntu-24-04-x64)
E2E_SIZE - Machine size slug (default: s-1vcpu-512mb-10gb)
E2E_PROJECT - DO project name to assign droplets toRequired environment variables:
E2E_DO_TOKEN - DigitalOcean API token
E2E_SSH_KEY - Name of an SSH key already registered in DO
E2E_DNS_ZONE - DNS zone managed by DO (e.g. "test.example.com")
E2E_PROJECT - DO project name to assign droplets to
Optional environment variables:
E2E_REGION - Region slug (default: nyc1)
E2E_IMAGE - Image slug (default: ubuntu-24-04-x64)
E2E_SIZE - Machine size slug (default: s-1vcpu-512mb-10gb)The file /home/david/projects/stirlingbridge/machine/tests/test_e2e.py has been updated successfully.
Now update the skip-check logic to also require E2E_PROJECT, and remove the default of "":
E2E_PROJECT = os.environ.get("E2E_PROJECT", "")E2E_PROJECT = os.environ.get("E2E_PROJECT")The file /home/david/projects/stirlingbridge/machine/tests/test_e2e.py has been updated successfully.
if not E2E_DNS_ZONE:
_MISSING.append("E2E_DNS_ZONE")if not E2E_DNS_ZONE:
_MISSING.append("E2E_DNS_ZONE")
if not E2E_PROJECT:
_MISSING.append("E2E_PROJECT")The file /home/david/projects/stirlingbridge/machine/tests/test_e2e.py has been updated successfully.
E2E_PROJECT=Testing uv run pytest tests/test_e2e.py -m e2e -x -v --tb=short 2>&1
Exit code 1
============================= test session starts ==============================
platform linux -- Python 3.12.3, pytest-9.0.2, pluggy-1.6.0 -- /home/david/projects/stirlingbridge/machine/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/david/projects/stirlingbridge/machine
configfile: pyproject.toml
collecting ... collected 4 items
tests/test_e2e.py::TestDropletLifecycle::test_create_and_list_and_destroy FAILED [ 25%]
=================================== FAILURES ===================================
____________ TestDropletLifecycle.test_create_and_list_and_destroy _____________
tests/test_e2e.py:156: in test_create_and_list_and_destroy
assert result.returncode == 0, f"create failed: {result.stderr}"
E AssertionError: create failed: Traceback (most recent call last):
E File "/home/david/projects/stirlingbridge/machine/.venv/bin/machine", line 10, in <module>
E sys.exit(main())
E ^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
E return self.main(*args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
E rv = self.invoke(ctx)
E ^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
E return _process_result(sub_ctx.command.invoke(sub_ctx))
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
E return ctx.invoke(self.callback, **ctx.params)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
E return __callback(*args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
E return f(get_current_context(), *args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/machine/subcommands/create.py", line 103, in command
E droplet.create()
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/digitalocean/Droplet.py", line 568, in create
E data = self.get_data("droplets/", type=POST, params=data)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/digitalocean/Droplet.py", line 165, in get_data
E data = super(Droplet, self).get_data(*args, **kwargs)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/digitalocean/baseapi.py", line 233, in get_data
E raise DataReadError(msg)
E digitalocean.DataReadError: You are missing the required permission tag:create.
E
E assert 1 == 0
E + where 1 = CompletedProcess(args=['uv', 'run', 'machine', '--config-file', '/tmp/pytest-of-david/pytest-4/e2e0/config.yml', '--session-id', '5373eb7c', 'create', '--name', 'e2etest-5b2d50d4', '--no-initialize', '--wait-for-ip'], returncode=1, stdout='', stderr='Traceback (most recent call last):\n File "/home/david/projects/stirlingbridge/machine/.venv/bin/machine", line 10, in <module>\n sys.exit(main())\n ^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__\n return self.main(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main\n rv = self.invoke(ctx)\n ^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke\n return _process_result(sub_ctx.command.invoke(sub_ctx))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke\n return ctx.invoke...__callback(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func\n return f(get_current_context(), *args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/machine/subcommands/create.py", line 103, in command\n droplet.create()\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/digitalocean/Droplet.py", line 568, in create\n data = self.get_data("droplets/", type=POST, params=data)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/digitalocean/Droplet.py", line 165, in get_data\n data = super(Droplet, self).get_data(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/home/david/projects/stirlingbridge/machine/.venv/lib/python3.12/site-packages/digitalocean/baseapi.py", line 233, in get_data\n raise DataReadError(msg)\ndigitalocean.DataReadError: You are missing the required permission tag:create.\n').returncode
=========================== short test summary info ============================
FAILED tests/test_e2e.py::TestDropletLifecycle::test_create_and_list_and_destroy
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
============================== 1 failed in 1.90s ===============================The config parsing fix is working — the test got past config loading and successfully called the DigitalOcean API. The new error is You are missing the required permission tag:create, which is a DigitalOcean API token permissions issue — the token doesn't have the tag:create scope. That's an infrastructure/credentials issue, not a code bug.
To summarize the two code fixes made:
YAML indentation bug in _write_config() — textwrap.dedent with multi-line f-string interpolation broke indentation for all but the first line of do_lines, producing invalid YAML.
Missing required project field — the test treated E2E_PROJECT as optional, but config.py requires it. Updated the test to require it and moved it to the required env vars in the docs.
The remaining tag:create permission error is on the API token side — it needs broader scopes to create droplets and tags.
Ok thanks I will update the token permissions. Can you also update the reference to the environment variable in the markdown documentation? It says it's optional there.
1→# End-to-End Tests
2→
3→These tests verify that `machine` can create and manage real VMs on DigitalOcean, including DNS record lifecycle. They are **not** run as part of the normal test suite due to cost and runtime.
4→
5→## Prerequisites
6→
7→You need a DigitalOcean account with:
8→
9→- An API token
10→- An SSH key registered in the account
11→- A DNS zone managed by DigitalOcean (e.g. `test.example.com`)
12→
13→## Environment Variables
14→
15→| Variable | Required | Description |
16→|---|---|---|
17→| `E2E_DO_TOKEN` | Yes | DigitalOcean API token |
18→| `E2E_SSH_KEY` | Yes | Name of an SSH key in your DO account |
19→| `E2E_DNS_ZONE` | Yes | DO-managed DNS zone for test records |
20→| `E2E_REGION` | No | Region slug (default: `nyc1`) |
21→| `E2E_IMAGE` | No | Image slug (default: `ubuntu-24-04-x64`) |
22→| `E2E_SIZE` | No | Droplet size (default: `s-1vcpu-512mb-10gb`) |
23→| `E2E_PROJECT` | No | DO project to assign droplets to |
24→
25→## Running
26→
27→```bash
28→# Set credentials
29→export E2E_DO_TOKEN="dop_v1_..."
30→export E2E_SSH_KEY="my-ssh-key"
31→export E2E_DNS_ZONE="test.example.com"
32→
33→# Run e2e tests
34→make test-e2e
35→
36→# Or directly
37→uv run pytest tests/test_e2e.py -v -m e2e
38→```
39→
40→The normal `make test` (and CI) will **skip** these tests automatically.
41→
42→## What's Tested
43→
44→- **Droplet lifecycle** — create, list, destroy, verify removal
45→- **DNS lifecycle** — create with `--update-dns`, verify A record, destroy with `--delete-dns`, verify record removal
46→- **Cloud-init initialization** — create with `--type`, verify type tag
47→- **Custom tags** — create with `--tag`, verify tag filtering
48→
49→## Cleanup
50→
51→Each test cleans up after itself. A safety fixture also destroys any leftover droplets if a test fails mid-run. All test droplets use unique names prefixed with `e2etest-` so they are easy to identify.
52→
53→## Cost
54→
55→Tests use the smallest available droplet size (`s-1vcpu-512mb-10gb`) and destroy VMs immediately after verification, so cost is minimal.
56→| `E2E_DNS_ZONE` | Yes | DO-managed DNS zone for test records | | `E2E_REGION` | No | Region slug (default: `nyc1`) | | `E2E_IMAGE` | No | Image slug (default: `ubuntu-24-04-x64`) | | `E2E_SIZE` | No | Droplet size (default: `s-1vcpu-512mb-10gb`) | | `E2E_PROJECT` | No | DO project to assign droplets to |
| `E2E_DNS_ZONE` | Yes | DO-managed DNS zone for test records | | `E2E_PROJECT` | Yes | DO project to assign droplets to | | `E2E_REGION` | No | Region slug (default: `nyc1`) | | `E2E_IMAGE` | No | Image slug (default: `ubuntu-24-04-x64`) | | `E2E_SIZE` | No | Droplet size (default: `s-1vcpu-512mb-10gb`) |
The file /home/david/projects/stirlingbridge/machine/tests/E2E.md has been updated successfully.
export E2E_DNS_ZONE="test.example.com"
export E2E_DNS_ZONE="test.example.com" export E2E_PROJECT="my-project"
The file /home/david/projects/stirlingbridge/machine/tests/E2E.md has been updated successfully.
Done. Updated tests/E2E.md to mark E2E_PROJECT as required, moved it up with the other required variables, and added it to the example exports.