The current project is a Python CLI tool designed to support the building, configuring and deploying of container-based software services. I'm interested in adding a new feature: the capability to backup (and restore) service data. At present when a stack is deployed its software components and its configuration can be specified and held under revision control in git. However once it the stack is started it will ingest and otherwise update whatever persistent data it holds by modifying files in mounted volumes. Those mounted volumes are available in the host machine(s) filesystem and can in theory be backed up and restored from a previous epoch through that host side access. However we have never actually implemented this. I'm wondering if a different approach to the backup problem might be attempted: add a special container that is tasked with backing up data. It could host some filesystem to S3 (object storage) tool and run that tool on a regular basis. This container could be "mixed in" to any stack in a similar manner to how docker-based HTTPS/TLS ingress is done now (see ../docker-ingress-stack and ./docs/ingress.md#how-docker-ingress-works). Said backup container could have a way to be told which mounted directories in its filesystem need to be backed up, that config generated automatically since the stack tool already knows about the mounted volumes for all services. Restore is a bit more tricky, but I think can be done by first starting a version of the stack that only has the "backup" container service enabled, and perhaps given a command to run that restores the volumes from S3. Then that backup-mode stack can be stopped and the full stack subsequently started. It will now have the restored data in its mounted volumes. I'm interested in your thoughts on this scheme, and any ideas you have on tools to write/read object storage. Some sort of encryption of the payload data would need to be done since object storage isn't typically very secure. If the scheme is implemented then I expect we'll need a new repo for the "backup" container, similar to how the docker ingress stack has its own repository.
This is a great design question. Before I give you my thoughts, let me ground myself in how the "mixed in" ingress pattern actually works in your codebase, since the whole scheme hinges on replicating it.
1 # Ingress
2
3 Automatic configuration of HTTP routes for an ingress controller / reverse-proxy can be performed using comment-based
4 annotations in your `composefile.yml`.
5
6 ## Example
7
8 ```
9 services:
10 backend:
11 image: bozemanpass/todo-backend:stack
12 ports:
13 - "5000" # @stack http-proxy /api/todos
14
15 frontend:
16 image: bozemanpass/todo-frontend:stack
17 ports:
18 - "3000" # @stack http-proxy /
19 ```
20
21 These annotations will instruct the `stack init` command that `frontend:3000` should be published at the root path `/`
22 and that `backend:5000` should be published at `/api/todos`.
23
24 The hostname to be used is set by the `--http-proxy-fqdn` (the default is the result of `socket.getfqdn()`).
25
26 The result will be `http-proxy` settings in the `init` output similar to:
27
28 ```
29 network:
30 http-proxy:
31 - host-name: example.hostname.com
32 routes:
33 - path: /api/todos
34 proxy-to: backend:5000
35 - path: /
36 proxy-to: frontend:3000
37 ```
38
39 ## Kubernetes
40
41 When using Kubernetes, support for [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx) is built in.
42 SSL support is also handled automatically by [cert-manager](https://github.com/cert-manager/cert-manager) if installed.
43 The `cluster-issuer` to use for requesting a certificate can be specified using the `--http-proxy-clusterissuer` flag.
44 No additional configuration is required.
45
46 ## Docker
47
48 By using [bozemanpass/docker-ingress-stack](https://github.com/bozemanpass/docker-ingress-stack), similar functionality
49 can be achieved with Docker. This stack uses [nginxproxy/nginx-proxy](https://github.com/nginx-proxy/nginx-proxy) to
50 provide reverse proxy servies with automatically configured routes, similar to [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx).
51
52 > NOTE: If you do not need SSL, you can use the `docker-ingress-no-ssl` stack instead, which does not require any `--config` options.
53
54 The `docker-ingress` stack can simply be "mixed-in" to your existing stack at deployment time like this example below:
55
56 ```
57 # Fetch and init the docker-ingress stack
58 $ stack fetch repo bozemanpass/docker-ingress-stack
59
60 $ FQDN=todo.mydomain.com
61
62 # '--map-ports-to-host any-same' uses ports 80 and 443 on the host
63 # NOTE: The default ACME CA is https://acme-v02.api.letsencrypt.org/directory, but we override that for this example to use the staging CA.
64
65 $ stack init --stack docker-ingress \
66 --output ~/specs/docker-ingress.yml \
67 --map-ports-to-host any-same \
68 --config ACME_CA_URI=https://acme-staging-v02.api.letsencrypt.org/directory \
69 --config LETSENCRYPT_EMAIL=example@mydomain.com \
70 --config LETSENCRYPT_HOST=$FQDN
71
72 # Fetch and prepare your stack (here, 'todo').
73 $ stack fetch repo bozemanpass/example-todo-list
74 $ stack prepare --stack todo
75
76 # We are using config to tell the app what the final URL will be.
77 $ stack init --stack todo \
78 --output ~/specs/todo.yml \
79 --http-proxy-fqdn $FQDN \
80 --config REACT_APP_API_URL=https://${FQDN}/api/todos
81
82 # "Mix-in" the docker-ingress stack at deployment time.
83 $ stack deploy \
84 --spec-file ~/specs/docker-ingress.yml \
85 --spec-file ~/specs/todo.yml \
86 --deployment-dir ~/deployments/todo
87
88 $ stack manage --dir ~/deployments/todo start
89 $ stack manage --dir ~/deployments/todo ps
90 id: 882964b2300de, name: stack-3285f74574bd152c-backend-1, ports: 0.0.0.0:56462->5000/tcp
91 id: f8f39dc35c9e4, name: stack-3285f74574bd152c-db-1, ports: 0.0.0.0:56455->5432/tcp
92 id: 6c4d42c1c0f03, name: stack-3285f74574bd152c-nginx-proxy-1, ports: 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp
93 id: 700202db6e6ab, name: stack-3285f74574bd152c-frontend-1, ports: 0.0.0.0:56454->3000/tcp
94 ```
95
96 ## How Docker Ingress Works
97
98 The stack tool does not write nginx configuration files directly. Instead, it relies on
99 [nginxproxy/nginx-proxy](https://github.com/nginx-proxy/nginx-proxy), which automatically generates and reloads
100 nginx configuration by discovering environment variables on containers via the Docker API.
101
102 The process has three stages:
103
104 1. **Annotations to spec**: During `stack init`, port annotations in `composefile.yml` (e.g.,
105 `"3000" # @stack http-proxy /`) are parsed and written into an `http-proxy` section in the output spec file,
106 mapping paths to backend services and ports.
107
108 2. **Spec to environment variables**: During `stack deploy`, the tool reads the `http-proxy` spec and injects
109 environment variables into each matching service's Docker Compose definition:
110 - `VIRTUAL_HOST_MULTIPORTS` — a JSON object describing the hostname, path routing rules, and destination ports.
111 For non-root paths, a regex rule is generated that strips the prefix so the backend sees requests at `/`.
112 - `LETSENCRYPT_HOST` — set to the hostname (unless it is `localhost`), triggering automatic SSL certificate
113 provisioning via the companion letsencrypt container in the ingress stack.
114
115 3. **nginx-proxy generates nginx config**: The `nginx-proxy` container in the
116 [docker-ingress-stack](https://github.com/bozemanpass/docker-ingress-stack) watches the Docker socket for
117 container events. When it sees containers with `VIRTUAL_HOST_MULTIPORTS` environment variables, it generates
118 the appropriate nginx configuration and reloads nginx automatically.
119
120 ## Combining Stacks
121
122 It is often useful to create "super stacks" by combining and deploying multiple stacks together as a single unit.
123 In that case, conflicts can arise among the HTTP paths which are published by the individual stacks. A configuration
124 option, `http-proxy-prefix`, is available to "shift" all the published paths of a given stack to avoid conflicts.
125
126 Example:
127 ```
128 name: siwe-on-fixturenet
129 requires:
130 stacks:
131 - ref: bozemanpass/fixturenet-eth-stack
132 path: stacks/fixturenet-eth
133 http-proxy-prefix: /eth
134 - ref: bozemanpass/siwe-express-example
135 path: stacks/siwe-express-example
136 http-proxy-prefix: /
137 ```
138
139 In this case, the `fixturenet-eth` stack is shifted, so that all its published routes are prefixed with `/eth`. This
140 prefix will be stripped automatically by the proxy, so that the individual services will not see a different path than
141 if they had been deployed directly. The other stack in this example, `siwe-express-example`, is not shifted, and it
142 will be published at the root path (`/`).
143 ls -la /home/david/projects/bpi/ 2>/dev/null && echo "---DOCKER-INGRESS---" && ls -la /home/david/projects/bpi/docker-ingress-stack/ 2>/dev/null
total 24 drwxr-xr-x 6 david david 4096 Jun 29 19:19 . drwxr-xr-x 9 david david 4096 Mar 27 13:02 .. drwxr-xr-x 6 david david 4096 Jun 29 19:19 docker-ingress-stack drwxr-xr-x 12 david david 4096 Apr 8 12:43 machine drwxr-xr-x 12 david david 4096 Apr 7 17:57 stack drwxr-xr-x 9 david david 4096 Mar 27 13:01 test-python-cli-tool ---DOCKER-INGRESS--- total 28 drwxr-xr-x 6 david david 4096 Jun 29 19:19 . drwxr-xr-x 6 david david 4096 Jun 29 19:19 .. drwxr-xr-x 8 david david 4096 Jun 29 19:19 .git -rw-r--r-- 1 david david 249 Jun 29 19:19 README.md drwxr-xr-x 2 david david 4096 Jun 29 19:19 docker-ingress drwxr-xr-x 2 david david 4096 Jun 29 19:19 docker-ingress-no-ssl drwxr-xr-x 4 david david 4096 Jun 29 19:19 stacks
Let me look at the structure of the docker-ingress-stack and how the stack tool models volumes, to understand exactly what would need to be replicated.
cd /home/david/projects/bpi/docker-ingress-stack && find . -not -path './.git/*' -type f | head -50 && echo "=== docker-ingress stack.yml ===" && cat stacks/*/stack.yml 2>/dev/null; cat docker-ingress/*.yml 2>/dev/null | head -60
cd /home/david/projects/bpi/docker-ingress-stack && for d in stacks/*/; do echo "=== $d ==="; ls -la "$d"; done; echo "=== tree of stacks ==="; find stacks -not -path '*/.git/*' -type f
./README.md
./docker-ingress/composefile.yml
./docker-ingress-no-ssl/composefile.yml
./stacks/docker-ingress/stack.yml
./stacks/docker-ingress-no-ssl/stack.yml
=== docker-ingress stack.yml ===
name: docker-ingress-no-ssl
description: "docker-ingress-no-ssl"
pods:
- name: docker-ingress-no-ssl
path: docker-ingress-no-ssl
name: docker-ingress
description: "docker-ingress"
pods:
- name: docker-ingress
path: docker-ingressservices:
nginx-proxy:
image: nginxproxy/nginx-proxy
container_name: nginx-proxy
restart: always
volumes:
- certs:/etc/nginx/certs:ro
- conf:/etc/nginx/conf.d
- html:/usr/share/nginx/html
- vhost:/etc/nginx/vhost.d
- /var/run/docker.sock:/tmp/docker.sock:ro
ports:
- 80
- 443
acme-companion:
image: nginxproxy/acme-companion
container_name: nginx-proxy-acme
restart: always
environment:
ACME_CA_URI: ${ACME_CA_URI:-https://acme-staging-v02.api.letsencrypt.org/directory}
DEFAULT_EMAIL: ${LETSENCRYPT_EMAIL}
DEFAULT_HOST: ${LETSENCRYPT_HOST}
volumes_from:
- nginx-proxy
volumes:
- acme:/etc/acme.sh
- certs:/etc/nginx/certs:rw
- /var/run/docker.sock:/var/run/docker.sock:ro
volumes:
acme:
certs:
conf:
html:
vhost:
Shell cwd was reset to /home/david/projects/bpi/stack=== stacks/docker-ingress-no-ssl/ === total 12 drwxr-xr-x 2 david david 4096 Jun 29 19:19 . drwxr-xr-x 4 david david 4096 Jun 29 19:19 .. -rw-r--r-- 1 david david 135 Jun 29 19:19 stack.yml === stacks/docker-ingress/ === total 12 drwxr-xr-x 2 david david 4096 Jun 29 19:19 . drwxr-xr-x 4 david david 4096 Jun 29 19:19 .. -rw-r--r-- 1 david david 106 Jun 29 19:19 stack.yml === tree of stacks === stacks/docker-ingress/stack.yml stacks/docker-ingress-no-ssl/stack.yml Shell cwd was reset to /home/david/projects/bpi/stack
Now let me understand how the stack tool models volumes and where they live on the host — this is central to whether the backup container can actually see the data.
grep -rln "volume" src/stack/deploy/ --include=*.py | head && echo "===VOLUMES IN SPEC===" && grep -rn "volume" src/stack/deploy/spec.py | head -40
src/stack/deploy/deployer.py
src/stack/deploy/deployment.py
src/stack/deploy/deploy_util.py
src/stack/deploy/stack.py
src/stack/deploy/spec.py
src/stack/deploy/deploy.py
src/stack/deploy/k8s/deploy_k8s.py
src/stack/deploy/webapp/run_webapp.py
src/stack/deploy/deployment_create.py
src/stack/deploy/k8s/cluster_info.py
===VOLUMES IN SPEC===
src/stack/deploy/spec.py:119: def get_volumes(self):
src/stack/deploy/spec.py:120: return self.obj.get(constants.volumes_key, {})
src/stack/deploy/spec.py:128: vol_path = self.get_volumes().get(cfg_map_or_vol_name)
src/stack/deploy/spec.py:137: def get_volume_resources(self, volume_name):
src/stack/deploy/spec.py:138: return Resources(self.obj.get(constants.resources_key, {}).get(constants.volumes_key, {}).get(volume_name, {}))
src/stack/deploy/spec.py:298: # Check for conflicts on volume names
src/stack/deploy/spec.py:299: current_volume_names = list(self.get_volumes().keys())
src/stack/deploy/spec.py:300: other_volume_names = list(other.get_volumes().keys())
src/stack/deploy/spec.py:301: for vol_name in other_volume_names:
src/stack/deploy/spec.py:302: if vol_name in current_volume_names:{
"subagent_type": "Explore",
"prompt": "I'm researching the BPI Stack codebase at /home/david/projects/bpi/stack (a Python CLI that abstracts Docker Compose and Kubernetes deployments). I need to understand how it models and handles persistent VOLUMES, because I'm designing a backup/restore feature.\n\nPlease investigate and report concisely on:\n\n1. How are volumes/mounts defined and parsed? Look in src/stack/deploy/spec.py, src/stack/deploy/stack.py, and the compose deployer src/stack/deploy/compose/deploy_docker.py. What does the \"volumes\" section of a spec look like?\n\n2. Where do named volumes physically land on the HOST filesystem? Search for how host paths for volumes are computed — look for \"volumes\" directory handling, bind mounts, deployment_context.py, and any code that maps a named volume to a host directory inside the deployment dir.\n\n3. How does the K8s deployer handle volumes (src/stack/deploy/k8s/)? PersistentVolumeClaims? hostPath? Look at deploy_k8s.py and helpers.py.\n\n4. How does \"mixing in\" a stack work mechanically at deploy time? When multiple --spec-file args are passed, how are services merged? Look for MergedSpec in spec.py and how deployment.py / deployer_factory.py combine multiple stacks into one deployment. Specifically: do all mixed-in services share the same Docker Compose project / network, and can one service see another's named volumes?\n\n5. Is there any existing notion of a service that needs access to the Docker socket or to other services' volumes (like the nginx-proxy ingress example which mounts /var/run/docker.sock)?\n\nReport file paths and line numbers for the key code. Quote short relevant snippets. Don't write any code — this is research only."
}[
{
"type": "text",
"text": "Async agent launched successfully.\nagentId: a555ae4f4fdd1dfac (internal ID - do not mention to user. Use SendMessage with to: 'a555ae4f4fdd1dfac', summary: '<5-10 word recap>' to continue this agent.)\nThe agent is working in the background. You will be notified automatically when it completes.\nDo not duplicate this agent's work — avoid working with the same files or topics it is using.\noutput_file: /tmp/claude-1000/-home-david-projects-bpi-stack/83ff7993-2200-4857-b374-743227b368dd/tasks/a555ae4f4fdd1dfac.output\nDo NOT Read or tail this file via the shell tool — it is the full subagent JSONL transcript and reading it will overflow your context. If the user asks for progress, say the agent is still running; you'll get a completion notification."
}
]{
"file_path": "/home/david/projects/bpi/stack/src/stack/deploy/spec.py",
"offset": 110,
"limit": 60
}110 def get_image_registry(self):
111 return self.obj.get(constants.image_registry_key)
112
113 def get_config(self):
114 return self.obj.get(constants.config_key, {})
115
116 def get_kube_config(self):
117 return self.obj.get(constants.kube_config_key, None)
118
119 def get_volumes(self):
120 return self.obj.get(constants.volumes_key, {})
121
122 def get_configmaps(self):
123 return self.obj.get(constants.configmaps_key, {})
124
125 def fully_qualified_path(self, cfg_map_or_vol_name):
126 vol_path = self.get_configmaps().get(cfg_map_or_vol_name)
127 if not vol_path:
128 vol_path = self.get_volumes().get(cfg_map_or_vol_name)
129 if vol_path and not vol_path.startswith("/"):
130 vol_path = os.path.join(os.path.dirname(self.file_path), vol_path)
131
132 return vol_path
133
134 def get_container_resources(self, service_name):
135 return Resources(self.obj.get(constants.resources_key, {}).get("containers", {}).get(service_name, {}))
136
137 def get_volume_resources(self, volume_name):
138 return Resources(self.obj.get(constants.resources_key, {}).get(constants.volumes_key, {}).get(volume_name, {}))
139
140 def get_http_proxy(self):
141 return self.obj.get(constants.network_key, {}).get(constants.http_proxy_key, [])
142
143 def _clear_http_proxy(self):
144 if constants.network_key in self.obj:
145 if constants.http_proxy_key in self.obj[constants.network_key]:
146 del self.obj[constants.network_key][constants.http_proxy_key]
147
148 def _set_http_proxy(self, val):
149 if constants.network_key not in self.obj:
150 self.obj[constants.network_key] = {}
151 self.obj[constants.network_key][constants.http_proxy_key] = val
152
153 def get_annotations(self):
154 return self.obj.get(constants.annotations_key, {})
155
156 def get_replicas(self):
157 return self.obj.get(constants.replicas_key, 1)
158
159 def get_node_affinities(self):
160 return self.obj.get(constants.node_affinities_key, [])
161
162 def get_node_tolerations(self):
163 return self.obj.get(constants.node_tolerations_key, [])
164
165 def get_labels(self):
166 return self.obj.get(constants.labels_key, {})
167
168 def get_privileged(self, container_name):
169 return "true" == str(self.obj.get(constants.security_key, {}).get(container_name, {}).get("privileged", "false")).lower()The ingress pattern is clear. Let me quickly check the K8s volume mechanics myself while the research agent finishes, since K8s is where this design gets tricky.
grep -n "PersistentVolume\|hostPath\|ReadWrite\|claim\|pvc\|PVC\|access_mode\|AccessMode" src/stack/deploy/k8s/*.py | head -40
src/stack/deploy/k8s/helpers.py:196: claim = client.V1PersistentVolumeClaimVolumeSource(claim_name=volume_name)
src/stack/deploy/k8s/helpers.py:197: volume = client.V1Volume(name=volume_name, persistent_volume_claim=claim)
src/stack/deploy/k8s/helpers.py:214: claim = client.V1PersistentVolumeClaimVolumeSource(claim_name=volume_name)
src/stack/deploy/k8s/helpers.py:215: volume = client.V1Volume(name=volume_name, persistent_volume_claim=claim)
src/stack/deploy/k8s/helpers.py:258: f" - hostPath: {_make_absolute_host_path(volume_host_path_map[volume_name], deployment_dir)}\n"
src/stack/deploy/k8s/helpers.py:358:# - hostPath: /path/to/my/files
src/stack/deploy/k8s/deploy_k8s.py:133: # Figure out the PVCs for this deployment
src/stack/deploy/k8s/deploy_k8s.py:134: pvcs = self.cluster_info.get_pvcs()
src/stack/deploy/k8s/deploy_k8s.py:135: for pvc in pvcs:
src/stack/deploy/k8s/deploy_k8s.py:136: log_debug(f"Sending this pvc: {pvc}")
src/stack/deploy/k8s/deploy_k8s.py:140: pvc_resp = self.core_api.read_namespaced_persistent_volume_claim(
src/stack/deploy/k8s/deploy_k8s.py:141: name=pvc.metadata.name, namespace=self.k8s_namespace
src/stack/deploy/k8s/deploy_k8s.py:143: if pvc_resp:
src/stack/deploy/k8s/deploy_k8s.py:144: log_debug("PVCs already present:")
src/stack/deploy/k8s/deploy_k8s.py:145: log_debug(f"{pvc_resp}")
src/stack/deploy/k8s/deploy_k8s.py:150: pvc_resp = self.core_api.create_namespaced_persistent_volume_claim(body=pvc, namespace=self.k8s_namespace)
src/stack/deploy/k8s/deploy_k8s.py:151: log_debug("PVCs created:")
src/stack/deploy/k8s/deploy_k8s.py:152: log_debug(f"{pvc_resp}")
src/stack/deploy/k8s/deploy_k8s.py:295: # Figure out the PVCs for this deployment
src/stack/deploy/k8s/deploy_k8s.py:296: pvcs = self.cluster_info.get_pvcs()
src/stack/deploy/k8s/deploy_k8s.py:297: for pvc in pvcs:
src/stack/deploy/k8s/deploy_k8s.py:298: log_debug(f"Deleting this pvc: {pvc}")
src/stack/deploy/k8s/deploy_k8s.py:300: pvc_resp = self.core_api.delete_namespaced_persistent_volume_claim(
src/stack/deploy/k8s/deploy_k8s.py:301: name=pvc.metadata.name, namespace=self.k8s_namespace
src/stack/deploy/k8s/deploy_k8s.py:303: log_debug("PVCs deleted:")
src/stack/deploy/k8s/deploy_k8s.py:304: log_debug(f"{pvc_resp}")
src/stack/deploy/k8s/cluster_info.py:204: def get_pvcs(self):
src/stack/deploy/k8s/cluster_info.py:232: spec = client.V1PersistentVolumeClaimSpec(
src/stack/deploy/k8s/cluster_info.py:233: access_modes=["ReadWriteOnce"],
src/stack/deploy/k8s/cluster_info.py:238: pvc = client.V1PersistentVolumeClaim(
src/stack/deploy/k8s/cluster_info.py:242: result.append(pvc)
src/stack/deploy/k8s/cluster_info.py:280: # Otherwise, we create the PVC and expect the node to allocate the volume for us.
src/stack/deploy/k8s/cluster_info.py:282: log_debug(f"{volume_name} does not require an explicit PersistentVolume, since it is not a bind-mount.")
src/stack/deploy/k8s/cluster_info.py:301: spec = client.V1PersistentVolumeSpec(
src/stack/deploy/k8s/cluster_info.py:303: access_modes=["ReadWriteOnce"],
src/stack/deploy/k8s/cluster_info.py:307: pv = client.V1PersistentVolume(Confirmed the key constraint I suspected: K8s PVCs are created ReadWriteOnce (cluster_info.py:233,303). That matters a lot for this design. Let me wait for the volume-mapping research to come back before I synthesize.
I have the critical facts. Let me wait for the research agent to confirm the merge/host-path details before writing my full assessment — that's the one piece that determines whether the Docker design works as cleanly as you hope.
{
"file_path": "/home/david/projects/bpi/stack/src/stack/deploy/k8s/helpers.py",
"offset": 180,
"limit": 90
}180 for pod in parsed_pod_files:
181 parsed_pod_file = parsed_pod_files[pod]
182 if "services" in parsed_pod_file:
183 services = parsed_pod_file["services"]
184 for service_name in services:
185 if service_name == service:
186 service_obj = services[service_name]
187 if "volumes" in service_obj:
188 volumes = service_obj["volumes"]
189 for mount_string in volumes:
190 volume_name = mount_string.split(":")[0]
191 if volume_name in spec.get_configmaps():
192 config_map = client.V1ConfigMapVolumeSource(name=volume_name)
193 volume = client.V1Volume(name=volume_name, config_map=config_map)
194 result.append(volume)
195 else:
196 claim = client.V1PersistentVolumeClaimVolumeSource(claim_name=volume_name)
197 volume = client.V1Volume(name=volume_name, persistent_volume_claim=claim)
198 result.append(volume)
199 return result
200
201
202 def volumes_for_pod_files(parsed_pod_files, spec, app_name):
203 result = []
204 for pod in parsed_pod_files:
205 parsed_pod_file = parsed_pod_files[pod]
206 if "volumes" in parsed_pod_file:
207 volumes = parsed_pod_file["volumes"]
208 for volume_name in volumes.keys():
209 if volume_name in spec.get_configmaps():
210 config_map = client.V1ConfigMapVolumeSource(name=volume_name)
211 volume = client.V1Volume(name=volume_name, config_map=config_map)
212 result.append(volume)
213 else:
214 claim = client.V1PersistentVolumeClaimVolumeSource(claim_name=volume_name)
215 volume = client.V1Volume(name=volume_name, persistent_volume_claim=claim)
216 result.append(volume)
217 return result
218
219
220 def _get_host_paths_for_volumes(deployment_context):
221 return deployment_context.spec.get_volumes()
222
223
224 def _make_absolute_host_path(data_mount_path: Path, deployment_dir: Path) -> Path:
225 if os.path.isabs(data_mount_path):
226 return data_mount_path
227 else:
228 # Python Path voodo that looks pretty odd:
229 return Path.cwd().joinpath(deployment_dir.joinpath(data_mount_path)).resolve()
230
231
232 def _generate_kind_mounts(parsed_pod_files, deployment_dir, deployment_context):
233 volume_definitions = []
234 volume_host_path_map = _get_host_paths_for_volumes(deployment_context)
235 # Note these paths are relative to the location of the pod files (at present)
236 # So we need to fix up to make them correct and absolute because kind assumes
237 # relative to the cwd.
238 for pod in parsed_pod_files:
239 parsed_pod_file = parsed_pod_files[pod]
240 if "services" in parsed_pod_file:
241 services = parsed_pod_file["services"]
242 for service_name in services:
243 service_obj = services[service_name]
244 if "volumes" in service_obj:
245 volumes = service_obj["volumes"]
246 for mount_string in volumes:
247 # Looks like: test-data:/data or test-data:/data:ro or test-data:/data:rw
248 log_debug(f"mount_string: {mount_string}")
249 mount_split = mount_string.split(":")
250 volume_name = mount_split[0]
251 mount_path = mount_split[1]
252 log_debug(f"volume_name: {volume_name}")
253 log_debug(f"map: {volume_host_path_map}")
254 log_debug(f"mount path: {mount_path}")
255 if volume_name not in deployment_context.spec.get_configmaps():
256 if volume_host_path_map[volume_name]:
257 volume_definitions.append(
258 f" - hostPath: {_make_absolute_host_path(volume_host_path_map[volume_name], deployment_dir)}\n"
259 f" containerPath: {get_kind_pv_bind_mount_path(volume_name)}\n"
260 )
261 return "" if len(volume_definitions) == 0 else (" extraMounts:\n" f"{"".join(volume_definitions)}")
262
263
264 # TODO: decide if we need this functionality
265 def _generate_kind_port_mappings_from_services(parsed_pod_files):
266 port_definitions = []
267 for pod in parsed_pod_files:
268 parsed_pod_file = parsed_pod_files[pod]
269 if "services" in parsed_pod_file:I've now got a solid picture of both the ingress mix-in mechanism and the volume internals. Here's my assessment.
The scheme is sound, and the ingress-style "mix-in" is a genuinely good fit — for the Docker target. For Kubernetes the same idea runs into a hard wall (ReadWriteOnce PVCs) and needs a different mechanism. The S3/encryption part is the easy part — solved by off-the-shelf tools. The hard parts are (1) concurrent access to other services' volumes and (2) application-level consistency. Your restore design actually sidesteps the worst of #1, which is the cleverest part of your proposal.
Use restic (or kopia) as the engine. Both are single-binary Go tools that do exactly what you want and treat object storage as hostile by design:
keep-daily 7 keep-weekly 4 …) and easy point-in-time restore — your "previous epoch."restic is the more battle-tested/simpler; kopia adds a built-in scheduler and policy engine. I'd default to restic for a first cut.
For the container itself, you very likely don't need to build much: offen/docker-volume-backup is purpose-built for exactly this use case — a container that mounts Docker volumes, backs them up on a cron schedule to S3/restic/WebDAV with encryption, and supports pre/post hooks for quiescing. It would let your new repo be mostly a thin stack wrapper rather than original code. Worth evaluating before committing to a from-scratch image. (mazzolino/restic is a simpler restic-on-cron alternative.)
The ingress precedent confirms the plumbing: mixed-in specs merge into a single Compose project sharing one network (the ps output in ingress.md shows all containers under one stack-<hash> project, and spec.py:298-302 already reconciles volume-name conflicts across merged specs). So a backup service dropped in as another spec lands in the same project.
The one difference from ingress: nginx-proxy only needs the Docker socket to discover env vars. Your backup container needs to actually read the bytes, which means each named volume must be mounted into the backup container (read-only). That's the auto-generated config you anticipated — the stack tool already enumerates volumes via Spec.get_volumes(), so generating - <vol>:/backup/<vol>:ro for every volume in the merged deployment is straightforward. (volumes_from: is an alternative but per-container; explicit read-only mounts of named volumes are cleaner and let you back up volumes regardless of which service owns them.)
cluster_info.py:233,303 creates every PVC as ReadWriteOnce. That single fact breaks the "backup as a separate mixed-in pod" model: an RWO volume already attached to the app pod cannot be mounted into a second pod (especially across nodes). So on K8s you can't have a standalone backup pod read live volumes the way Docker can. Options, roughly in order of cleanliness:
ReadWriteMany — requires an RWX-capable storage class (NFS, CephFS, etc.), which many clusters don't have.I'd flag this asymmetry explicitly in the design doc: Docker gets the clean mix-in; K8s backup-while-running needs a sidecar or an RWX story. It's fine to ship Docker-first.
Starting a stripped deployment with only the backup service, restoring into the (freshly-created, empty) volumes, then stopping it and starting the full stack — this is elegant precisely because at restore time nothing else holds the volumes. That eliminates the concurrent-mount problem on both targets: on Docker compose up with only the backup service still creates all declared named volumes empty; on K8s a one-shot Job can mount the RWO PVCs because no app pod is attached yet. So restore works uniformly even though backup doesn't. I'd lean into this and make restore a first-class stack manage … restore --snapshot <id> mode that deploys exactly this minimal variant.
The ingress analogy is reassuring but slightly misleading — ingress is stateless; backup is deeply stateful. File-level copy of a live volume under a running database (Postgres, etc.) can produce a torn, unrestorable snapshot. You need a quiesce mechanism:
pg_dump, fsfreeze, or briefly stopping the writer), post-hook to release.# @stack http-proxy /api drives ingress config, you could have # @stack backup-volume /data and # @stack backup-pre-hook pg_dump … annotations parsed at init into a backup: spec section. That keeps the per-service knowledge with the service and reuses machinery you already have.docker-ingress-stack's layout (stacks/<name>/stack.yml + <name>/composefile.yml) — e.g. bozemanpass/backup-stack.Build it Docker-first as a backup-stack repo wrapping restic (evaluate offen/docker-volume-backup to avoid writing the engine), auto-generate read-only volume mounts and the backup target list from Spec.get_volumes(), drive per-service quiesce via @stack backup-* annotations, and implement restore as a dedicated minimal-deployment mode. Treat K8s backup-while-running as a separate, later phase (sidecar or RWX), but note that restore can work on K8s from day one via a Job, thanks to your start-stripped-then-restore design.
Want me to (a) write this up as a design doc in docs/, (b) prototype the spec-generation changes (get_volumes() → backup container mounts) in the stack tool, or (c) scaffold the new backup-stack repo layout? I'd suggest starting with (a) so we pin down the annotation schema and the Docker-vs-K8s scope before any code.
Note: I had a background agent mapping the volume-merge internals in more detail — I can fold its findings in if you want to go deeper on the spec-merge mechanics before deciding.
I've thoroughly analyzed the BPI Stack codebase to understand how it handles volumes, persistent storage, and multi-stack deployments. Here are my findings for your 5 research questions:
Spec Volume Definition (/home/david/projects/bpi/stack/src/stack/deploy/spec.py, lines 119-120):
def get_volumes(self):
return self.obj.get(constants.volumes_key, {})
The spec file contains a top-level volumes dictionary with named volumes as keys and paths as values. Example structure:
volumes:
my-data: ./data/my-data # Docker Compose: relative path → bind mount
my-volume: /absolute/path # K8s: absolute path → HostPath PV
another-vol: null # K8s: no path → auto-provisioned PVC
Service-Level Volume Mounts (/home/david/projects/bpi/stack/src/stack/deploy/stack.py, lines 240-274):
Services in compose files define volumes using standard Docker syntax: volume-name:/container/mount/path[:ro|rw]
The get_named_volumes() method (stack.py, line 240) parses all service mount points and categorizes volumes as "rw" (read-write) or "ro" (read-only) based on the mount options.
Volume Specification in Deployment (/home/david/projects/bpi/stack/src/stack/deploy/deployment_create.py, lines 362-382):
When generating a deployment spec, the system auto-creates volume descriptors based on whether services reference them:
- For Docker Compose: ./data/{volume-name} (relative paths to deployment dir)
- For Kubernetes: None (triggers auto-provisioning) or absolute paths (for bind mounts)
Docker Compose (Bind Mount Approach):
(/home/david/projects/bpi/stack/src/stack/deploy/deployment_create.py, lines 77-100):
The _fixup_pod_file() function transforms named volumes into Docker bind mounts with this structure:
new_volume_spec = {
"driver": "local",
"driver_opts": {
"type": "none",
"device": volume_spec_fixedup, # Relative or absolute path
"o": "bind",
},
}
Host Path Calculation:
- Relative paths (e.g., ./data/my-vol) are resolved relative to the deployment directory (not the spec file)
- Absolute paths (e.g., /var/data/my-vol) are used directly
- The code creates the bind directories if they don't exist (_create_bind_dir_if_relative(), line 66)
Directory Structure on Disk:
deployment-001/
├── compose/ # Generated Docker Compose files
├── config/ # Shared config files
├── data/
│ ├── my-data/ # Volume 1 (bind mount)
│ └── another-vol/ # Volume 2 (bind mount)
├── pods/
├── spec.yml
├── config.env
└── deployment.yml
Kubernetes (HostPath PV Approach):
(/home/david/projects/bpi/stack/src/stack/deploy/k8s/cluster_info.py, lines 274-315):
For K8s, volumes use hostPath PersistentVolumes (PVs) only when the spec explicitly provides an absolute path. The code:
1. Creates a PersistentVolume with hostPath pointing to the specified path (line 300)
2. Creates a PersistentVolumeClaim that binds to the PV (lines 204-243)
3. For Kind clusters, remaps paths to /mnt/{volume_name} via Kind's extraMounts mechanism (line 298)
PersistentVolumeClaim (PVC) Generation (/home/david/projects/bpi/stack/src/stack/deploy/k8s/cluster_info.py, lines 204-243):
For each named volume referenced in services:
1. If spec provides a path → creates both PV (hostPath) and PVC (claim)
2. If spec provides null (no path) → creates only PVC (auto-provisioned by storage class)
spec = client.V1PersistentVolumeClaimSpec(
access_modes=["ReadWriteOnce"],
storage_class_name=storage_class_name, # "manual" for hostPath, None for auto
resources=to_k8s_resource_requirements(resources),
volume_name=k8s_volume_name, # References PV name
)
Volume Mounting in Deployments (/home/david/projects/bpi/stack/src/stack/deploy/k8s/helpers.py, lines 146-174):
The volume_mounts_for_service() function creates K8s VolumeMount objects from compose-style mount strings:
volume_mount = client.V1VolumeMount(
mount_path=mount_path,
name=volume_name,
read_only="ro" == mount_options,
)
Pod Volume References (/home/david/projects/bpi/stack/src/stack/deploy/k8s/helpers.py, lines 177-199):
The volumes_for_service() function determines if a volume is:
- A ConfigMap (if listed in spec.get_configmaps()) → uses ConfigMapVolumeSource
- A PVC (otherwise) → uses PersistentVolumeClaimVolumeSource
Multiple Spec Files Handling (/home/david/projects/bpi/stack/src/stack/deploy/deployment_create.py, lines 457-462):
When multiple --spec-file args are passed:
if len(spec_file) == 1:
spec = load_spec(spec_file[0])
else:
spec = MergedSpec()
for sf in spec_file:
spec.merge(load_spec(sf))
MergedSpec.merge() Logic (/home/david/projects/bpi/stack/src/stack/deploy/spec.py, lines 272-351):
The merge process performs validation checks first, then combines specs:
Port mappings must not conflict
HTTP Proxy Merging (lines 318-337):
Uses mergedeep Strategy.ADDITIVE to combine routes
Object Merging (lines 335-349):
mergedeep librarymerge(self.obj, other.obj, strategy=Strategy.ADDITIVE)
# All added specs tracked in self._specs list
self._specs.append(other)
Critical Finding: Each spec's volumes dictionary is merged additively. Services in different specs access their own volumes only — there is no automatic cross-service volume sharing. When specs are merged:
service-name.namespace.svc.cluster.local in K8s)Stack Merging (/home/david/projects/bpi/stack/src/stack/deploy/spec.py, lines 227-232):
def merge_stacks(self):
stacks = self.load_stacks()
ret = Stack()
for stack in stacks:
merge(ret.obj, stack.obj, strategy=Strategy.ADDITIVE)
return ret
All pods from all referenced stacks are merged into a single Stack object. The merged stack's services all run together with a shared Docker Compose project/K8s namespace.
Current State: There is no existing built-in pattern for socket sharing in BPI Stack. The codebase does not contain references to /var/run/docker.sock or similar socket mounting scenarios.
Potential Path Forward (based on code structure):
For a service that needs access to Docker socket or another service's volumes:
For Docker Compose: Services in the same compose file can be modified to add socket mounts via the pod's volumes section in the compose file itself:
yaml
services:
my-service:
volumes:
- /var/run/docker.sock:/var/run/docker.sock # Direct path, not a named volume
For Kubernetes: Would require:
hostPath volume in the pod spec (not currently done by BPI Stack)configmaps spec keyVolume Access Across Services:
The http_proxy feature (lines 550-574 in deployment_create.py) shows the pattern for inter-service communication. Services are identified by name and port, allowing HTTP routing. Volumes follow a similar principle — services must be in the same deployment and explicitly reference volumes by name.
| Question | Key Files |
|---|---|
| 1. Volume parsing | spec.py:119-120, stack.py:240-274, deployment_create.py:362-382 |
| 2. Host filesystem paths | deployment_create.py:66-100, cluster_info.py:274-315 |
| 3. K8s volume handling | cluster_info.py:204-243, helpers.py:146-199, deploy_k8s.py:114-152 |
| 4. Spec merging | spec.py:272-351, deployment_create.py:457-462, stack.py:161-169 |
| 5. Socket sharing | No existing patterns found; would require custom compose/K8s extensions |
The research confirms my analysis and sharpens two points materially — both worth folding in:
1. On Docker, the data is already sitting on the host at a predictable path. Named volumes aren't opaque Docker volumes — deployment_create.py:66-100 rewrites every named volume into a bind mount pointing at <deployment-dir>/data/<volume-name>/. Two consequences:
- The backup container's mounts are fully deterministic, and
- It quietly revives the "host-side backup" option you dismissed: a backup container could bind-mount <deployment-dir>/data as a single tree and back the whole thing up, rather than enumerating volumes one by one. That's simpler to generate, though it loses per-volume/per-service granularity for the quiesce hooks — so I'd still prefer explicit per-volume mounts, but it's a viable fallback and a nice property to know about.
2. The backup service can't be a purely static mix-in — it needs codegen, exactly like ingress. This is the key correction to my earlier framing. The merge step requires unique volume names across specs (spec.py:298-303) and does not auto-share volumes — a service only sees volumes it explicitly lists. So the backup stack's composefile.yml cannot statically name the app's volumes (it doesn't know them, and naming them would collide at merge). The volume-mount list must be injected by the stack tool at deploy time, reading the merged spec's get_volumes() and writing - <vol>:/backup/<vol>:ro into the backup service — precisely parallel to how deploy injects VIRTUAL_HOST_MULTIPORTS env vars into matching services for ingress (ingress.md stage 2).
So the architecture is: static stack repo (the backup container image + a skeletal service def) + a new deploy-time injection pass in the stack tool that wires the merged deployment's volumes (and any @stack backup-* annotations) into that service. That injection pass is the real implementation work, and it has a clean precedent to copy.
Nothing else in my assessment changes — Docker-first, restic engine, RWO blocks K8s backup-while-running, and your start-stripped-then-restore mode remains the elegant unifier for restore.
My recommendation stands: start with a design doc so we pin the annotation schema and the injection-pass contract. Want me to draft it into docs/backup.md (mirroring the structure of docs/ingress.md)?
Hmm ok, good analysis. Let's kick this around some more: I'm aware of the general problem of gaining cross-cluster access to filesystem data. Addressing that problem is not a goal for now. We already run on k8s in a more constrained mode where either there's only one cluster node, or if there are multiple nodes we use node affinity to place services adjacent to others so their persistent data is available without a network traversal. That said, I do wonder what people are already doing for backup in a k8s-centric deployment. Perhaps it's like the ingress situation where k8s already has a solid story and we only need to cover the corresponding docker case?
Good question — and this is exactly the kind of thing I shouldn't answer from memory alone, since the k8s backup ecosystem moves fast. Let me research the current state.
[
{
"type": "tool_reference",
"tool_name": "WebSearch"
},
{
"type": "tool_reference",
"tool_name": "WebFetch"
}
]{
"query": "Velero Kubernetes backup persistent volume restic kopia node-agent file system backup 2025"
}{
"query": "K8up CNCF restic kubernetes backup operator vs Velero Stash Kanister comparison"
}Web search results for query: "K8up CNCF restic kubernetes backup operator vs Velero Stash Kanister comparison"
Links: [{"title":"Backups for K8s and Beyond :: Jacob Colvin — My Website","url":"https://jacobcolvin.com/posts/2023/01/backups-for-k8s-and-beyond/"},{"title":"Velero: Strategic Open Source Alternative to Kasten K10","url":"https://www.opentechhub.io/velero/"},{"title":"Kubernetes Backup Tools: Comparing Cohesity, Kasten, OpenEBS, Portworx, Rancher Longhorn, and Velero","url":"https://portworx.com/blog/kubernetes-backup-tools/"},{"title":"K8up - Cloud Native Storage | Rawkode Academy","url":"https://rawkode.academy/technology/k8up"},{"title":"Kubernetes Data Only Backup with K8up | Yet Another Techblog","url":"https://wazaari.dev/blog/k8s-backup-with-k8up"},{"title":"Show HN: K8up – Kubernetes Backup Operator Based on Restic | Hacker News","url":"https://news.ycombinator.com/item?id=20769362"}]
Based on the search results, here's a comparison of K8up and other Kubernetes backup operators:
## K8up Overview
K8up is a CNCF backup operator built on Restic; lighter and more opinionated than Velero, focused on backing up PVCs and application data rather than full cluster-resource capture and migration. K8up is an active CNCF sandbox project, and it's actively maintained and has a growing community.
## Key Differences from Velero
Velero focuses on whole-cluster disaster recovery and is heavier; K8up is simpler and per-namespace, which tends to suit multi-tenant clusters where each team wants to own its own backup policy. Additionally, Velero does not support the Restic REST server backend, whereas K8up supports using the Restic REST server as a backend.
## K8up Technical Features
Because it is restic under the hood, you get deduplication, encryption at rest, and incremental backups for free, and you can restore using plain restic from outside the cluster if the operator is unavailable. K8up also supports pre-backup hooks — it can exec into pods to run pg_dump, mysqldump, or custom commands and stream the output into the same restic repository, so databases get consistent logical backups alongside volume snapshots.
## Kanister
Kanister is an application-centric, open-source data-management framework that expresses app-aware backup workflows as blueprints; commonly used alongside Velero's cluster-level backup rather than instead of it.
REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.Web search results for query: "Velero Kubernetes backup persistent volume restic kopia node-agent file system backup 2025"
Links: [{"title":"Velero Docs - File System Backup","url":"https://velero.io/docs/v1.16/file-system-backup/"},{"title":"Configure Velero Restic Integration for File-Level Backup of Persistent Volumes","url":"https://oneuptime.com/blog/post/2026-02-09-velero-restic-file-level-backup/view"},{"title":"Velero Reports Partial Failure \"Error backing up item error: pod volume backup failed: error to initialize data path: error to boost backup repository connection\"","url":"https://knowledge.broadcom.com/external/article/405677/velero-reports-partial-failure-error-bac.html"},{"title":"Kubernetes Backup using Velero","url":"https://afi.ai/blog/kubernetes-velero-backup"},{"title":"How to Back Up and Restore Kubernetes Clusters with Velero","url":"https://oneuptime.com/blog/post/2026-01-19-kubernetes-velero-backup-restore/view"},{"title":"Velero Kubernetes Backup: Namespace, PVC, DR | ComputingForGeeks","url":"https://computingforgeeks.com/velero-kubernetes-backup-restore-pvc/"},{"title":"Using Velero for backup and restoration of your Managed Kubernetes - Knowledge Base - Leaseweb","url":"https://kb.leaseweb.com/kb/kubernetes/using-velero-for-backup-and-restoration-of-your-managed-kubernetes/"},{"title":"Velero File System Backup Performance Guide","url":"https://velero.io/docs/v1.10/performance-guidance/"},{"title":"Backing Up and Restoring TKG Cluster Workloads","url":"https://techdocs.broadcom.com/us/en/vmware-cis/vsphere/vsphere-supervisor/8-0/using-tkg-service-with-vsphere-supervisor/backup-and-restore-workloads-using-the-velero-plugin-for-vsphere.html"},{"title":"Velero in Depth — Kubernetes Backup & Disaster Recovery — Babulal Tamang","url":"https://babulal.com.np/blogs/velero-in-depth.html"}]
Based on the search results, here's comprehensive information about Velero Kubernetes backup with persistent volume backup using restic, kopia, and node-agent:
## Overview of File System Backup in Velero
Velero supports backing up and restoring Kubernetes volumes attached to pods from the file system of the volumes, called File System Backup (FSB), using modules from free open-source backup tools restic and kopia.
## Node Agent Architecture
Velero Node Agent is a Kubernetes daemonset that hosts FSB modules, i.e., restic, kopia uploader & repository. To install Node Agent, use the --use-node-agent flag in the velero install command.
## Current State of Restic and Kopia Support
Kopia is the default uploader in current Velero releases; Restic is in the deprecation process. Restic is deprecated in Velero 1.15+. In current Velero releases, the Restic uploader is deprecated and disabled for new backups, so use the default Kopia uploader for new installations.
## Backup Methods Supported
Velero supports two approaches for persistent volume data: the first uses CSI volume snapshots if your storage provider supports them; the second uses the node-agent (formerly Restic, now Kopia-based) to copy file-level data from mounted volumes to object storage, which works with any storage backend, including NFS and local-path-provisioner.
## Performance Characteristics
For backup large amounts of data, Kopia uploader is both less time-consuming and less resource usage, with Kopia uploader showing less time consumption when backup compared to the same specification resources, with performance being better if choosing Kopia uploader for scenarios in backup large amounts of data or massive small files.
## Installation and Configuration
Velero's File System Backup feature provides file-level backups directly from mounted volumes, creating portable backups that work across any supported object storage backend, offering broad compatibility at the cost of slightly longer backup times, and uses the node-agent DaemonSet and the Kopia uploader by default to create encrypted, deduplicated backups.
REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.This is a productive direction, and your instinct is right: k8s has a mature, ingress-like story here. But there's a twist that actually makes the whole design cleaner than the ingress analogy.
Two dominant open-source options, both of which back up PV file data to S3-compatible object storage with client-side encryption:
Schedule/Backup/Restore/Prune/Check). Lighter, per-namespace, focused on PVC data + app-consistent dumps rather than whole-cluster DR.This is the part that's better than ingress. With ingress, the Docker and k8s implementations are genuinely different mechanisms (nginx-proxy vs ingress-nginx) that merely produce the same effect. Here, Velero (kopia) and K8up (restic) use the exact same backup engines I proposed for the Docker container. So you're not maintaining two unrelated stories — you're standardizing on one repository format (restic or kopia) and emitting different config for each target:
Because it's one repo format, a backup written on one target is in principle restorable on the other — same encryption, same snapshots. That's a strong property to design toward.
For a per-stack, per-volume, app-consistent tool, K8up maps almost 1:1 onto what you already have:
Schedule per namespace backs up all PVCs in it. Minimal generated config.k8up.io/backupcommand, e.g. pg_dump) are literally the annotation-driven quiesce mechanism I sketched — it already exists. Your @stack backup-* annotations would translate directly into these on k8s, and into pre/post hooks for the restic container on Docker.Velero is the better fit only if you later want whole-namespace disaster recovery (resources + data together), which is a different feature than "back up service data."
ingress-nginx + cert-manager are effectively assumed present in a cluster. Velero/K8up are not pre-installed — they need operator deployment, CRDs, and object-storage/secret config. So "k8s is already covered, just build Docker" is almost right but not quite: on k8s you still have a generation job — emit a K8up Schedule + backupcommand annotations per deployment (exactly parallel to how deploy injects ingress annotations), plus a way to ensure/declare the operator dependency. It's "generate CRs," not "build a container" — much less work than the Docker side, but not zero.
Schedule + backupcommand annotations from @stack backup-*). Treat K8up as a declared dependency the way ingress treats cert-manager.:ro volume mounts + hooks (the codegen pass we identified).Restore CR into freshly-created PVCs (which, conveniently, also sidesteps the RWO problem since nothing else is attached yet).So the symmetry you were hoping for does exist — just at the engine layer rather than the "k8s already does it for free" layer. The stack tool's job is identical on both sides: enumerate volumes, carry the quiesce annotations, emit target-appropriate config.
Want me to capture this as the design doc now, structured as Docker-container vs K8up-CR with a shared restic/kopia repo contract? That'd be the natural place to lock in the engine choice and the annotation schema.
Sources: - Velero Docs — File System Backup - K8up — Rawkode Academy - Kubernetes Data-Only Backup with K8up - Backups for K8s and Beyond — Jacob Colvin
Ok this approach sounds promising. Do we need to build all of the docker-case solution or do you think we could re-use parts of k8sup?
Good question to pin down before committing. The key issue is that K8up is two distinct pieces — an operator and a restic wrapper — and only one of them could conceivably be reused outside k8s. Let me verify what's actually separable.
{
"query": "K8up architecture restic wrapper binary wrestic standalone \"k8up restic\" command operator separate"
}{
"query": "offen docker-volume-backup restic backend encryption pre post hooks S3 features"
}Web search results for query: "K8up architecture restic wrapper binary wrestic standalone "k8up restic" command operator separate"
Links: [{"title":"GitHub - k8up-io/wrestic: Restic Backup Kubernetes and OpenShift Wrapper - Part of K8up","url":"https://github.com/k8up-io/wrestic"},{"title":"Unlocking the Power of Kubernetes Backups with K8up - Engineering Blog","url":"https://engineering.01cloud.com/2024/06/13/unlocking-the-power-of-kubernetes-backups-with-k8up/"},{"title":"K8up Architecture :: K8up Kubernetes Backup Operator Documentation","url":"https://docs.k8up.io/k8up/2.15/explanations/architecture.html"},{"title":"GitHub - k8up-io/k8up: Kubernetes and OpenShift Backup Operator · GitHub","url":"https://github.com/k8up-io/k8up"},{"title":"k8up 2.0.5 · helm/appuio","url":"https://artifacthub.io/packages/helm/appuio/k8up"},{"title":"Restic Configuration Reference :: K8up Kubernetes Backup Operator Documentation","url":"https://docs.k8up.io/k8up/2.10/references/restic-config-reference.html"},{"title":"wrestic/README.md at master · k8up-io/wrestic","url":"https://github.com/k8up-io/wrestic/blob/master/README.md"},{"title":"Unifi CLI of backup and restic modules of k8up · Issue #477 · k8up-io/k8up","url":"https://github.com/k8up-io/k8up/issues/477"},{"title":"k8up 4.9.0 · k8up/k8up","url":"https://artifacthub.io/packages/helm/k8up/k8up"},{"title":"Backups for K8s and Beyond :: Jacob Colvin — My Website","url":"https://jacobcolvin.com/posts/2023/01/backups-for-k8s-and-beyond/"}]
Based on the search results, here's information about K8up's architecture and the relationship between the operator and restic wrapper binary:
## K8up Architecture Overview
K8up is a Kubernetes backup operator that operates with two main components:
### The Operator Module
The operator module is the part that runs constantly within K8s and contains the various reconciliation loops. K8up operator is deployed as a cluster wide Kubernetes Operator that is responsible for processing the Backup resources and to satisfy the defined Schedule resources, and when it's time to do a backup, the operator scans the namespace for matching PVCs and creates a backup job in the corresponding namespace.
### The Restic Module (k8up restic)
The restic module is the interface to the restic binary and is invoked whenever a Backup or Restore (or similar) custom resource is instantiated, and once its job is done, the process ends. The Job that runs k8up restic contains the logic to perform the requested actions using the underlying restic binary, uses the same Docker image as the k8up operator, and while k8up operator is usually running for a long time, k8up restic will do the job it is tasked to do and then exit.
## Wrestic Integration
Wrestic has been merged into K8up 2.0 and is available as k8up restic command. Wrestic is not needed anymore in K8up v2. The standalone wrestic binary was a restic wrapper that previously served as a separate component but has now been integrated directly into K8up's architecture.
REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.Web search results for query: "offen docker-volume-backup restic backend encryption pre post hooks S3 features"
Links: [{"title":"GitHub - offen/docker-volume-backup: Backup Docker volumes locally or to any S3, WebDAV, Azure Blob Storage, Dropbox, Google Drive or SSH compatible storage · GitHub","url":"https://github.com/offen/docker-volume-backup"},{"title":"How to Back Up and Restore Docker Volumes","url":"https://oneuptime.com/blog/post/2026-01-06-docker-volume-backup-restore/view"},{"title":"Home | docker-volume-backup","url":"https://offen.github.io/docker-volume-backup/"},{"title":"Restic Docker Volume Backup: Compose + systemd (2026)","url":"https://servercrate.net/restic-docker-backup/"},{"title":"GitHub - lobaro/restic-backup-docker: A docker container to automate backups with restic · GitHub","url":"https://github.com/lobaro/restic-backup-docker"},{"title":"offen/docker-volume-backup - Docker Image","url":"https://hub.docker.com/r/offen/docker-volume-backup"},{"title":"Recipes | docker-volume-backup","url":"https://offen.github.io/docker-volume-backup/recipes/"},{"title":"How to Back Up Docker and Kubernetes to S3 Storage: Complete Guide (2025) | DanubeData | DanubeData","url":"https://danubedata.ro/blog/backup-docker-kubernetes-s3-storage-guide-2025"},{"title":"Configuration Reference | docker-volume-backup","url":"https://offen.github.io/docker-volume-backup/reference/"},{"title":"Automate backing up your Docker volumes - DEV Community","url":"https://dev.to/offen-software/automate-backing-up-your-docker-volumes-3gdk"}]
I've found comprehensive information about the offen/docker-volume-backup project and its features. Here's what the search results reveal:
## Key Features
**Storage Backends:**
offen/docker-volume-backup handles recurring or one-off backups of Docker volumes to a local directory, any S3, WebDAV, Azure Blob Storage, Dropbox, Google Drive or SSH compatible storage (or any combination thereof).
**Encryption:**
It supports encrypting your backups using GPG and sending notifications for (failed) backup runs.
**S3 Features:**
The image adds the option to rotate away old backups through the same command so this functionality can also be offered for non-AWS storage backends like MinIO. Local copies of backups can also be pruned once they reach a certain age.
## Pre/Post Hooks
While the search results mention that other docker-volume-backup implementations (like lobaro/restic-backup-docker) support pre and post hooks for scripting, these can include running scripts post-backup.sh after every backup and post-check.sh after every check, the offen-specific results did not explicitly detail pre/post hook functionality in their documentation excerpts.
## Restic Backend
The search results show that while restic is mentioned in related Docker backup solutions, restic performs backup with identifying tags and can apply retention policy, verify repository integrity, but there isn't explicit evidence that offen/docker-volume-backup uses restic as a backend format in these search results.
For complete documentation on pre/post hooks and restic backend support specifically for offen/docker-volume-backup, you should visit their official documentation at https://offen.github.io/docker-volume-backup/.
REMINDER: You MUST include the sources above in your response to the user using markdown hyperlinks.Short answer: the reusable thing is restic (the engine and repo format), not K8up's code. K8up's two components split cleanly along the line you'd care about, and only the wrong half is portable.
K8up is two pieces (per their architecture docs):
Schedule/Backup/Restore CRDs, scanning namespaces for PVCs, creating Jobs, exec'ing pod backupcommand annotations. This is all the scheduling/orchestration/ergonomics value, and it is hard-bound to the k8s API. Zero reuse for Docker.k8up restic (the old wrestic, merged into the binary in v2) — a short-lived restic wrapper that runs in a Job, does one backup/restore/prune/check, reports status back, exits.You could in theory run the k8up restic binary in a Docker container, but it's the wrong bet: its standalone use is undocumented, and it still carries k8s assumptions (reporting status to CRs, annotation-driven command discovery, metrics the operator expects). You'd be depending on an unsupported mode of someone else's binary — to wrap restic, which you can invoke yourself in one line. The juice isn't worth it.
This is the leverage, and it's real. K8up writes standard restic repositories. If your Docker container also runs plain restic, then:
restic from a laptop if both your tooling and K8up are unavailable.So you get the "one engine, two targets" symmetry without sharing a line of implementation between the Docker and k8s paths. That's the right kind of coupling — at the data-format layer, not the code layer.
For the Docker side, adopt an existing restic-cron image rather than writing the cron/hook/retention plumbing:
mazzolino/restic or lobaro/restic-backup-docker — restic + cron + pre/post hooks + retention. These preserve the restic-repo cross-compat property.offen/docker-volume-backup is more feature-rich (many backends, notifications, pruning) but its default format is tar + GPG, not restic — which would break the cross-restore symmetry with K8up. Only use it if you abandon the shared-format goal.I'd go restic-native (mazzolino/lobaro) precisely to keep the K8up interop.
Whichever images you pick, the stack tool's job is the same and unavoidable: enumerate the merged deployment's volumes, carry the @stack backup-* annotations, and emit target-appropriate config — compose :ro mounts + restic env on Docker; Schedule + backupcommand annotations on k8s. That glue is the actual feature work, and it's a few hundred lines, not a backup engine.
Since your k8s model is deliberately constrained (single node, or affinity-colocated data), you could run the same restic-cron container on k8s as a CronJob/sidecar instead of adopting K8up. Trade-off:
backupcommand quiesce convention, RWO-on-same-node mounting, prune/check scheduling.Given that K8up also hits the RWO-while-running constraint (it mounts the PVC in a Job; fine on a single node, which is your model), and that its annotation/CR ergonomics map so neatly onto your namespace-per-deployment design, I'd still lean K8up on k8s + a restic-cron container on Docker, united by the restic repo format — but the single-container-everywhere route is a legitimate "less to depend on" alternative if you'd rather own the whole thing.
Bottom line: build ~none of the backup engine, reuse restic's format as the contract, reuse an off-the-shelf restic image for the Docker container, and write only the volume-enumeration/annotation/config-injection glue — which you'd write either way.
Want me to put the design doc together now, with the engine choice (restic) and this reuse boundary baked in?
Sources:
- K8up Architecture
- k8up-io/wrestic (now k8up restic)
- mazzolino/restic concept
- offen/docker-volume-backup
No, I think it makes sense to go with the grain in the k8s case. By the way we already assume that certain k8s-hosted components are present on the cluster we deploy to, so also assuming k8up has been deployed is ok. We do have an unfinished tool called "cluster" that is intended to provide a batteries-included solution for checking and if necessary fixing missing k8s components prior to use by "stack".
That settles the architecture nicely, and the "cluster" tool is exactly the right home for the K8up dependency. Let me confirm how the existing assumed-components (cert-manager, ingress-nginx) are handled today so K8up slots in the same way.
grep -rin "cert-manager\|certmanager\|ingress-nginx\|clusterissuer\|cluster-issuer" src/ --include=*.py -l; echo "=== refs to a 'cluster' tool/command ==="; grep -rin "cluster" src/stack/main.py | head; echo "=== sibling machine repo (cluster tool?) ==="; ls /home/david/projects/bpi/machine 2>/dev/null
src/stack/constants.py src/stack/init/init.py src/stack/deploy/k8s/deploy_k8s.py src/stack/deploy/spec.py src/stack/deploy/k8s/helpers.py src/stack/deploy/k8s/cluster_info.py src/stack/deploy/deployment_create.py === refs to a 'cluster' tool/command === === sibling machine repo (cluster tool?) === CLAUDE.md LICENSE MANIFEST.in Makefile README.md machine pyproject.toml sh tests uv.lock
grep -rin "cert-manager\|clusterissuer\|cluster-issuer\|ingress-nginx\|nginx" src/stack/deploy/k8s/cluster_info.py src/stack/deploy/k8s/deploy_k8s.py | head -25
src/stack/deploy/k8s/cluster_info.py:160: spec = client.V1IngressSpec(tls=tls, rules=rules, ingress_class_name="nginx") src/stack/deploy/k8s/cluster_info.py:163: "kubernetes.io/ingress.class": "nginx", src/stack/deploy/k8s/cluster_info.py:164: "nginx.ingress.kubernetes.io/rewrite-target": "/$2", src/stack/deploy/k8s/cluster_info.py:165: "nginx.ingress.kubernetes.io/use-regex": "true", src/stack/deploy/k8s/cluster_info.py:169: ingress_annotations["cert-manager.io/cluster-issuer"] = http_proxy_info.get( src/stack/deploy/k8s/deploy_k8s.py:187: group="cert-manager.io", src/stack/deploy/k8s/deploy_k8s.py:385: group="cert-manager.io",
That confirms the precedent precisely — and it's a clean fit. Stack today does not install ingress-nginx or cert-manager; it just emits resources that reference them:
ingress_class_name="nginx" and nginx.ingress.kubernetes.io/* annotations (cluster_info.py:160-165)cert-manager.io/cluster-issuer annotation, and it reads cert-manager CRs at deploy time (deploy_k8s.py:187,385)So the assume-present contract is already the established pattern, and K8up slots in identically: stack emits K8up CRs (Schedule + k8up.io/backupcommand annotations) that reference an operator it presumes is running; it never installs or manages K8up itself. No new architectural concept needed — it's the third member of the set {ingress-nginx, cert-manager, K8up}.
That gives a clean separation of concerns:
stack = data-plane config generation. Enumerate volumes, carry @stack backup-* annotations, emit K8up CRs (k8s) or restic-container config (Docker). Format contract = the restic repo.cluster = control-plane prerequisites. Owns "is K8up's operator deployed, are its CRDs registered, is it healthy" — the same check-and-fix it would do for cert-manager/ingress-nginx. K8up brings its own CRDs (Schedule, Backup, Restore, Prune, Check), so the readiness probe is concrete: CRDs registered + operator Deployment ready in its namespace.A nice consequence: nothing about backup needs to be wired into stack's k8s install path. If cluster hasn't provisioned K8up, a backup-enabled deployment fails the same recognizable way an ingress deployment fails today when cert-manager is missing — consistent operator experience, and you can later add a friendly preflight check that points at cluster.
So the full picture is now settled:
| Concern | Docker | k8s |
|---|---|---|
| Engine / format | restic (off-the-shelf restic-cron image) | restic (via K8up) |
| Scheduling | cron in the backup container | K8up Schedule CR |
| Quiesce | pre/post hooks from @stack backup-* |
k8up.io/backupcommand from @stack backup-* |
| Config generation | stack deploy-time injection |
stack emits K8up CRs |
| Operator/prereqs | the mixed-in backup stack | cluster ensures K8up present |
| Restore | start-stripped → restic restore → start full | K8up Restore CR into fresh PVCs |
Shall I write this up as docs/backup.md now (mirroring docs/ingress.md), locking in restic as the engine, the @stack backup-* annotation schema, and this Docker/K8up/cluster split? That's the artifact to review before any code.