Claude Code transcript - page 2/8

# Backup & Restore

> **Status: design proposal — not yet implemented.** This document describes the intended design for
> backing up and restoring service data. It is written to be reviewed and refined before any code is
> written. Where it describes commands or behaviour, read those as *proposed*.

Stacks keep their software components and configuration under revision control in git. Once a stack is
running, however, it accumulates **persistent data** in mounted volumes that git does not track. This
document describes how that data is backed up to object storage (S3) and restored from a previous epoch.

The approach mirrors the [ingress](./ingress.md) design: a small amount of generated configuration wires
a backup capability into any stack, transparently across both deployment targets (Docker Compose and
Kubernetes), while leaning on a mature, off-the-shelf engine to do the actual work.

## Engine: restic

All backups — on both targets — are stored as a [restic](https://restic.net) repository. restic is the
contract, not an implementation detail:

- **Client-side encryption is mandatory** (AES-256). The payload is encrypted *before* upload, so the
  object store never sees plaintext. This is what makes backing up to commodity object storage acceptable.
- **Content-addressed dedup + incremental snapshots.** A daily backup of a mostly-static volume costs
  almost nothing.
- **Snapshots and retention policies** give point-in-time restore (your "previous epoch").
- **Native S3 backend** (and any S3-compatible store: MinIO, Wasabi, etc.).

Standardising on the restic repository format means a backup written on the Docker target is restorable on
the Kubernetes target and vice-versa, and that in a pinch an operator can restore with the bare `restic`
CLI from outside the deployment entirely.

The two targets differ only in *what runs restic and how it is scheduled*:

| Concern            | Docker                                            | Kubernetes                                   |
| ------------------ | ------------------------------------------------- | -------------------------------------------- |
| Engine / format    | restic (off-the-shelf restic container image)     | restic (via **K8up**)                        |
| Scheduling         | cron in the backup container                      | K8up `Schedule` resource                     |
| Quiesce / hooks    | pre/post hooks in the backup container            | `k8up.io/backupcommand` pod annotation       |
| Config generation  | `stack` injects config at deploy time             | `stack` emits K8up resources                 |
| Prerequisites      | the mixed-in `backup` stack                       | the `cluster` tool ensures K8up is present   |
| Restore            | start-stripped → restic restore → start full      | K8up `Restore` into freshly-created PVCs     |

## Annotations

Which data gets backed up — and how it is made consistent — is described with comment-based annotations in
`composefile.yml`, exactly as HTTP routes are for ingress.

By default, **all read-write named volumes** in a deployment are backed up (file-level). Annotations are
used to *exclude* volumes or to add an *application-consistent logical dump* for stateful services such as
databases.

```yaml
services:
  db:
    image: bozemanpass/todo-db:stack
    volumes:
      - "pgdata:/var/lib/postgresql/data"   # @stack backup-exclude
    # Prefer a consistent logical dump over a file-level copy of a live database.
    # @stack backup-command pg_dump -U postgres -d todos
    # @stack backup-file-extension sql

  backend:
    image: bozemanpass/todo-backend:stack
    volumes:
      - "uploads:/app/uploads"              # backed up by default (file-level)

volumes:
  pgdata:
  uploads:
```

| Annotation                          | Applies to     | Meaning                                                                 |
| ----------------------------------- | -------------- | ----------------------------------------------------------------------- |
| `@stack backup-exclude`             | a volume mount | Do not include this volume in the file-level backup.                    |
| `@stack backup-command <cmd>`       | a service      | Run `<cmd>` in the container and capture its stdout into the backup.    |
| `@stack backup-file-extension <ext>`| a service      | Name the captured `backup-command` output with this extension.          |

These map **one-to-one** onto K8up's pod annotations on the Kubernetes target (`k8up.io/backupcommand`,
`k8up.io/file-extension`, and PVC exclusion), and onto pre-backup hooks in the restic container on the
Docker target. The same annotations drive both.

### Why a logical dump matters

The ingress analogy is reassuring but slightly misleading: ingress is *stateless*, whereas backup is
*deeply stateful*. A file-level copy of a database's data directory **while it is being written** can
produce a torn, unrestorable snapshot. For such services, prefer `@stack backup-command` (e.g. `pg_dump`)
and `@stack backup-exclude` the underlying data volume.

## Enabling backup

As with ingress, the backup capability is **mixed in** to a deployment. The connection details for the
object store and the encryption key are supplied as configuration, the same way the ACME/Let's Encrypt
settings are supplied to the ingress stack.

```bash
# Fetch the backup stack.
$ stack fetch repo bozemanpass/backup-stack

# Configure the object store + encryption key for this deployment.
$ stack init --stack backup \
  --output ~/specs/backup.yml \
  --config BACKUP_S3_ENDPOINT=s3.us-west-2.amazonaws.com \
  --config BACKUP_S3_BUCKET=my-stack-backups \
  --config AWS_ACCESS_KEY_ID=AKIA... \
  --config AWS_SECRET_ACCESS_KEY=... \
  --config RESTIC_PASSWORD=... \
  --config BACKUP_SCHEDULE="0 3 * * *" \
  --config BACKUP_RETENTION="--keep-daily 7 --keep-weekly 4 --keep-monthly 6"

# Init the application stack as usual.
$ stack fetch repo bozemanpass/example-todo-list
$ stack prepare --stack todo
$ stack init --stack todo --output ~/specs/todo.yml

# "Mix in" the backup stack at deployment time.
$ stack deploy \
  --spec-file ~/specs/backup.yml \
  --spec-file ~/specs/todo.yml \
  --deployment-dir ~/deployments/todo

$ stack manage --dir ~/deployments/todo start
```

> The encryption key and object-store credentials are **secrets**. They live in the deployment's generated
> configuration (and as a Kubernetes `Secret`), never in a spec under git.

## How Docker backup works

Like ingress, the stack tool does **not** write the backup engine's job logic by hand. It generates
configuration for an existing restic container image and lets that image do the work. The backup service
cannot be a purely *static* mix-in, because at the time its image is built it does not know which volumes
the application stack will contribute — and naming them statically would collide with the merge step,
which requires unique volume names across mixed-in specs.

Instead, the volume wiring is **injected at deploy time**, precisely parallel to the way `deploy` injects
`VIRTUAL_HOST_MULTIPORTS` environment variables into matching services for ingress:

1. **Annotations to spec** — during `stack init`, the `@stack backup-*` annotations are parsed out of
   `composefile.yml` into a `backup` section of the output spec.

2. **Spec to backup container** — during `stack deploy`, the tool reads the merged deployment's volumes
   (`Spec.get_volumes()`) and:
   - mounts every backed-up volume **read-only** into the backup container
     (`- <volume>:/backup/<volume>:ro`), and
   - injects the schedule, retention, object-store, and per-service `backup-command` hook settings as
     environment / config for the container.

3. **The backup container runs restic on a schedule** — on each cron tick it runs any configured
   pre-backup hooks (the logical dumps) and then `restic backup` of the mounted volume tree, pruning
   per the retention policy.

Because every named volume on the Docker target is already realised as a bind mount under
`<deployment-dir>/data/<volume-name>/`, the set of paths to back up is fully deterministic.

## Kubernetes

On Kubernetes the work is delegated to [K8up](https://k8up.io), a restic-based backup operator. This
follows the same assume-present contract the stack tool already uses for `ingress-nginx` and
`cert-manager`: `stack` does not install K8up; it only emits resources that *reference* it.

- During `stack deploy`, the tool emits a K8up `Schedule` for the deployment's namespace (one namespace
  per deployment), so all of its PVCs are backed up, together with `k8up.io/backupcommand` /
  `k8up.io/file-extension` annotations derived from the `@stack backup-*` annotations.
- K8up writes a **standard restic repository** to the same object store, with the same encryption — so the
  repositories are interchangeable with those produced on the Docker target.

K8up itself is provisioned by the **`cluster`** tool (the batteries-included checker/fixer for required
cluster components), exactly as `cluster` is responsible for `cert-manager` and `ingress-nginx`. A
backup-enabled deployment fails recognisably — the same way an ingress deployment fails today when
`cert-manager` is absent — if K8up has not been provisioned. The readiness probe is concrete: K8up's CRDs
registered and its operator `Deployment` healthy.

> Because the deployment uses a single node (or node affinity to co-locate data), K8up's backup `Job` can
> mount the `ReadWriteOnce` PVCs alongside the running application pod. Cross-node volume access is
> explicitly out of scope.

## Restore

Restore is deliberately modelled as a distinct mode rather than a live operation, which neatly sidesteps
the problem of two consumers mounting the same volume at once. At restore time **nothing else holds the
volumes**, so even `ReadWriteOnce` PVCs can be attached by the restore job.

The flow:

1. The full stack is stopped (if running).
2. The volumes are (re)created empty.
3. A **backup-only** variant of the deployment is brought up, running a restore of the chosen snapshot:
   - **Docker:** bring up only the backup service with a restore command, restoring into the now-empty
     bind-mounted volumes.
   - **Kubernetes:** create a K8up `Restore` resource targeting the freshly-created PVCs.
4. The backup-only variant is torn down once the restore completes.
5. The full stack is started; its volumes now contain the restored data.

This is driven by a single command (see below) so the operator does not orchestrate the steps by hand.

## Command structure

Backup *configuration* is established at `init` / `deploy` time (above). Backup *operations* act on an
existing deployment, so they live under `stack manage --dir <dir>`, alongside `start`, `stop`, `ps`, and
`logs` — **not** under `stack deploy`, which only *creates* a deployment.

```
stack manage --dir <dir> backup <subcommand>
```

| Command                                                        | Description                                                        |
| -------------------------------------------------------------- | ------------------------------------------------------------------ |
| `stack manage --dir <dir> backup now`                          | Run a backup immediately, outside the schedule.                    |
| `stack manage --dir <dir> backup status`                       | Show the result of the last run and repository health.             |
| `stack manage --dir <dir> backup list`                         | List snapshots (the available epochs) with timestamps and tags.    |
| `stack manage --dir <dir> backup restore [--snapshot <id>]`    | Orchestrate the full stop → restore → restart flow. Defaults to the latest snapshot; `--volume <name>` restores a single volume. `--and-start` restarts the full stack on completion. |
| `stack manage --dir <dir> backup prune`                        | Apply the retention policy (`forget` + `prune`).                   |
| `stack manage --dir <dir> backup check`                        | Verify repository integrity.                                       |

`backup` is a Click sub-group of the existing `manage` group, so it inherits `--dir` and the deployment
context. Internally each subcommand dispatches to the active target: on Docker it `exec`s restic in the
backup container (or runs a one-off restore deployment); on Kubernetes it creates/reads the corresponding
K8up resources.

> The earlier sketch of `stack deploy backup status` is intentionally **not** the chosen shape: `deploy`
> creates a deployment from specs and exits, whereas backup status/restore/list are operations *on an
> existing* deployment — which is precisely what `manage` is for.

## Open questions

- **Engine choice.** restic is proposed for its simplicity and K8up compatibility. Kopia (Velero's default,
  better at very large volumes) is the main alternative; both produce encrypted, deduplicated repositories,
  but the repository formats are *not* mutually compatible, so this choice must be made once, up front.
- **Off-the-shelf Docker image.** Candidates that preserve the restic-repo format include
  `mazzolino/restic` and `lobaro/restic-backup-docker`. (`offen/docker-volume-backup` is feature-rich but
  defaults to tar+GPG, which would break cross-target compatibility.)
- **Secret delivery.** Exactly how the restic password and object-store credentials flow from
  configuration into the Docker container env and the Kubernetes `Secret`.
- **Monitoring.** Surfacing backup success/failure (a healthcheck or status that `backup status` can read)
  so that a silently failing backup is not mistaken for a working one.
- **New repository.** The Docker backup stack lives in its own repo (e.g. `bozemanpass/backup-stack`),
  mirroring `bozemanpass/docker-ingress-stack`.

# Backup & Restore

> **Status: design proposal — not yet implemented.** This document describes the intended design for
> backing up and restoring service data. It is written to be reviewed and refined before any code is
> written. Where it describes commands or behaviour, read those as *proposed*.

Stacks keep their software components and configuration under revision control in git. Once a stack is
running, however, it accumulates **persistent data** in mounted volumes that git does not track. This
document describes how that data is backed up to object storage (S3) and restored from a previous epoch.

## Design goal: "backup my stuff", and nothing more

The feature is designed to be as transparent as possible. In the common case the person deploying a stack
provides **no backup-specific information at all** — not which volumes to back up, not where to send them,
not on what schedule. Backup is configured once at the environment/profile level and then applies to every
deployment automatically.

This is achieved by deriving everything possible from what the stack tool already knows or can source
ambiently:

- **What to back up** is derived from the deployment itself: *all* read-write named volumes are backed up
  by default. The tool already enumerates them (`Spec.get_volumes()`).
- **Where to back up, with which credentials, on what schedule** is sourced from the environment using the
  existing configuration precedence (see [Configuration](#configuration)).

Annotations exist only to *refine* this default, and are written by the **author** of a stack, never by
the person deploying it.

## Engine: restic

All backups — on both targets — are stored as a [restic](https://restic.net) repository. restic is the
contract, not an implementation detail:

- **Client-side encryption is mandatory** (AES-256). The payload is encrypted *before* upload, so the
  object store never sees plaintext. This is what makes backing up to commodity object storage acceptable.
- **Content-addressed dedup + incremental snapshots.** A daily backup of a mostly-static volume costs
  almost nothing.
- **Snapshots and retention policies** give point-in-time restore (your "previous epoch").
- **Native S3 backend** (and any S3-compatible store: MinIO, Wasabi, etc.).

Standardising on the restic repository format means a backup written on the Docker target is restorable on
the Kubernetes target and vice-versa, and that in a pinch an operator can restore with the bare `restic`
CLI from outside the deployment entirely.

The two targets differ only in *what runs restic and how it is scheduled*:

| Concern            | Docker                                            | Kubernetes                                   |
| ------------------ | ------------------------------------------------- | -------------------------------------------- |
| Engine / format    | restic (off-the-shelf restic container image)     | restic (via **K8up**)                        |
| Scheduling         | cron in the backup container                      | K8up `Schedule` resource                     |
| Quiesce / hooks    | pre/post hooks in the backup container            | `k8up.io/backupcommand` pod annotation       |
| Config generation  | `stack` injects config at deploy time             | `stack` emits K8up resources                 |
| Prerequisites      | the auto-injected backup container                | the `cluster` tool ensures K8up is present   |
| Restore            | start-stripped → restic restore → start full      | K8up `Restore` into freshly-created PVCs     |

## Configuration

Backup settings are resolved with the standard stack configuration precedence
(`config/util.py:get_config_setting`): **environment variable → active profile → built-in default**. This
is what lets backup be ambient — set it once in a profile and every deployment under that profile inherits
it, with no per-stack input.

| Setting (profile key / `STACK_…` env var)   | Purpose                                         | Default        |
| ------------------------------------------- | ----------------------------------------------- | -------------- |
| `backup` / `STACK_BACKUP`                   | Master switch — enable backup for deployments.  | `false`        |
| `backup-s3-endpoint`                        | Object store endpoint.                          | —              |
| `backup-s3-bucket`                          | Bucket / repository location.                   | —              |
| `aws-access-key-id`, `aws-secret-access-key`| Object store credentials.                       | —              |
| `restic-password`                           | **Encryption key** (see warning below).         | —              |
| `backup-schedule`                           | Cron schedule.                                  | `0 3 * * *`    |
| `backup-retention`                          | `forget`/`prune` policy.                        | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` |

Typical one-time setup for an environment:

```bash
$ stack config set backup true
$ stack config set backup-s3-endpoint s3.us-west-2.amazonaws.com
$ stack config set backup-s3-bucket my-stack-backups
$ stack config set aws-access-key-id AKIA...
$ stack config set aws-secret-access-key ...
$ stack config set restic-password ...
```

After that, **every** deployment is backed up with no further action:

```bash
$ stack deploy --spec-file ~/specs/todo.yml --deployment-dir ~/deployments/todo
$ stack manage --dir ~/deployments/todo start
# ...the deployment's volumes are now backed up on the configured schedule.
```

> #### ⚠ The encryption key cannot be purely ephemeral
>
> restic cannot decrypt a repository without its password. If `restic-password` is auto-generated and
> lives *only* in an environment that is later lost, the backups become **permanently unrecoverable** — an
> encrypted bucket that can never be read. The password must therefore either be set explicitly by the
> operator, or be auto-generated **and persisted and surfaced for the operator to escrow**. This is the one
> piece of backup configuration that must not be treated as disposable ambient state. Object-store
> credentials, by contrast, can be rotated freely.

## Volume selection (automatic)

By default **all read-write named volumes** in a deployment are backed up, file-level. Read-only mounts and
config maps are skipped. No annotation or flag is required for this — it is derived entirely from the
merged spec.

The only optional refinement is to *exclude* a volume that is a cache, scratch space, or otherwise cheaply
reconstructable, using an annotation in the stack's `composefile.yml`:

```yaml
services:
  backend:
    image: bozemanpass/todo-backend:stack
    volumes:
      - "uploads:/app/uploads"              # backed up by default
      - "cache:/app/cache"                  # @stack backup-exclude
```

Excluding a volume is an **author** decision encoded in the component, not something the deployer supplies.

## Application consistency

This is the one place where "just back up everything" needs care, and it is worth stating plainly: the
ingress analogy is misleading because ingress is *stateless* whereas backup is *deeply stateful*. A
file-level copy of a **live database's** data directory, read file-by-file while the database writes, can
produce a torn, unrestorable snapshot.

For such services the stack author adds a single annotation specifying a logical dump command, whose stdout
is captured into the backup instead of (or alongside) the raw files:

```yaml
services:
  db:
    image: bozemanpass/todo-db:stack
    volumes:
      - "pgdata:/var/lib/postgresql/data"   # @stack backup-exclude
    # @stack backup-command pg_dump -U postgres -d todos
    # @stack backup-file-extension sql
```

Crucially this is **author-time** metadata: whoever packages the database component writes it once, and
every deployer of that stack gets consistent backups for free, having supplied nothing. The annotation maps
one-to-one onto K8up's `k8up.io/backupcommand` / `k8up.io/file-extension` pod annotations on the Kubernetes
target, and onto a pre-backup hook in the restic container on the Docker target.

### Annotation summary

There are only two optional annotations, both author-time, both with safe "just back it up" defaults:

| Annotation                          | Applies to     | Meaning                                                                 |
| ----------------------------------- | -------------- | ----------------------------------------------------------------------- |
| `@stack backup-exclude`             | a volume mount | Do not include this volume in the file-level backup.                    |
| `@stack backup-command <cmd>`       | a service      | Capture `<cmd>`'s stdout into the backup (e.g. a consistent DB dump).   |
| `@stack backup-file-extension <ext>`| a service      | Name the captured `backup-command` output with this extension.         |

A stack that uses none of these is still fully backed up — every read-write volume, file-level.

## How Docker backup works

Like ingress, the stack tool does **not** write the backup engine's job logic by hand. It generates
configuration for an existing restic container image and lets that image do the work. The backup container
cannot be a purely *static* mix-in, because at the time its image is built it does not know which volumes
the application stack will contribute — and naming them statically would collide with the merge step, which
requires unique volume names across mixed-in specs.

Instead, when the `backup` master switch is enabled, the backup container is **injected at deploy time**,
precisely parallel to the way `deploy` injects `VIRTUAL_HOST_MULTIPORTS` environment variables into
matching services for ingress:

1. **Annotations to spec** — during `stack init`, any `@stack backup-*` annotations are parsed out of
   `composefile.yml` into a `backup` section of the output spec.

2. **Spec to backup container** — during `stack deploy`, the tool reads the merged deployment's volumes
   (`Spec.get_volumes()`) and:
   - mounts every non-excluded read-write volume **read-only** into the backup container
     (`- <volume>:/backup/<volume>:ro`), and
   - injects the ambiently-resolved schedule, retention, object-store, and per-service `backup-command`
     hook settings as environment / config for the container.

3. **The backup container runs restic on a schedule** — on each cron tick it runs any configured
   pre-backup hooks (the logical dumps) and then `restic backup` of the mounted volume tree, applying the
   retention policy.

Because every named volume on the Docker target is already realised as a bind mount under
`<deployment-dir>/data/<volume-name>/`, the set of paths to back up is fully deterministic.

## Kubernetes

On Kubernetes the work is delegated to [K8up](https://k8up.io), a restic-based backup operator. This
follows the same assume-present contract the stack tool already uses for `ingress-nginx` and `cert-manager`
(it references an `ingress_class_name="nginx"` and a `cert-manager.io/cluster-issuer` it does not install):
`stack` does not install K8up; it only emits resources that *reference* it.

- During `stack deploy`, the tool emits a K8up `Schedule` for the deployment's namespace (one namespace per
  deployment), so all of its PVCs are backed up, together with `k8up.io/backupcommand` /
  `k8up.io/file-extension` annotations derived from the `@stack backup-*` annotations.
- K8up writes a **standard restic repository** to the same object store, with the same encryption — so the
  repositories are interchangeable with those produced on the Docker target.

K8up itself is provisioned by the **`cluster`** tool (the batteries-included checker/fixer for required
cluster components), exactly as `cluster` is responsible for `cert-manager` and `ingress-nginx`. A
backup-enabled deployment fails recognisably — the same way an ingress deployment fails today when
`cert-manager` is absent — if K8up has not been provisioned. The readiness probe is concrete: K8up's CRDs
registered and its operator `Deployment` healthy.

> Because the deployment uses a single node (or node affinity to co-locate data), K8up's backup `Job` can
> mount the `ReadWriteOnce` PVCs alongside the running application pod. Cross-node volume access is
> explicitly out of scope.

## Restore

Restore is deliberately modelled as a distinct mode rather than a live operation, which neatly sidesteps the
problem of two consumers mounting the same volume at once. At restore time **nothing else holds the
volumes**, so even `ReadWriteOnce` PVCs can be attached by the restore job.

The flow:

1. The full stack is stopped (if running).
2. The volumes are (re)created empty.
3. A **backup-only** variant of the deployment is brought up, running a restore of the chosen snapshot:
   - **Docker:** bring up only the backup container with a restore command, restoring into the now-empty
     bind-mounted volumes.
   - **Kubernetes:** create a K8up `Restore` resource targeting the freshly-created PVCs.
4. The backup-only variant is torn down once the restore completes.
5. The full stack is started; its volumes now contain the restored data.

This is driven by a single command (see below) so the operator does not orchestrate the steps by hand.

## Command structure

Backup *configuration* is ambient (above) and backup runs automatically. The commands below are for
*operating on* an existing deployment — inspecting, triggering off-schedule, and restoring — so they live
under `stack manage --dir <dir>`, alongside `start`, `stop`, `ps`, and `logs`. They are **not** under
`stack deploy`, which only *creates* a deployment and exits.

```
stack manage --dir <dir> backup <subcommand>
```

| Command                                                        | Description                                                        |
| -------------------------------------------------------------- | ------------------------------------------------------------------ |
| `stack manage --dir <dir> backup now`                          | Run a backup immediately, outside the schedule.                    |
| `stack manage --dir <dir> backup status`                       | Show the result of the last run and repository health.             |
| `stack manage --dir <dir> backup list`                         | List snapshots (the available epochs) with timestamps and tags.    |
| `stack manage --dir <dir> backup restore [--snapshot <id>]`    | Orchestrate the full stop → restore → restart flow. Defaults to the latest snapshot; `--volume <name>` restores a single volume; `--and-start` restarts the full stack on completion. |
| `stack manage --dir <dir> backup prune`                        | Apply the retention policy (`forget` + `prune`).                   |
| `stack manage --dir <dir> backup check`                        | Verify repository integrity.                                       |

`backup` is a Click sub-group of the existing `manage` group, so it inherits `--dir` and the deployment
context. Internally each subcommand dispatches to the active target: on Docker it `exec`s restic in the
backup container (or runs a one-off restore deployment); on Kubernetes it creates/reads the corresponding
K8up resources.

> The earlier sketch of `stack deploy backup status` is intentionally **not** the chosen shape: `deploy`
> creates a deployment from specs and exits, whereas backup status/restore/list are operations *on an
> existing* deployment — which is precisely what `manage` is for.

## Open questions

- **Auto-enable vs explicit switch.** This design gates backup on an explicit `backup` master switch
  (profile setting / `STACK_BACKUP`) so behaviour is predictable. The most transparent alternative —
  enabling backup automatically whenever a destination is configured — is rejected as too implicit, but is
  worth revisiting.
- **Encryption-key escrow.** The concrete mechanism for persisting and surfacing an auto-generated
  `restic-password` so it cannot be silently lost (see the warning above).
- **Off-the-shelf Docker image.** Candidates that preserve the restic-repo format include `mazzolino/restic`
  and `lobaro/restic-backup-docker`. (`offen/docker-volume-backup` is feature-rich but defaults to
  tar+GPG, which would break cross-target compatibility.)
- **Monitoring.** Surfacing backup success/failure (a healthcheck or status that `backup status` can read)
  so that a silently failing backup is not mistaken for a working one.
- **New repository.** The Docker backup container lives in its own repo (e.g. `bozemanpass/backup-stack`),
  mirroring `bozemanpass/docker-ingress-stack`.

# Backup — implementation sketch

> **Status: design sketch — not yet implemented.** Companion to [backup.md](./backup.md). This pins the
> design down to concrete functions and insertion points in the existing code, mirroring how ingress is
> implemented, so the work can be reviewed before it is written. Code below is illustrative, not final.

## Scope of this sketch

This covers the **deploy-time injection pass** (the first concrete step) and the two things it depends on:

1. parsing the `@stack backup-*` annotations into the spec (mirrors `Stack.get_http_proxy_targets`);
2. an accessor for that spec section (mirrors `Spec.get_http_proxy`);
3. injecting a backup service into the generated deployment at deploy time (mirrors the
   `VIRTUAL_HOST_MULTIPORTS` injection).

**Out of scope here** (later steps): the `bozemanpass/backup-stack` container image, the
`stack manage … backup` subcommands, and the Kubernetes K8up-resource emission. Those are tracked in
[backup.md](./backup.md).

## The ingress pattern we are mirroring

For reference, ingress works in three touch-points, all of which have a backup analogue:

| Ingress                                                   | Backup analogue                                            |
| --------------------------------------------------------- | --------------------------------------------------------- |
| `Stack.get_http_proxy_targets()` parses port annotations (`stack.py:203`) | `Stack.get_backup_targets()` parses volume/service annotations |
| targets written into `network.http-proxy` spec section    | targets written into a `backup` spec section              |
| `Spec.get_http_proxy()` accessor (`spec.py:140`)          | `Spec.get_backup()` accessor                              |
| inject `VIRTUAL_HOST_MULTIPORTS` env at deploy (`deployment_create.py:550`) | inject a `backup` service + `:ro` mounts at deploy |

## 1. Parse annotations — `deploy/stack.py`

Add a method alongside `get_http_proxy_targets` (`stack.py:203`). Unlike ingress, backup annotations attach
to two different places — a **volume mount line** (`backup-exclude`) and a **service** (`backup-command`,
`backup-file-extension`). Both are read from `ruamel`'s comment attributes (`.ca`), exactly as the ingress
parser reads port-line comments.

```python
# constants.py (new)
backup_key = "backup"
backup_exclude_annotation = "backup-exclude"
backup_command_annotation = "backup-command"
backup_file_extension_annotation = "backup-file-extension"

# deploy/stack.py — new method on Stack
def get_backup_targets(self):
    """Parse @stack backup-* annotations from the stack's composefiles.

    Returns {"exclude": [volume_name, ...],
             "commands": {service_name: {"command": str, "file_extension": str}}}
    """
    exclude = []
    commands = {}
    for pod in self.get_pod_list():
        parsed_pod_file = self.load_pod_file(pod)
        for svc_name, svc in parsed_pod_file.get(constants.services_key, {}).items():
            # Service-level annotations (backup-command / backup-file-extension) live in the
            # comment block attached to the service mapping.
            for ann in _stack_annotations_for(svc):       # small helper over svc.ca
                if constants.backup_command_annotation in ann:
                    commands.setdefault(svc_name, {})["command"] = _annotation_value(ann)
                elif constants.backup_file_extension_annotation in ann:
                    commands.setdefault(svc_name, {})["file_extension"] = _annotation_value(ann)

            # Volume-level annotation (backup-exclude) lives on the mount line, like ports do.
            volumes_section = svc.get(constants.volumes_key, [])
            for i, mount in enumerate(volumes_section):
                comment = _line_comment(volumes_section, i)   # mirrors ports_section.ca.items[i]
                if comment and constants.stack_annotation_marker in comment \
                        and constants.backup_exclude_annotation in comment:
                    exclude.append(str(mount).split(":")[0])
    return {"exclude": exclude, "commands": commands}
```

`_line_comment()` is the same `ports_section.ca.items[i][0].value` access used at `stack.py:213-215`,
factored out so it can be reused for the volumes list.

## 2. Spec section + accessor — `deploy/spec.py`

The parsed targets are written into a top-level `backup` section of the spec during `stack init` (alongside
where `http-proxy` targets are written, `deployment_create.py:285-326`). Add the read accessor next to
`get_http_proxy` (`spec.py:140`):

```python
# deploy/spec.py
def get_backup(self):
    return self.obj.get(constants.backup_key, {})
```

Because `backup` is a plain dict it merges additively across mixed-in specs with no special handling, like
the other spec sections.

## 3. Inject the backup service at deploy time — `deploy/deployment_create.py`

This is the core of the pass. After the per-pod compose files are written (the loop ending at
`deployment_create.py:577`), and only for the `compose` target with the master switch enabled, write **one
synthetic compose file** containing the backup service.

The backup service mounts the deployment's data **read-only**. On Docker every named volume is already a
bind mount under `<deployment-dir>/data/<volume-name>/` (`_fixup_pod_file`, line 77), so the backup service
can bind-mount those host paths directly — sidestepping cross-compose-file named-volume declaration
entirely.

```python
# deploy/deployment_create.py — after the `for pod in pods:` loop (~line 577)

from stack.config.util import get_config_setting

def _maybe_write_backup_compose(parsed_spec, deployment_type, deployment_dir_path, destination_compose_dir):
    if deployment_type != "compose":
        return                                    # k8s path emits K8up resources instead (separate step)
    if not get_config_setting("backup", False):   # explicit master switch (env → profile → default false)
        return

    backup_cfg = parsed_spec.get_backup()
    exclude = set(backup_cfg.get("exclude", []))

    # All read-write named volumes, minus author-excluded ones.
    volumes = parsed_spec.get_volumes()           # {name: host_path_or_None}
    backed_up = [v for v in volumes if v not in exclude]

    mounts = [f"../../data/{v}:/backup/{v}:ro" for v in backed_up]   # relative to the compose dir

    # Per-service consistent-dump hooks (run inside the backup container, reaching services by name
    # over the shared compose network).
    pre_hooks = []
    for svc, c in backup_cfg.get("commands", {}).items():
        ext = c.get("file_extension", "dump")
        pre_hooks.append(f"{svc}:{c['command']}:{ext}")   # parsed by the backup image entrypoint

    backup_service = {
        "image": f"bozemanpass/backup:{deployment_command_context.cluster_context.cluster}",
        "restart": "always",
        "volumes": mounts,
        "env_file": [relative_path_to(constants.config_file_name)],   # brings in S3 creds + restic-password
        "environment": {
            "BACKUP_DEST": get_config_setting("backup-s3-bucket"),
            "BACKUP_S3_ENDPOINT": get_config_setting("backup-s3-endpoint"),
            "BACKUP_SCHEDULE": get_config_setting("backup-schedule", "0 3 * * *"),
            "BACKUP_RETENTION": get_config_setting(
                "backup-retention", "--keep-daily 7 --keep-weekly 4 --keep-monthly 6"),
            "BACKUP_PRE_HOOKS": ";".join(pre_hooks),
        },
    }

    doc = {"services": {"backup": backup_service}}
    out = destination_compose_dir.joinpath(f"{constants.compose_file_prefix}-backup.yml")
    with open(out, "w") as f:
        get_yaml().dump(doc, f)
```

Call it once, right after the pod loop:

```python
    _maybe_write_backup_compose(parsed_spec, deployment_type, deployment_dir_path, destination_compose_dir)
```

Notes:

- **One mount per volume, `:ro`.** This matches the doc and lets restic see a clean per-volume tree. A
  simpler variant is a single `../../data:/backup:ro` mount with restic `--exclude` patterns; either works.
- **`restic-password` and the S3 credentials** arrive via the shared `config.env` (`env_file`), which is
  how `get_config_setting` values already reach containers — no new secret-plumbing path is introduced.
- The backup service joins the deployment's default network automatically, so `backup-command` hooks such
  as `pg_dump -h db …` resolve the `db` service by name.
- The injected service appears in `stack manage … ps` like any other container, which is what the
  forthcoming `backup` subcommands will `exec` into.

## What this deliberately does *not* do yet

- It does not start a restic engine — the `bozemanpass/backup` image (its entrypoint, cron loop, hook
  runner, and restore command) is the next step and lives in its own repo.
- It does not emit anything on the Kubernetes target; that path generates a K8up `Schedule` + annotations
  and is a sibling to this function.
- It does not add the `stack manage … backup` subcommands.

These are intentionally separable: this pass produces a deployment that *declares* its backup intent in a
single, inspectable compose file, which is the foundation the remaining pieces build on.

name: backup
description: "Backup and restore of stack service data (restic to S3 object storage)"
# The backup container image (built from ./containers/backup in this repo).
containers:
  - name: bozemanpass/backup
    path: ./containers/backup
# A single-service pod. When backup is enabled, `stack deploy` injects read-only mounts
# of the application's data volumes into this pod's `backup` service.
pods:
  - name: backup
    path: ./backup

services:
  backup:
    image: bozemanpass/backup:stack
    restart: always
    volumes:
      # The Docker socket lets the backup service run consistency hooks (e.g. pg_dump)
      # *inside* the target service container, the same way the ingress proxy uses it.
      - /var/run/docker.sock:/var/run/docker.sock:ro
      # restic's local cache, to keep incremental backups fast.
      - backup-restic-cache:/root/.cache/restic
      #
      # NOTE: read-only mounts of the application's data volumes are INJECTED here by
      # `stack deploy` when backup is enabled. Each becomes `- <vol>:/backup/<vol>:ro`.
      # See ../stack/docs/backup-implementation.md.

volumes:
  backup-restic-cache:

#!/usr/bin/env bash
# Build bozemanpass/backup
set -euo pipefail

SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

# `stack build` provides the target tag (e.g. bozemanpass/backup:stack, or :<cluster>
# when publishing). Fall back to the local :stack tag for a manual build.
TAG="${STACK_DEFAULT_CONTAINER_IMAGE_TAG:-bozemanpass/backup:stack}"

docker build -t "$TAG" ${build_command_args:-} -f "$SCRIPT_DIR/Containerfile" "$SCRIPT_DIR"

FROM alpine:3.20

# restic     - the backup engine (mandatory client-side encryption, dedup, S3-native)
# docker-cli - to exec consistency hooks (pg_dump, etc.) inside target containers
# bash       - script runtime
# tzdata     - correct cron scheduling in the configured timezone
# ca-certificates - TLS to the object store
RUN apk add --no-cache restic bash docker-cli tzdata ca-certificates

COPY scripts /scripts
RUN chmod +x /scripts/*.sh

WORKDIR /backup

# Default mode is the scheduler. Other modes: backup | restore | prune | check.
ENTRYPOINT ["/scripts/docker-entrypoint.sh"]
CMD ["schedule"]

#!/usr/bin/env bash
# Shared restic environment setup, sourced by the other scripts.
set -euo pipefail

# Build RESTIC_REPOSITORY from the S3 settings if it was not supplied directly.
if [ -z "${RESTIC_REPOSITORY:-}" ]; then
  if [ -n "${BACKUP_S3_ENDPOINT:-}" ] && [ -n "${BACKUP_S3_BUCKET:-}" ]; then
    export RESTIC_REPOSITORY="s3:https://${BACKUP_S3_ENDPOINT}/${BACKUP_S3_BUCKET}"
  else
    echo "backup: no RESTIC_REPOSITORY (or BACKUP_S3_ENDPOINT + BACKUP_S3_BUCKET) configured" >&2
    exit 1
  fi
fi

# The encryption key is mandatory: without it the repository cannot be read back.
if [ -z "${RESTIC_PASSWORD:-}" ] && [ -z "${RESTIC_PASSWORD_FILE:-}" ]; then
  echo "backup: RESTIC_PASSWORD is not set - refusing to run without an encryption key" >&2
  exit 1
fi

# Create the repository on first use (idempotent).
ensure_repo() {
  if ! restic cat config >/dev/null 2>&1; then
    echo "backup: initializing restic repository at ${RESTIC_REPOSITORY}"
    restic init
  fi
}

#!/usr/bin/env bash
set -euo pipefail
source /scripts/lib.sh

mode="${1:-schedule}"
case "$mode" in
  schedule)
    ensure_repo
    schedule="${BACKUP_SCHEDULE:-0 3 * * *}"
    echo "backup: scheduling '${schedule}'  ->  /scripts/backup.sh"
    # Single cron entry; send job output to PID 1's stdout so it shows in container logs.
    echo "${schedule} /scripts/backup.sh >> /proc/1/fd/1 2>&1" > /etc/crontabs/root
    exec crond -f -l 8
    ;;
  backup)  exec /scripts/backup.sh ;;
  restore) shift; exec /scripts/restore.sh "$@" ;;
  prune)   exec /scripts/prune.sh ;;
  check)   ensure_repo; exec restic check ;;
  *)
    echo "backup: unknown mode '$mode' (expected: schedule|backup|restore|prune|check)" >&2
    exit 2
    ;;
esac

#!/usr/bin/env bash
set -euo pipefail
source /scripts/lib.sh
ensure_repo

# 1. Run consistency hooks (logical dumps) into the backup tree.
/scripts/run-hooks.sh

# 2. Back up everything mounted under /backup: the read-only data volumes plus any dumps.
echo "backup: starting restic backup of /backup"
restic backup --host "${STACK_DEPLOYMENT:-stack}" /backup

# 3. Apply the retention policy.
/scripts/prune.sh

echo "backup: complete"

#!/usr/bin/env bash
# Run per-service consistency-dump commands *inside* their containers (via the Docker
# socket) and write the output under /backup/_dumps so restic captures a consistent
# logical backup alongside the file-level volume data.
#
# BACKUP_PRE_HOOKS format:  "service:command:ext;service2:command2:ext2"
#   e.g.  "db:pg_dump -U postgres -d todos:sql"
set -euo pipefail

hooks="${BACKUP_PRE_HOOKS:-}"
[ -z "$hooks" ] && exit 0

dump_dir="/backup/_dumps"
mkdir -p "$dump_dir"

# Resolve the compose project of THIS container, so hooks only exec into sibling services
# in the same deployment.
self_id="$(grep -o -m1 '[0-9a-f]\{64\}' /proc/self/cgroup || true)"
project="$(docker inspect -f '{{ index .Config.Labels "com.docker.compose.project" }}' "$self_id" 2>/dev/null || true)"

IFS=';' read -ra entries <<< "$hooks"
for entry in "${entries[@]}"; do
  [ -z "$entry" ] && continue
  svc="${entry%%:*}"; rest="${entry#*:}"
  cmd="${rest%:*}"; ext="${rest##*:}"

  cid="$(docker ps -q \
      --filter "label=com.docker.compose.project=${project}" \
      --filter "label=com.docker.compose.service=${svc}" | head -n1)"
  if [ -z "$cid" ]; then
    echo "backup: hook target service '${svc}' not found - skipping" >&2
    continue
  fi

  echo "backup: dumping '${svc}' (${cmd})"
  docker exec "$cid" sh -c "$cmd" > "${dump_dir}/${svc}.${ext}"
done

#!/usr/bin/env bash
set -euo pipefail
source /scripts/lib.sh

# Word-splitting of $retention is intentional (the flags are passed through to restic).
retention="${BACKUP_RETENTION:---keep-daily 7 --keep-weekly 4 --keep-monthly 6}"
echo "backup: applying retention (${retention})"
# shellcheck disable=SC2086
restic forget --prune $retention

#!/usr/bin/env bash
# Restore a snapshot into the volume tree.
#
# Usage: restore.sh [snapshot-id]    (default: latest)
#
# Restore is run as a distinct mode: the full application stack is stopped first and the
# data volumes are mounted READ-WRITE under /backup, so restic writes the chosen epoch's
# data back in place. The full stack is then started again. See ../stack/docs/backup.md.
set -euo pipefail
source /scripts/lib.sh
ensure_repo

snapshot="${1:-latest}"
echo "backup: restoring snapshot '${snapshot}' into /backup"
restic restore "$snapshot" --target /

# NOTE: logical dumps land back under /backup/_dumps as files. Replaying them into a live
# database (e.g. pg_restore) is a deliberate follow-up, not yet automated - see the
# "Restore" and open-questions sections of docs/backup.md.
echo "backup: restore complete"

# backup-stack

Backup and restore of persistent service data for the
[`stack`](https://github.com/bozemanpass/stack) tool's Docker deployment mode.

> **Status: initial scaffold — not yet functional.** The container scripts are a starting point and have
> not been run end-to-end. See the design in the `stack` repo: `docs/backup.md` and
> `docs/backup-implementation.md`.

## What this provides

`bozemanpass/backup` — a small Alpine image bundling:

- [restic](https://restic.net) — the backup engine: mandatory client-side encryption, deduplication, and
  native S3 support, so commodity object storage can be used safely;
- a cron scheduler;
- the Docker CLI — used to run application-consistency hooks (e.g. `pg_dump`) *inside* the target service
  container, the same way the ingress proxy uses the Docker socket.

On Kubernetes the equivalent role is played by [K8up](https://k8up.io) (also restic-based), so the two
targets produce interchangeable repositories. This repo covers the Docker case only.

## Layout

| Path | Purpose |
|------|---------|
| `stacks/backup/stack.yml` | Stack definition — declares the container and the pod. |
| `backup/composefile.yml` | The canonical `backup` service. `stack deploy` injects read-only mounts of the application's data volumes here when backup is enabled. |
| `containers/backup/` | The `bozemanpass/backup` image: `Containerfile`, `build.sh`, and `scripts/`. |

## Container modes

The image entrypoint takes a mode argument (default `schedule`):

| Mode | Action |
|------|--------|
| `schedule` | Install a cron entry (`BACKUP_SCHEDULE`) that runs `backup` periodically. |
| `backup` | Run hooks, then `restic backup` of `/backup`, then apply retention. |
| `restore [snapshot]` | Restore a snapshot into the (rw-mounted) volumes. Default `latest`. |
| `prune` | Apply the retention policy (`restic forget --prune`). |
| `check` | Verify repository integrity. |

## Configuration

Supplied by `stack` from the deployment environment (see `docs/backup.md` for the profile keys):

| Variable | Purpose |
|----------|---------|
| `BACKUP_S3_ENDPOINT`, `BACKUP_S3_BUCKET` | Object store location (or set `RESTIC_REPOSITORY` directly). |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | Object store credentials. |
| `RESTIC_PASSWORD` | **Encryption key** — mandatory. Without it the repository is unreadable. |
| `BACKUP_SCHEDULE` | Cron schedule (default `0 3 * * *`). |
| `BACKUP_RETENTION` | `forget`/`prune` flags (default `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`). |
| `BACKUP_PRE_HOOKS` | `service:command:ext;…` consistency dumps, generated from `@stack backup-command` annotations. |

## Build & use (intended)

```bash
stack fetch repo bozemanpass/backup-stack
stack prepare --stack backup          # builds bozemanpass/backup:stack
# Backup is then enabled per-deployment via the `backup` config switch; see docs/backup.md.
```

# backup-stack

Backup and restore of persistent service data for the
[`stack`](https://github.com/bozemanpass/stack) tool's Docker deployment mode.

> **Status: initial scaffold — not yet functional.** The container scripts are a starting point and have
> not been run end-to-end. See the design in the `stack` repo: `docs/backup.md` and
> `docs/backup-implementation.md`.

## What this provides

`bozemanpass/backup` — a small Alpine image bundling:

- [restic](https://restic.net) — the backup engine: mandatory client-side encryption, deduplication, and
  native S3 support, so commodity object storage can be used safely;
- a cron scheduler;
- the Docker CLI — used to run application-consistency hooks (e.g. `pg_dump`) *inside* the target service
  container, the same way the ingress proxy uses the Docker socket.

On Kubernetes the equivalent role is played by [K8up](https://k8up.io) (also restic-based), so the two
targets produce interchangeable repositories. This repo covers the Docker case only.

## Layout

| Path | Purpose |
|------|---------|
| `stacks/backup/stack.yml` | Stack definition — declares the container and the pod. |
| `backup/composefile.yml` | The canonical `backup` service. `stack deploy` injects read-only mounts of the application's data volumes here when backup is enabled. |
| `containers/backup/` | The `bozemanpass/backup` image: `Containerfile`, `build.sh`, and `scripts/`. |

## Container modes

The image entrypoint takes a mode argument (default `schedule`):

| Mode | Action |
|------|--------|
| `schedule` | Install a cron entry (`BACKUP_SCHEDULE`) that runs `backup` periodically. |
| `backup` | Run hooks, then `restic backup` of `/backup`, then apply retention. |
| `restore [snapshot]` | Restore a snapshot into the (rw-mounted) volumes. Default `latest`. |
| `prune` | Apply the retention policy (`restic forget --prune`). |
| `check` | Verify repository integrity. |

## Configuration

Supplied by `stack` from the deployment environment (see `docs/backup.md` for the profile keys):

| Variable | Purpose |
|----------|---------|
| `BACKUP_S3_ENDPOINT`, `BACKUP_S3_BUCKET` | Object store location (or set `RESTIC_REPOSITORY` directly). |
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | Object store credentials. |
| `RESTIC_PASSWORD` | **Encryption key** — mandatory. Without it the repository is unreadable. |
| `BACKUP_SCHEDULE` | Cron schedule (default `0 3 * * *`). |
| `BACKUP_RETENTION` | `forget`/`prune` flags (default `--keep-daily 7 --keep-weekly 4 --keep-monthly 6`). |
| `BACKUP_PRE_HOOKS` | `service:command:ext;…` consistency dumps, generated from `@stack backup-command` annotations. |

## Build & use (intended)

```bash
stack fetch repo bozemanpass/backup-stack
stack prepare --stack backup          # builds bozemanpass/backup:stack
# Backup is then enabled per-deployment via the `backup` config switch; see docs/backup.md.
```