No description
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
2026-11-16 14:07:55 +01:00
automations style(compose): apply dclint key order and formatting across all stacks 2026-11-12 14:05:00 +01:00
caddy chore(infra): refresh caddy 2.11.9-alpine digest 2026-11-15 17:42:38 +01:00
db-backups feat(infra): bump db-backup 5.0.10 to 5.0.11 2026-11-15 17:33:40 +01:00
floriandormann-de feat(infra): bump pocketbase-docker 0.41.1 to 0.42.2 2026-11-15 17:41:48 +01:00
gatus feat(konfiapp): monitor externe-quellen auto-sync providers in gatus 2026-11-14 13:53:51 +01:00
gymh chore(infra): refresh postgres 18.7-alpine3.24 digest 2026-11-15 12:20:22 +01:00
kirchapp feat(infra): bump pocketbase-docker 0.41.1 to 0.42.2 2026-11-15 17:41:48 +01:00
kirchen-apps-de feat(infra): bump pocketbase-docker 0.41.1 to 0.42.2 2026-11-15 17:41:48 +01:00
konfiapp/core chore(infra): refresh nginx 1-alpine-slim digest 2026-11-15 17:43:33 +01:00
philippdormann-de style(compose): apply dclint key order and formatting across all stacks 2026-11-12 14:05:00 +01:00
s3 feat(infra): bump rustfs 1.0.5 to 1.0.6 2026-11-16 14:07:55 +01:00
splat feat(infra): bump splat 1.4.1 to 1.4.4 2026-11-15 13:01:57 +01:00
tca style(compose): apply dclint key order and formatting across all stacks 2026-11-12 14:05:00 +01:00
timetag feat(infra): bump pocketbase-docker 0.41.1 to 0.42.2 2026-11-15 17:41:48 +01:00
transaktio feat(infra): bump pocketbase-docker 0.41.1 to 0.42.2 2026-11-15 17:41:48 +01:00
umami chore(infra): refresh postgres 18.7-alpine3.24 digest 2026-11-15 12:20:22 +01:00
wakapi feat(wakapi): add explicit healthcheck mirroring image default 2026-11-14 11:32:45 +01:00
watchtower style(compose): apply dclint key order and formatting across all stacks 2026-11-12 14:05:00 +01:00
wg feat(infra): bump wireguard 1.0.20260223-r0-ls129 to 1.0.20260223-r0-ls130 2026-11-12 23:16:55 +01:00
.gitignore fix(infra): extend gitignore for env and runtime data 2026-11-09 17:54:29 +01:00
.gitlab-ci.yml feat(infra): caddy upstream cross-check in CI + branch docs 2026-11-09 21:59:38 +01:00
.sops.yaml docs(infra): bus-factor readme, cron docs, sops key location 2026-11-09 21:47:42 +01:00
AGENTS.md docs(infra): forbid merge commits ("Merge branch ...") 2026-11-10 11:24:52 +01:00
check-caddy-upstreams.sh feat(infra): caddy upstream cross-check in CI + branch docs 2026-11-09 21:59:38 +01:00
cleanup.sh feat: add server crontab scripts 2026-11-09 17:59:05 +01:00
deploy.sh feat(deploy): isolate per-stack failures, report at end 2026-11-12 15:00:51 +01:00
disk_check.sh feat: add server crontab scripts 2026-11-09 17:59:05 +01:00
docker_image_prune.sh feat: add server crontab scripts 2026-11-09 17:59:05 +01:00
README.md docs(infra): standardize on restart always 2026-11-12 14:04:37 +01:00
sops-decrypt.sh fix(infra): POSIX sh compat for dash (no read -d) 2026-11-09 20:30:41 +01:00
sops-encrypt.sh fix(infra): POSIX sh compat for dash (no read -d) 2026-11-09 20:30:41 +01:00

pd-infra

If Philipp is unavailable, start here. This repo runs everything on a single VPS (SSH as root@pd004). That VPS hosts ~15 customer/product websites plus internal tools. All internet traffic enters through one reverse proxy (Caddy), which forwards each domain to its app container. If you can SSH into the box and run docker ps, you can keep the lights on - this file tells you how. No commits needed for routine operation; see "Rules" below.

Infrastructure-as-code for the pd VPS. Each directory is one stack (docker-compose.yaml + env files), deployed with deploy.sh. The repo is the source of truth; the VPS at /usr/data is a checkout of it.

Access - where everything lives

What Where Notes
The server ssh root@pd004 (SSH key, no password) All work happens here as root
This repo gitlab.com/philippdormann/infra, checked out at /usr/data on the VPS Read-only deploy key in /root/.ssh/infra_deploy; remote must stay SSH, not HTTPS
Secrets (passwords, API keys) Encrypted in this repo (*.env.enc); plaintext only in /usr/data/*/[.]env (600) Decrypt needs the age key: /root/.age.key on VPS, ~/.config/sops/age/keys.txt on the admin Mac, a second copy in the password manager. If nobody besides Philipp can open that password manager (emergency-access contact or sealed master password), the secrets are unrecoverable - fix that before anything else in this file.
Backups db-backup service (db-backups/); S3 targets are in its .env Verify targets exist before you need them
"Is it working?" status.philippdormann.de, status.konfiapp.de, status.kirch.app Green = fine. Alerts go to Telegram (token/chat in gatus/.env)
Analytics a.* domains umami; irrelevant in an outage, ignore it

Service map - which domain is what

Traffic: internet -> Caddy (caddy/) -> container in the listed directory. Redirect-only entries need no action (they're just forwarding addresses).

Domain(s) Directory What it is
konfiapp.de + portal/elternportal/verwaltung/api/cdn/img/pb konfiapp/core KonfiApp (flagship product, most traffic)
kirch.app + web/api/pb/img kirchapp/ KirchApp
transaktio.de + app/api/pb transaktio/ Transaktio (Stripe billing tool, handles money - extra care)
timetag.cc + api/pb timetag/ TimeTag
kirchen-apps.de + pb kirchen-apps-de/ Kirchen-Apps site
gymh*.philippdormann.de (3 subdomains) gymh/ Customer: Gymnasium Herzogenaurach
tcaurachtal.de (+ 3 redirect aliases) tca/ Customer: TC Aurachtal (Ghost blog)
philippdormann.de philippdormann-de/ Personal site
dormann-media.de + backend floriandormann-de/ Customer: Dormann Media (also hosts Florian's pocketbase)
splat.philippdormann.de splat/ Splat error tracker (own product, devs rely on it)
wakapi.philippdormann.de wakapi/ Coding-time tracker
n8n.philippdormann.de automations/ n8n-style automation webhooks (incl. disk-space alerts)
s3.* + console s3/ S3-compatible storage (backends use it, incl. backups)
*.stripe2sevdesk.de, kirchnah.de, notfall-infos.de - (deleted) Redirects only, no containers. If a redirect 404s, it's in caddy/Caddyfile

When something breaks - triage in order

One site is down, rest fine: 1) docker ps --filter name=<container> - running? 2) docker logs <container> --tail 20. 3) cd /usr/data/<dir> && docker compose up -d (restarts only what changed). 4) Check status.* page - was Gatus already alerting on Telegram?

Everything is down: it's Caddy or the host. docker ps | grep caddy, docker logs caddy-caddy-1 --tail 20, df -h / (full disk kills everything), systemctl status docker. Caddy certs renew automatically; if port 443 broke after a Caddyfile edit, cd /usr/data/caddy && docker compose up -d with the last-known-good file (git log -p -- Caddyfile).

I deployed something bad: every prod change is a git commit, so roll back: cd /usr/data && git log --oneline -5, git revert <commit>, ./deploy.sh. Digests make this exact - you return to byte-identical images.

Disk full: df -h /, du -sh /var/lib/docker/*, docker image prune -af frees layers (safe, images re-pull), ./docker_image_prune.sh is the legacy script. Then find what filled it - usually runaway json logs (rotation is set everywhere, but verify) or pb_data growth.

VPS is dead (rebuild): new box -> install docker/sops/age (see Prerequisites) -> place /root/.age.key (600) -> clone repo to /usr/data -> ./deploy.sh (recreates everything except data) -> restore data volumes from backup. Practice this on a throwaway box before you need it.

Rules (please don't...)

  1. Never edit compose files, env files, or the Caddyfile directly on the VPS. Edit here in the repo, commit, push, git pull && ./deploy.sh on the VPS. The git log is the deploy log - SSH edits make it a lie.
  2. Never git add a plain .env. git status must never show one. Secrets travel only as .env.enc.
  3. Never delete a docker volume (docker volume rm) unless you have verified its backup restores. Volumes hold the databases; images are disposable, volumes are not.
  4. Two robots update containers, know which: Watchtower auto-updates app images (backend-1, website-1, ...) within a minute of CI pushing them; deploy.sh (scheduled, see below, plus manual runs) updates everything else, especially digest-pinned infra images. If a container restarted and you didn't touch anything, it was Watchtower - check docker logs watchtower*.

Layout

caddy/               reverse proxy (single entrypoint, port 80/443)
<project>/           one stack per directory (kirchapp/, umami/, ...)
konfiapp/core            full KonfiApp stack (backend, portals, CDN, DBs, imageproxy, osmcache tile proxy)
deploy.sh            git pull + decrypt + `compose up` for every stack + prune
sops-encrypt.sh      one-time/onboarding: plaintext .env -> committed .env.enc
sops-decrypt.sh      deploy-time: committed .env.enc -> plaintext .env (600)
.sops.yaml           age recipients for encryption (public keys only)
.gitlab-ci.yml       validate-only CI (no secrets, no deploys)
cleanup.sh / disk_check.sh / docker_image_prune.sh   legacy VPS cron scripts

Per-project files:

<project>/docker-compose.yaml   in git, no secrets
<project>/*.env.example         in git, keys only, values empty/placeholder
<project>/*.env.enc             in git, sops+age ciphertext (decryptable by key holders)
<project>/*.env                 NEVER in git (gitignored), VPS-local plaintext, chmod 600

Prerequisites

Machine Needs
Mac (admin) brew install sops age, age key in ~/.config/sops/age/keys.txt
VPS (root) sops binary in /usr/local/bin (not apt-packaged, see below), age via apt, private key at /root/.age.key (600)
CI nothing (validates against .env.example stand-ins)

VPS sops install (apt has no sops package):

ARCH=$(dpkg --print-architecture)  # amd64 or arm64
SOPS_VERSION=3.13.3
curl -fsSL -o /tmp/sops \
  "https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.${ARCH}"
sudo install -m 0755 /tmp/sops /usr/local/bin/sops
sudo apt-get update && sudo apt-get install -y age

Secrets (sops + age)

Public parts (.sops.yaml recipients, .env.example key names) are committable and VPS-pullable without risk. The only secret is the age private key, which lives in exactly 3 places: admin Mac, /root/.age.key on the VPS, and a second copy in the password manager. Always keep 2 age recipients in .sops.yaml (main + backup) so losing one key doesn't lock the repo.

Key transfer Mac -> VPS (pipe, never commit, never paste into chat):

ssh root@pd004 'cat > /root/.age.key && chmod 600 /root/.age.key' < ~/.config/sops/age/keys.txt
Task Command (repo root)
Encrypt new/changed plaintext ./sops-encrypt.sh [path/to/file.env] (skips existing .enc)
Decrypt everything ./sops-decrypt.sh (needs SOPS_AGE_KEY_FILE or default key paths)
Edit a secret sops --input-type dotenv <file>.env.enc (--input-type is mandatory: .enc is not a sops-known suffix)
Rekey (add/remove recipient) edit .sops.yaml, then sops updatekeys -y <file>.env.enc per file
Pre-commit leak check git status --short must never show a plain .env (only .enc/.example)

deploy.sh runs sops-decrypt.sh after git pull. Pre-migration (zero .enc files) it warns and continues; once .enc files exist, missing keys or an unconfigured .sops.yaml are hard failures. Plaintext .env on the VPS is a regenerable build artifact, not a backup - the backup is the repo + offline key.

Deploy (VPS)

cd /usr/data && ./deploy.sh
# == git pull --ff-only
# == decrypt secrets (sops-decrypt.sh)
# == <each stack>: docker compose up -d --pull always --wait --remove-orphans
# prune images older than 7 days

up without --force-recreate only restarts changed services. Changed services still have seconds of downtime (single replica) - expected.

Scheduled jobs (VPS root crontab)

All output is discarded (> /dev/null 2>&1) - on failure, look at docker ps, container logs, and /var/log/syslog (cron itself), not here.

Schedule Command Purpose
5,35 * * * * /bin/bash /usr/data/disk_check.sh Disk usage check, reports to n8n webhook
2,22,42 * * * * /bin/bash /usr/data/deploy.sh Pull + decrypt + compose up per stack (idempotent; catches drift and ships digest bumps ~20 min after merge)
0 * * * * /bin/bash /usr/data/docker_image_prune.sh docker image prune -af (no age filter - removes all unused images, incl. rollback cache)
10,20,30,40,50 * * * * /bin/bash /usr/data/cleanup.sh Truncate container logs, apt autoclean, vacuum journal

Note the overlap: deploy.sh already prunes (with a 7-day filter) and every stack caps its own logs (10m x 1). The hourly full prune and the 6x-hourly cleanup are belt-and-braces from before those existed - harmless, but if disk pressure ever looks self-inflicted, this table is the map.

First-time VPS conversion (repo becomes truth; /usr/data must stay put because stacks bind-mount ./pb_data, ./data, ./cdnfiles relatively):

cd /usr/data
tar -czf /root/usr-data-backup-$(date +%F).tar.gz .   # + pg dumps per project
for n in proxynet db_backup_network; do docker network inspect $n >/dev/null 2>&1 || docker network create $n; done
git init -b main 2>/dev/null
git remote get-url origin 2>/dev/null || git remote add origin git@gitlab.com:philippdormann/infra.git
git fetch origin && git status --short | head -30   # eyeball before proceeding
git reset --hard origin/main   # replaces tracked files only; ignored .env/data dirs survive
./sops-decrypt.sh              # must reproduce live .envs; any diff = stop, .enc doesn't match prod

VPS git access: read-only deploy key (repo Settings -> Repository -> Deploy keys), not a user/token. Generate on VPS (ssh-keygen -t ed25519 -f /root/.ssh/infra_deploy), pin via /root/.ssh/config (Host gitlab.com, IdentityFile, IdentitiesOnly yes), ssh-keyscan the host key once, verify with ssh -T git@gitlab.com. Remote URL must be SSH (git@gitlab.com:...), not HTTPS.

Image updates

Third-party images are pinned tag@sha256:<manifest-list digest> (arch-independent). Moving app tags (backend-1, website-1, ...) intentionally stay floating for Watchtower (com.centurylinklabs.watchtower.scope=autoupdate). Infra images (postgres, imgproxy, ...) must NOT carry the autoupdate label - bump them deliberately: resolve digest (docker buildx imagetools inspect --format '{{ .Manifest.Digest }}' <tag>), update ref, compose config, commit. db-backup/splat bumps are the template (single-ref commits).

CI (.gitlab-ci.yml, validate only)

  • compose-validate: copies .env.example -> stand-in .env, runs docker compose config -q for every stack. Catches YAML/env errors pre-merge.
  • caddy-validate: caddy validate on the Caddyfile.
  • CI holds no age key and cannot read real secrets by design.

Branches & merging

Two long-lived branches: main (prod - what the VPS runs) and dev (integration). Work lands on dev or short-lived branches and reaches main only through a merge request.

Enforced server-side (GitLab repo settings, not convention):

  • main: nobody can push, not even the owner. Merge requires Maintainer role
    • green pipeline + resolved threads. No force-push.
  • dev: Maintainers can push and merge, no force-push.
  • Auto-merge is per-MR, not global: glab mr merge <id> (on by default while a pipeline runs) or the "Set to auto-merge" button in the UI.

Flow: hack on dev (or feature/x) -> push -> MR to main -> pipeline (compose-validate, caddy-validate, caddy-upstreams) -> auto-merge on green -> VPS cron pulls main and deploys within ~20 min. A red pipeline blocks the merge, so broken config can't reach prod by accident.

Conventions (enforced, don't regress)

  • Filename is docker-compose.yaml everywhere (.yaml is the spec-correct extension). Tooling matches -name 'docker-compose.yaml', max depth 3.
  • Every service: pull_policy: always + json-file logging (10m x 1).
  • Healthchecks only where the binary exists: wget works in alpine/node images (caddy, pocketbase, umami, rustfs, ghost, backends) and pg_isready in postgres. Scratch images (gatus, wakapi) have no shell - no healthcheck.
  • postgres env-based healthcheck: pg_isready -U $${POSTGRES_USER:-<default>} -d $${POSTGRES_DB:-<default>}, default matching the project's user.
  • postgres:18 requires a single parent mount (-v:/var/lib/postgresql, PGDATA is /var/lib/postgresql/18/docker). A /var/lib/postgresql/data suffix makes the container refuse to start. Named volumes persist across recreates; moving an old-layout volume means relocating root contents into 18/docker/.
  • DBs stay off wgeasy_default (commented, one-line re-enable). VPN network itself (wg/) is untouched.
  • No container_name sprawl for new stacks (Compose name: namespaces); restart: always everywhere (single VPS, containers must come back after reboot).

Backups / DR

  • db-backup covers databases (DBxx_*); file data (pb_data/, ghost data/, cdnfiles/) and .envs ride along only if included in the backup job - verify, and restore-test (unverified backups are wishes).
  • Full loss: fresh VPS -> install docker/sops/age -> place /root/.age.key -> clone repo -> ./deploy.sh regenerates everything except data volumes, which come from the backup.

Troubleshooting

Symptom Cause / fix
E: Unable to locate package sops expected on VPS distro; install upstream binary (see above)
age: command not found separate package: apt-get install age
sops no matching creation rules found on encrypt path_regex must match the plaintext path; ours covers \.env(\.enc)?$
sops-decrypt.sh placeholder error real age public keys not yet in .sops.yaml
Healthcheck never healthy, up --wait hangs image lacks the binary (scratch) or wrong endpoint; verify with docker run --rm --entrypoint sh <img> -c 'command -v wget'
postgres container exits citing /var/lib/postgresql/data PG18 parent-mount requirement (see above)
ssh -T git@gitlab.com asks for password wrong key offered; check IdentitiesOnly yes + key actually registered as deploy key