- Shell 100%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| automations | ||
| caddy | ||
| db-backups | ||
| floriandormann-de | ||
| gatus | ||
| gymh | ||
| kirchapp | ||
| kirchen-apps-de | ||
| konfiapp/core | ||
| philippdormann-de | ||
| s3 | ||
| splat | ||
| tca | ||
| timetag | ||
| transaktio | ||
| umami | ||
| wakapi | ||
| watchtower | ||
| wg | ||
| .gitignore | ||
| .gitlab-ci.yml | ||
| .sops.yaml | ||
| AGENTS.md | ||
| check-caddy-upstreams.sh | ||
| cleanup.sh | ||
| deploy.sh | ||
| disk_check.sh | ||
| docker_image_prune.sh | ||
| README.md | ||
| sops-decrypt.sh | ||
| sops-encrypt.sh | ||
pd-infra
If Philipp is unavailable, start here. This repo runs everything on a single VPS (SSH as
root@pd004). That VPS hosts ~15 customer/product websites plus internal tools. All internet traffic enters through one reverse proxy (Caddy), which forwards each domain to its app container. If you can SSH into the box and rundocker ps, you can keep the lights on - this file tells you how. No commits needed for routine operation; see "Rules" below.
Infrastructure-as-code for the pd VPS. Each directory is one stack
(docker-compose.yaml + env files), deployed with deploy.sh.
The repo is the source of truth; the VPS at /usr/data is a checkout of it.
Access - where everything lives
| What | Where | Notes |
|---|---|---|
| The server | ssh root@pd004 (SSH key, no password) |
All work happens here as root |
| This repo | gitlab.com/philippdormann/infra, checked out at /usr/data on the VPS |
Read-only deploy key in /root/.ssh/infra_deploy; remote must stay SSH, not HTTPS |
| Secrets (passwords, API keys) | Encrypted in this repo (*.env.enc); plaintext only in /usr/data/*/[.]env (600) |
Decrypt needs the age key: /root/.age.key on VPS, ~/.config/sops/age/keys.txt on the admin Mac, a second copy in the password manager. If nobody besides Philipp can open that password manager (emergency-access contact or sealed master password), the secrets are unrecoverable - fix that before anything else in this file. |
| Backups | db-backup service (db-backups/); S3 targets are in its .env |
Verify targets exist before you need them |
| "Is it working?" | status.philippdormann.de, status.konfiapp.de, status.kirch.app |
Green = fine. Alerts go to Telegram (token/chat in gatus/.env) |
| Analytics | a.* domains |
umami; irrelevant in an outage, ignore it |
Service map - which domain is what
Traffic: internet -> Caddy (caddy/) -> container in the listed directory.
Redirect-only entries need no action (they're just forwarding addresses).
| Domain(s) | Directory | What it is |
|---|---|---|
konfiapp.de + portal/elternportal/verwaltung/api/cdn/img/pb |
konfiapp/core |
KonfiApp (flagship product, most traffic) |
kirch.app + web/api/pb/img |
kirchapp/ |
KirchApp |
transaktio.de + app/api/pb |
transaktio/ |
Transaktio (Stripe billing tool, handles money - extra care) |
timetag.cc + api/pb |
timetag/ |
TimeTag |
kirchen-apps.de + pb |
kirchen-apps-de/ |
Kirchen-Apps site |
gymh*.philippdormann.de (3 subdomains) |
gymh/ |
Customer: Gymnasium Herzogenaurach |
tcaurachtal.de (+ 3 redirect aliases) |
tca/ |
Customer: TC Aurachtal (Ghost blog) |
philippdormann.de |
philippdormann-de/ |
Personal site |
dormann-media.de + backend |
floriandormann-de/ |
Customer: Dormann Media (also hosts Florian's pocketbase) |
splat.philippdormann.de |
splat/ |
Splat error tracker (own product, devs rely on it) |
wakapi.philippdormann.de |
wakapi/ |
Coding-time tracker |
n8n.philippdormann.de |
automations/ |
n8n-style automation webhooks (incl. disk-space alerts) |
s3.* + console |
s3/ |
S3-compatible storage (backends use it, incl. backups) |
*.stripe2sevdesk.de, kirchnah.de, notfall-infos.de |
- (deleted) | Redirects only, no containers. If a redirect 404s, it's in caddy/Caddyfile |
When something breaks - triage in order
One site is down, rest fine: 1) docker ps --filter name=<container> - running?
2) docker logs <container> --tail 20. 3) cd /usr/data/<dir> && docker compose up -d
(restarts only what changed). 4) Check status.* page - was Gatus already
alerting on Telegram?
Everything is down: it's Caddy or the host. docker ps | grep caddy,
docker logs caddy-caddy-1 --tail 20, df -h / (full disk kills everything),
systemctl status docker. Caddy certs renew automatically; if port 443 broke
after a Caddyfile edit, cd /usr/data/caddy && docker compose up -d with the
last-known-good file (git log -p -- Caddyfile).
I deployed something bad: every prod change is a git commit, so roll back:
cd /usr/data && git log --oneline -5, git revert <commit>, ./deploy.sh.
Digests make this exact - you return to byte-identical images.
Disk full: df -h /, du -sh /var/lib/docker/*, docker image prune -af
frees layers (safe, images re-pull), ./docker_image_prune.sh is the legacy
script. Then find what filled it - usually runaway json logs (rotation is set
everywhere, but verify) or pb_data growth.
VPS is dead (rebuild): new box -> install docker/sops/age (see
Prerequisites) -> place /root/.age.key (600) -> clone repo to /usr/data
-> ./deploy.sh (recreates everything except data) -> restore data volumes
from backup. Practice this on a throwaway box before you need it.
Rules (please don't...)
- Never edit compose files, env files, or the Caddyfile directly on the
VPS. Edit here in the repo, commit, push,
git pull && ./deploy.shon the VPS. The git log is the deploy log - SSH edits make it a lie. - Never
git adda plain.env.git statusmust never show one. Secrets travel only as.env.enc. - Never delete a docker volume (
docker volume rm) unless you have verified its backup restores. Volumes hold the databases; images are disposable, volumes are not. - Two robots update containers, know which: Watchtower auto-updates app
images (
backend-1,website-1, ...) within a minute of CI pushing them;deploy.sh(scheduled, see below, plus manual runs) updates everything else, especially digest-pinned infra images. If a container restarted and you didn't touch anything, it was Watchtower - checkdocker logs watchtower*.
Layout
caddy/ reverse proxy (single entrypoint, port 80/443)
<project>/ one stack per directory (kirchapp/, umami/, ...)
konfiapp/core full KonfiApp stack (backend, portals, CDN, DBs, imageproxy, osmcache tile proxy)
deploy.sh git pull + decrypt + `compose up` for every stack + prune
sops-encrypt.sh one-time/onboarding: plaintext .env -> committed .env.enc
sops-decrypt.sh deploy-time: committed .env.enc -> plaintext .env (600)
.sops.yaml age recipients for encryption (public keys only)
.gitlab-ci.yml validate-only CI (no secrets, no deploys)
cleanup.sh / disk_check.sh / docker_image_prune.sh legacy VPS cron scripts
Per-project files:
<project>/docker-compose.yaml in git, no secrets
<project>/*.env.example in git, keys only, values empty/placeholder
<project>/*.env.enc in git, sops+age ciphertext (decryptable by key holders)
<project>/*.env NEVER in git (gitignored), VPS-local plaintext, chmod 600
Prerequisites
| Machine | Needs |
|---|---|
| Mac (admin) | brew install sops age, age key in ~/.config/sops/age/keys.txt |
| VPS (root) | sops binary in /usr/local/bin (not apt-packaged, see below), age via apt, private key at /root/.age.key (600) |
| CI | nothing (validates against .env.example stand-ins) |
VPS sops install (apt has no sops package):
ARCH=$(dpkg --print-architecture) # amd64 or arm64
SOPS_VERSION=3.13.3
curl -fsSL -o /tmp/sops \
"https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.${ARCH}"
sudo install -m 0755 /tmp/sops /usr/local/bin/sops
sudo apt-get update && sudo apt-get install -y age
Secrets (sops + age)
Public parts (.sops.yaml recipients, .env.example key names) are committable
and VPS-pullable without risk. The only secret is the age private key,
which lives in exactly 3 places: admin Mac, /root/.age.key on the VPS,
and a second copy in the password manager. Always keep 2 age recipients in
.sops.yaml (main + backup) so losing one key doesn't lock the repo.
Key transfer Mac -> VPS (pipe, never commit, never paste into chat):
ssh root@pd004 'cat > /root/.age.key && chmod 600 /root/.age.key' < ~/.config/sops/age/keys.txt
| Task | Command (repo root) |
|---|---|
| Encrypt new/changed plaintext | ./sops-encrypt.sh [path/to/file.env] (skips existing .enc) |
| Decrypt everything | ./sops-decrypt.sh (needs SOPS_AGE_KEY_FILE or default key paths) |
| Edit a secret | sops --input-type dotenv <file>.env.enc (--input-type is mandatory: .enc is not a sops-known suffix) |
| Rekey (add/remove recipient) | edit .sops.yaml, then sops updatekeys -y <file>.env.enc per file |
| Pre-commit leak check | git status --short must never show a plain .env (only .enc/.example) |
deploy.sh runs sops-decrypt.sh after git pull. Pre-migration (zero .enc
files) it warns and continues; once .enc files exist, missing keys or an
unconfigured .sops.yaml are hard failures. Plaintext .env on the VPS is a
regenerable build artifact, not a backup - the backup is the repo + offline key.
Deploy (VPS)
cd /usr/data && ./deploy.sh
# == git pull --ff-only
# == decrypt secrets (sops-decrypt.sh)
# == <each stack>: docker compose up -d --pull always --wait --remove-orphans
# prune images older than 7 days
up without --force-recreate only restarts changed services. Changed
services still have seconds of downtime (single replica) - expected.
Scheduled jobs (VPS root crontab)
All output is discarded (> /dev/null 2>&1) - on failure, look at
docker ps, container logs, and /var/log/syslog (cron itself), not here.
| Schedule | Command | Purpose |
|---|---|---|
5,35 * * * * |
/bin/bash /usr/data/disk_check.sh |
Disk usage check, reports to n8n webhook |
2,22,42 * * * * |
/bin/bash /usr/data/deploy.sh |
Pull + decrypt + compose up per stack (idempotent; catches drift and ships digest bumps ~20 min after merge) |
0 * * * * |
/bin/bash /usr/data/docker_image_prune.sh |
docker image prune -af (no age filter - removes all unused images, incl. rollback cache) |
10,20,30,40,50 * * * * |
/bin/bash /usr/data/cleanup.sh |
Truncate container logs, apt autoclean, vacuum journal |
Note the overlap: deploy.sh already prunes (with a 7-day filter) and every
stack caps its own logs (10m x 1). The hourly full prune and the
6x-hourly cleanup are belt-and-braces from before those existed - harmless,
but if disk pressure ever looks self-inflicted, this table is the map.
First-time VPS conversion (repo becomes truth; /usr/data must stay put
because stacks bind-mount ./pb_data, ./data, ./cdnfiles relatively):
cd /usr/data
tar -czf /root/usr-data-backup-$(date +%F).tar.gz . # + pg dumps per project
for n in proxynet db_backup_network; do docker network inspect $n >/dev/null 2>&1 || docker network create $n; done
git init -b main 2>/dev/null
git remote get-url origin 2>/dev/null || git remote add origin git@gitlab.com:philippdormann/infra.git
git fetch origin && git status --short | head -30 # eyeball before proceeding
git reset --hard origin/main # replaces tracked files only; ignored .env/data dirs survive
./sops-decrypt.sh # must reproduce live .envs; any diff = stop, .enc doesn't match prod
VPS git access: read-only deploy key (repo Settings -> Repository ->
Deploy keys), not a user/token. Generate on VPS
(ssh-keygen -t ed25519 -f /root/.ssh/infra_deploy), pin via
/root/.ssh/config (Host gitlab.com, IdentityFile, IdentitiesOnly yes),
ssh-keyscan the host key once, verify with ssh -T git@gitlab.com.
Remote URL must be SSH (git@gitlab.com:...), not HTTPS.
Image updates
Third-party images are pinned tag@sha256:<manifest-list digest>
(arch-independent). Moving app tags (backend-1, website-1, ...) intentionally
stay floating for Watchtower (com.centurylinklabs.watchtower.scope=autoupdate).
Infra images (postgres, imgproxy, ...) must NOT carry the autoupdate label -
bump them deliberately: resolve digest
(docker buildx imagetools inspect --format '{{ .Manifest.Digest }}' <tag>),
update ref, compose config, commit. db-backup/splat bumps are the template
(single-ref commits).
CI (.gitlab-ci.yml, validate only)
compose-validate: copies.env.example-> stand-in.env, runsdocker compose config -qfor every stack. Catches YAML/env errors pre-merge.caddy-validate:caddy validateon the Caddyfile.- CI holds no age key and cannot read real secrets by design.
Branches & merging
Two long-lived branches: main (prod - what the VPS runs) and dev
(integration). Work lands on dev or short-lived branches and reaches
main only through a merge request.
Enforced server-side (GitLab repo settings, not convention):
main: nobody can push, not even the owner. Merge requires Maintainer role- green pipeline + resolved threads. No force-push.
dev: Maintainers can push and merge, no force-push.- Auto-merge is per-MR, not global:
glab mr merge <id>(on by default while a pipeline runs) or the "Set to auto-merge" button in the UI.
Flow: hack on dev (or feature/x) -> push -> MR to main -> pipeline
(compose-validate, caddy-validate, caddy-upstreams) -> auto-merge on
green -> VPS cron pulls main and deploys within ~20 min. A red pipeline
blocks the merge, so broken config can't reach prod by accident.
Conventions (enforced, don't regress)
- Filename is
docker-compose.yamleverywhere (.yamlis the spec-correct extension). Tooling matches-name 'docker-compose.yaml', max depth 3. - Every service:
pull_policy: always+ json-file logging (10m x 1). - Healthchecks only where the binary exists:
wgetworks in alpine/node images (caddy, pocketbase, umami, rustfs, ghost, backends) andpg_isreadyin postgres. Scratch images (gatus, wakapi) have no shell - no healthcheck. - postgres env-based healthcheck:
pg_isready -U $${POSTGRES_USER:-<default>} -d $${POSTGRES_DB:-<default>}, default matching the project's user. - postgres:18 requires a single parent mount (
-v:/var/lib/postgresql, PGDATA is/var/lib/postgresql/18/docker). A/var/lib/postgresql/datasuffix makes the container refuse to start. Named volumes persist across recreates; moving an old-layout volume means relocating root contents into18/docker/. - DBs stay off
wgeasy_default(commented, one-line re-enable). VPN network itself (wg/) is untouched. - No
container_namesprawl for new stacks (Composename:namespaces);restart: alwayseverywhere (single VPS, containers must come back after reboot).
Backups / DR
db-backupcovers databases (DBxx_*); file data (pb_data/, ghostdata/,cdnfiles/) and.envs ride along only if included in the backup job - verify, and restore-test (unverified backups are wishes).- Full loss: fresh VPS -> install docker/sops/age -> place
/root/.age.key-> clone repo ->./deploy.shregenerates everything except data volumes, which come from the backup.
Troubleshooting
| Symptom | Cause / fix |
|---|---|
E: Unable to locate package sops |
expected on VPS distro; install upstream binary (see above) |
age: command not found |
separate package: apt-get install age |
sops no matching creation rules found on encrypt |
path_regex must match the plaintext path; ours covers \.env(\.enc)?$ |
sops-decrypt.sh placeholder error |
real age public keys not yet in .sops.yaml |
Healthcheck never healthy, up --wait hangs |
image lacks the binary (scratch) or wrong endpoint; verify with docker run --rm --entrypoint sh <img> -c 'command -v wget' |
postgres container exits citing /var/lib/postgresql/data |
PG18 parent-mount requirement (see above) |
ssh -T git@gitlab.com asks for password |
wrong key offered; check IdentitiesOnly yes + key actually registered as deploy key |