A homelab in two cities, kept in git
Two small servers, one in Amsterdam and one in Lyon, running photos, notes and files. What went into keeping them in git, watching them from somewhere else, and backing them up in a way a compromised machine cannot undo.
I live mostly in Amsterdam and partly in Lyon, and there is a small server in each: a Debian box with ZFS and a single-node k3s cluster, running Immich for photos, Confetti for notes, Nextcloud for files and Home Assistant for the house. They are joined by WireGuard and take almost nothing from the internet directly. Between spring and autumn 2026 I moved all of it into git — seven repositories, all private, since they describe the network in more detail than I want public. What follows is the part that is worth describing without the addresses.
Two directions of truth
Most of the setup is declarative in the usual way: Ansible describes each server, from packages and firewall to monitoring and backups, and the Pi-hole on each site is converged from a repository through its API. The routers went the other way. The Amsterdam one and the ISP box in Lyon are backed up by pull-only scripts that read their configuration, encrypt the sensitive parts and stage the result for me to commit, and never write anything back.
The ISP box made that choice easy, because its API is undocumented, its firmware updates itself — nine endpoints had already stopped answering when I wrote the backup — and a bad write could cut a site’s internet, and my remote access to it along with it. Restoring it is a manual job through the web interface, and a test checks that the code can only issue reads. I had first planned one shared tool for all the network devices and dropped it once it was clear that its central rule, never write to the device, was the opposite of what the Pi-hole needed.
Watching from somewhere else
Every backup, replication job and service pings a Healthchecks instance when it runs, and stays quiet otherwise, so a missing ping is what raises the alert. That instance runs on a small rented server at a provider that hosts nothing else of mine, together with the ntfy server that pushes alerts to my phone, because monitoring that lives on the cluster goes down with it. The rented server is itself watched by a free account on healthchecks.io, which it pings every five minutes, and that is the only thing that would notice it dying.
Backups a compromised server cannot delete
Until this summer, the Nextcloud data in Lyon existed only in Lyon. It is now copied every night to Amsterdam with zfs send, and the Amsterdam side does the pulling: it logs into Lyon as an account whose key can only run one command, which lists, snapshots and sends the allowed datasets and nothing else. A compromised Lyon server therefore has no way to reach the copies, let alone delete them. A ZFS snapshot is atomic, so the database and the files it refers to arrive as one consistent point in time, which the file synchronisation I use elsewhere cannot promise. The copy is encrypted under Amsterdam’s own key, so reading it does not depend on a key that would be lost along with the Lyon server. A run only counts as successful once the received snapshot is confirmed identical to the source, and the result goes to Healthchecks like everything else.
Dead ends and surprises
From June to November I ran OpenCloud next to Nextcloud as a possible replacement, and in the end kept Nextcloud and removed the trial, copying back the 28 files that only existed on the OpenCloud side first. Upgrading k3s also turned out to need its own playbook, kept out of the normal deploy, because k3s cannot skip minor versions: it steps through each one, backs up the cluster’s database before every hop, and stops if a node does not come back healthy.
The most useful surprise came from drawing a map of the whole setup from its repositories. The first version described Nextcloud as planned when it had been in production for months, because the copy on my laptop was eighteen commits behind the server it was supposed to document, and another repository had unpushed work that existed only on the laptop. I am bad at keeping that many repositories in step by hand, which is why a few weeks later I had Claude build GitRoster for me, and it has kept them in step since.
Was it worth it?
Yes. Both servers can be rebuilt from their repositories, every job that matters reports whether it ran, and the Lyon data now has a copy in Amsterdam that the Lyon server cannot touch.
Private repositories — Ansible, k3s, ZFS, WireGuard, Healthchecks, ntfy