Backups, Backups, Backups

Requirements

Sort of 3-2-1 strategy implemented as 3 copies of the data existing (machine itself, local backup server, cloud backup server), at 2+ different devices (local server, (different) cloud provider), and 1 in a separate location.

Backups need to be encrypted, so a compromised backup server doesn't have access to all the backed-up machine's data. Good deduplication and compression are vital to keep all this affordable. Unfortunately (client-side) deduplication is essentially incompatible with asymmetric crypto/access, so (compromised) machines will have access to their old backups, but should be prevented from changing/deleting them.

Status Quo

I've been using BorgBackup 1.x for at least . A Borg 2.x rewrite has been in the works for at least , and has more recently (around ) pivoted to full-time usage of Slopification as a Service (scan). This has also started to affect the current 1.4.x stable/maintenance branch.

A"I"

If you're reading this, you're probably aware of the myriad of reasons to avoid generative AI, but just in case I've compiled a short reading list:

We exist in a world where it’s impossible to live a regular life (within the world’s rich countries) without relying on the exploitation of countless people and finite environmental resources; any coherent pro-social moral stance is instantly compromised upon contact with this society.

The food we eat, clothes we wear, and every electronic device we touch may embody innumerable injuries to the world, and all this is inescapable. Eschewing AI is one thing that we can actually do to live out ethics that affirm values of human and environmental rights. It’s almost a gift! Just use a computer the same way you did three years ago!

And then some absolute son of a bitch created ChatGPT, and now look at us. Look at us, resplendent in our pauper's robes, stitched from corpulent greed and breathless credulity, spending half of the planet's engineering efforts to add chatbot support to every application under the sun when half of the industry hasn't worked out how to test database backups regularly. This is why I have to visit untold violence upon the next moron to propose that AI is the future of the business - not because this is impossible in principle, but because they are now indistinguishable from a hundred million willful fucking idiots.

If another stupid motherfucker asks me to try and implement LLM-based code review to "raise standards" instead of actually teaching people a shred of discipline, I am going to study enough judo to throw them into the goddamn sun.

Snapshots and Backups and Snapshots

Back(up) to the topic at hand; I maintain two handfuls of computers, with varying degrees of data integrity importance. Every night (every 4 hours for my personal laptop+desktop), ZFS creates snapshots, and Borg backs these up to a remote server of some kind; this is an organically grown constellation of cloud servers (separate from Hetzner, my primary cloud provider), with block storage volumes in the 1-2TB range.

A backup server is accessible to other machines with backup-specific SSH keys, using borg serve --append-only to allow creating backups in per-machine repos, but blocking those machines from deleting their old backups (borg prune can still mark for deletion, but borg compact which actually removes backup data is not allowed).

This is an important quality in a backup system: a (compromised) machine should not be able to retroactively delete it's old backups.

The current, actively used iteration, runs duplo-server, which tracks backup metrics (with internal webhooks logging repo init/create/finish), and creates server-side ZFS snapshots of the backup repo after completion. This is mostly an extra safeguard on top of Borg, as I've ran into situations where full disks cause repo corruption.

Full disks

The biggest painpoint over the years has been backup servers (quietly) running out of disk space, preventing the creation of new backups. In a fully trusted setup (if that actually existed, why would you need backups at all?), a Borg client can prune its old backups according to a set of intervals (keep x daily, y weekly, z monthly backups), and compact to actually reclaim the space.

This requires active trust in the backed-up machine to not prune all of your backups (or replace old backups with garbage); append-only only saves you if you catch malicious transactions before borg compacting. You also can't really automate running the pruning/compacting; running a more trusted prune from the backup server requires the backup server to store the decryption key, defeating the point of encrypted backups (compact is allowed without decrypting). You would also still need to (manually) verify that a compromised client hasn't messed with old backups or pruned additional backups before actually reclaiming space.

Running prune/compact elsewhere additionally presents problems by holding locks; these operations can't run simultaneously with a new backup being made.

Changing Horses Midstream

Restic seems to have caught up to Borg in relevant features (regarding encryption/deduplication/compression and append-only mode), and while not entirely free from sin (scan), it's doing decidedly better than Kopia (scan).

The idea here is to use the Restic Rest Server to provide append-only functionality, with repositories on local block storage (on the backup server).

For reference, the Threat Model page in the docs.

Backup Workflow

Client

  1. pg_dump Postgresql databases (through services.postgresqlBackup).
  2. Snapshot 'safe' ZFS datasets (excluding /volatile datasets), mount in /backup
  3. Backup everything from /backup (monitoring hooks for init/backup/prune/check/finish stages)

Server

Maintenance

Note: Prune is one of few operations that needs an exclusive lock on the repository.

This maintenance step will need to be ran from a trusted device with elevated (non-append-only) repository access permissions, though it can be partially scripted.