projectsdrawing of a fox, shaded in rainbow accentsgallery

Pixie Town Server Overview

The Good

These servers make up the core pixie.town infrastructure, and they're all running NixOS 22.11 on various hardware. They're deployed using morph through a central config repo on my gitea.

Cosmos

My server at home. It has the most resources and runs the heaviest services, like the Matrix homeserver (Synapse) and Mastodon (glitch-soc fork). It's also my remote-coding environment, running code-server (unfortunately as a Docker image), to provide a unified setup with all my code for both my laptop (ouroboros) and desktop (titan).

While the hardware and local custody are great, it's behind a rather slow DSL uplink (70mbit down, 20mbit up on a good day) which is it's biggest bottleneck. All my servers are in a full-mesh Wireguard network, but most importantly Cosmos proxies all it's traffic through Aura, so incoming and outgoing traffic (especially to federated servers) don't trace back to my home ip directly.

I've also written some software to help reduce traffic to the homeserver, most importantly synapse-media-proxy. The local component runs on Cosmos, while the remote components runs on Aura, where it caches local media (especially recent uploads) in-memory, and proxies remote media directly, without involving the local homeserver at all.

Software:

node configuration

Hardware:

Aura

The main webserver node, running on a small Hetzner Cloud VPS. Serves various (static) sites through NGINX, and reverse-proxies various sites to Cosmos. Also runs synapse-media-proxy for smarter caching of Matrix media requests, ingests Prometheus metrics from other nodes and exposes them through Grafana dashboards. There are also a few other lightweight webservices, like Hedgedoc, Ethercalc and Wakapi.

Software:

node configuration

Copia

Current backup-host, managing Borg backup repositories for all other machines on a snapshotting ZFS pool. Runs on an AlphaVPS Storage VPS, the very cheapest place I found for vm's with large storage. Setting up NixOS took some trial-and-error, which I wrote about here.

Software:

node configuration

Via

Matrix bridges, a lot of them. matrix-appservice-irc to the Libera, OFTC, Smurfnet and Nobleme irc networks. mautrix-telegram and mautrix-twitter, and Mjolnir for moderating Matrix rooms. All runs on another small Hetzner Cloud VPS.

Software:

node configuration

Arx

Legacy backup-host, a BuyVM KVM slice 512 with a 1TB Block Storage Slab. Used to be the main Borg backups repo host, but that moved to Copia. Does currently still run the Gitea Git repositories (likely to be migrated to Forgejo), and accompanying pixie-pages static site hosting (serving you this blog!).

Software:

node configuration

The Bad

Any deployment is constantly in flux, but there are some concrete things I want to change in the future:

Backups

The current setup with Copia works quite well, but the monitoring can be improved. I also want to add a separate machine at home for local backups, instead of just the external hdd that's plugged into Cosmos. There's currently also backups on a Hetzner Storage box, which needs consolidating or improvement.

Decommission Arx

There's not really a need for Arx specifically anymore, although the Git repo's do need a bit of storage that would have to be provided by an AlphaVPS Storage VPS or network-mounted Hetzner Storage Box.

Monitoring

I run Alertmanager with go-neb to spam alerts on Matrix, but it's not tuned well so it just causes alert fatigue without being useful.

Metrics storage

Currently retention is very limited due to Aura only having a 20GB disk, shared with other services. Will either move this responsibility to Copia or use SSHFS mounted storage to store longer-term metrics.

NixOS Morph refactor

Refactor refactor refactor. Continually in progress but the current node / node info.nix / services structure works well. There's still some old cruft to clean up though.

The Ugly

There's still a few legacy hosts from before the NixOSification began. Partial incentive to finally write this post is to properly inventorize all my servers, to see what still needs to be migrated before these get decommissioned.

Helios

Old general-purpose Hetzner VPS. Still serves some legacy static sites and an unused Mumble server.

Akropolis

Even older general-purpose Hetzner VPS, the first one I got exactly 5 years ago.. Pretty sure it's only remaining purpose is a backup Weechat session, in case something happens with the Matrix irc bridges.

Hetzner Storage Box

Storage box I used for backups and networked storage, but rather unsatisfied with because you don't have control over the filesystem. There is (automatic) snapshot functionality through the web configuration interface, but in a recent incident where a Borg repository bricked itself you can only restore the full disk which really really sucks. Especially because now every other server trying to make backups thinks a replay-attack is happening, so you have to fix them all manually. I might still use it as networked storage for metrics, but something cheaper would be more appropriate.

The Financial (idk that wasn't in the movie)

I pay ~18 EUR a month to Hetzner for pixie.town-related servers and storage box, 7 USD to BuyVM, and 6.05 EUR to AlphaVPS. There's also a (discounted) email subscription at Migadu shared with my other personal domains, and yearly domain renewals for pixie.town and pixie.homes. I also buy hardware for Cosmos when the need arises.
With the upcoming consolidation and decommissioning the monthly costs will probably halve, will update this page when it happens.

I get some monthly donations in return, around 9 EUR through my Liberapay (either for my software or pixie.town, not further specified), and 3.14 EUR direct transfer. In the past, someone has also donated for a Cosmos component upgrade.