Managing configs on my homeservers.

Managing configuration files - some lessons I've learned over the years.

Managing homeserver configs

I’ve run my home services a number of different ways over the years. I’ve split things between multiple virtual machines, I’ve set up a ‘bare metal ’ kubernetes cluster distributed between multiple VMs and hardware devices on my home network. I’ve used FreeBSD and its Jails to run things I compiled by scratch in an effort to lower attack surface. I ran (and run) VMs and containers on proxmox, truenas core and truenas scale. Each method brings its pros & cons, security tradeoffs, and configuration complexity. Though I’ve practiced more complex enterprise-level user & permission management (ldap/active directory) techniques, I’ve settled on “good enough” security practices for my uses/needs (I don’t have multiple people accessing things over ssh, for example, so I do the unthinkable and - gasp - ssh directly into root with an ed25519 keypair to administer servers). No SSH ports are exposed directly to the internet anyway - well, except for gitea. But that’s also protected with keypairs.

Similarly, I seek to reduce complexity of my configuration management. I like to do as much work as possible, in my text editor of choice (that’s VScode. I know, I know. I use the microsoft text editor. Controversial opinion: its good. Shoot me, emacs and vim nerds). That means using things like webuis to enter configs is out the window (looking at you, truenas scale kubernetes bullshit). Doing things in the text editor means I’m using git to version manage. I also like to use a combination of custom shell scripts and gitea actions config files to automate workflows. Any commands I run frequently get stuck into a shell script, no matter how mundane. I spent a long time manually deploying configs for docker - I know how that tool works. Hell, I know how all my tools work. I want to spend less time entering ssh host "docker compose down;docker compose up -d;" and more time doing a ./sripts/docker-down-up. I don’t want to enter an ever-changing esoteric webui for some haphazard k3s deployment to look for/edit a hacky series of docker-compose configs rearranged into different parts of said webui. That stuff just annoys me when I have to change things.

Speaking of k3s/k8s - Fuck that noise entirely in a home environment. Unless you’re doing it to learn, I recommend staying away from kubernetes. Its just docker with extra steps and its far more trouble than its worth for the home - in my very strong opinion.

So how do I do things?

Well, as I alluded to earlier - I work in my text editor, out of git repositories. All of my services are deployed with docker - its just… easier, this way. I’ve run services so many different ways over the years and docker is simply the easiest to deal with. I can grab premade containers. Or I can make my own, push them to my gitea deployment, and pull them for use later. Its great. And its distro agnostic. Sure there are some security issues associated with it. But there are also well documented methods to nullify them. I can also use docker volumes to store everything in /opt/{container_name} which is super handy when it comes time to archive/back up the host since all I need to do to grab any important data is backup /opt!

Most things get pushed directly to my gitea server. If there are actions that need to be run (such as building and pushing docker containers or other packages), I write a gitea actions config to handle that - its for all intents and puposes exactly the same as github actions. Which is nice. It simply uses a privilaged docker container to spin up other docker containers to do stuff that I would normally do by hand or with a script called build or deploy.

Though, there are some things that have to be managed manually. one of them is the repo for all the config files for services run on the deployment host. The other, is the repo for my nginx reverse proxy - because if I use gitea to deploy that docker container, it will… turn off the reverse proxy. Which is a link between gitea and the act runner. So… Yeah… can’t do that. Cus it causes issues.

These manually managed repos are pushed directly to a bare repo on the deployment host. Then, a script is run that SSHs into the host and runs some commands.

In the case of the main config repo, there also resides an ssh authorized_keys file, some scripts in a folder called ci, docker configs, and a big folder of scripts to deploy the thing, manually run actions on the docker configs, and more. The authorized_keys file and ci folder allow me to use gitea actions to deploy docker images on the host. I generate an ssh keypair, I store the private key as a secret in each individual repo that deploys to the host, then i put the public key in the authorized_keys file with a command that points to a script in ci that pulls and redeploys docker images.

conclusion

This is more or less some rambling about how I manage configs in git. I hope any amount of this made sense.