I've always had someone else handling infrastructure. At startups, I did enough to keep things running given competing priorities. I never had a reason to go deep on networking, storage, or how a cluster actually works under the covers.
So I built a reason. A full application on a managed Kubernetes cluster: an API gateway with typed SDK generation, a sharded tables service, AI agents that trigger on data mutations, PostgreSQL databases, WebSocket streams, a monitoring stack. A sandbox for building and a cluster to deploy into. Two nodes on Vultr, a load balancer, still under-provisioned.
My daughter, Camille Rose McKenney, arrived on February 15th. My new machine arrived three days earlier, on the 12th—128GB of RAM, two 1TB NVMes. I'd been coding far more than usual since Claude Code came out and wanted a proper machine to develop on, self-host services, run AI workloads. Some people assemble cribs. I bought a computer that could run local LLMs.
I didn't touch it until we got home from the hospital on the 18th. It turns out that sleepless nights and the slow, bleary days of feeding and consoling a newborn are a surprisingly good time to configure a machine. You're awake anyway. One hand is usually free. And there's something meditative about Linux sysadmin at 3am with a baby on your chest.
This series is about migrating that cluster to a VM on my desk. The networking, the deployment pipeline, the monitoring, and every gotcha along the way.
The VM
I didn't want to run k3s directly on my workstation. Workstations accumulate state. Package managers fighting each other, orphaned config files, the quiet entropy that makes you consider a fresh install every six months. A VM is a boundary. Break it, delete it, start over. Nobody gets hurt.
There's also the security question. I'm self-hosting something open to the internet, on a machine in my house. A VM is an isolation layer. If something goes sideways, the blast radius stops at the VM boundary, not at my NVMe full of family photos. More on how traffic actually reaches it in Part II.
The setup is libvirt/QEMU with an Arch Linux cloud image:
# Host prerequisitessudo pacman -S libvirt qemu-full virt-install dnsmasq edk2-ovmf# Create the VM (setup-host.sh handles image download, chroot config, virt-install)sudo ./setup-host.sh
The VM is called "rookery"—a nesting colony. Crowprose is my pseudonym, so the infrastructure inherits the bird theme.
Why btrfs
The VM's disk is a qcow2 image sitting on a btrfs subvolume at /var/lib/libvirt/images/. I chose btrfs for one reason: thin provisioning.
Kubernetes is voracious about storage. Prometheus wants 25Gi. Loki wants 10Gi. Three PostgreSQL databases each want 10Gi. That's 65Gi of persistent volume claims before you deploy a single line of application code. On ext4, you'd need to actually have that space allocated upfront. On btrfs, the 500GB disk is thin-provisioned. It claims the space but only consumes what's been written. The actual disk usage is a fraction of the allocated total.
btrfs also gives you snapshots for free. I haven't used them yet, but my next project is a systemd service that snapshots the VM disk before running system updates on some cadence. Cheap rollbacks for when pacman -Syu inevitably breaks something.
The Gotchas
Setting up a VM should be straightforward, but I ran into a few problems that aren't well documented anywhere.
Docker vs. libvirt iptables. Docker sets the iptables FORWARD chain default policy to DROP. This secures Docker's own container networking, but it's a global policy change that breaks anything else on the system that relies on packet forwarding. libvirt's NAT networking is one of those things. Your VM boots, gets an IP address via the virtual bridge, and cannot reach the internet. ping 8.8.8.8 just hangs.
This is a well-known issue that's been filed as a Debian bug, discussed on the Arch forums, and affects anyone running Docker alongside bridge-based virtualization on any Linux distro. The problem is documented, but scattered across forum threads and bug trackers. There's no single canonical "if you run Docker and libvirt together, do this" page.
My fix is a systemd service that inserts forwarding rules after both Docker and libvirt start:
# /etc/systemd/system/libvirt-forward.service[Unit]Description=iptables rules for libvirt VM forwarding (Docker compat)After=docker.service libvirtd.service[Service]Type=oneshotRemainAfterExit=yesExecStart=/usr/bin/iptables -I FORWARD -i virbr0 -o wlp192s0 -j ACCEPTExecStart=/usr/bin/iptables -I FORWARD -i wlp192s0 -o virbr0 \-m state --state RELATED,ESTABLISHED -j ACCEPT[Install]WantedBy=multi-user.target
Two rules: allow packets from the VM bridge to the host's NIC, and allow return traffic for established connections. The -I flag inserts them at the top of the FORWARD chain so they're evaluated before Docker's DROP policy takes effect.
Cloud-init doesn't enable sshd on Arch. Cloud-init on Arch Linux cloud images doesn't reliably start the SSH daemon. You write your cloud-init YAML, boot the VM, and sit there wondering why SSH hangs. The VM is running. The network is up. sshd just isn't.
The workaround is to skip cloud-init for SSH configuration entirely and use qemu-nbd to mount the disk image before first boot:
qemu-nbd -c /dev/nbd0 "$VM_DISK"mount /dev/nbd0p3 /mntmount --bind /dev /mnt/devmount --bind /proc /mnt/procmount --bind /sys /mnt/syschroot /mnt /bin/bash# Now you're inside the image filesystemsystemctl enable sshd# Add your SSH key, create users, etc.
It's not elegant. But it works every time, which is more than cloud-init can claim.
systemd-time-wait-sync. On a fresh Arch VM, systemd-time-wait-sync.service waits for NTP to synchronize before letting dependent services proceed. If NTP isn't reachable yet (say, because the network is still coming up) it waits. Indefinitely. Your VM boots to a state where sshd is enabled but blocked behind a service that's waiting for a clock that will never sync because the thing that would sync it is also waiting.
sudo systemctl mask systemd-time-wait-sync.service
One line. Forty-five minutes of systemctl list-jobs to diagnose. Classic Linux.
k3s
With the VM running and reachable, k3s is the easy part:
curl -sfL https://get.k3s.io | sh -s - \--disable traefik \--disable servicelb \--write-kubeconfig-mode 644
The --disable flags are deliberate. Traefik and ServiceLB are k3s defaults for ingress and load balancing. I don't need either. Cloudflare Tunnel handles all ingress, and on a single node there's nothing to load balance across.
Copy the kubeconfig to your host, fix the server address, and you've got kubectl from your desk:
scp joe@<VM_IP>:/etc/rancher/k3s/k3s.yaml ~/.kube/k3s-configsed -i 's/127.0.0.1/<VM_IP>/' ~/.kube/k3s-configexport KUBECONFIG="$HOME/.kube/k3s-config"
What's Running
$ KUBECONFIG=~/.kube/k3s-config kubectl get pods -ANAMESPACE NAME READY STATUScloudflared cloudflared-96bbd6b5d-9pzzg 1/1 Runningdefault agent-postgres-postgresql-0 1/1 Runningdefault agent-service-65659595ff-4vgxg 1/1 Runningdefault api-service-56d86b87d5-4fh7g 1/1 Runningdefault app-web-65bbc6d5cb-tb5f9 1/1 Runningdefault tables-postgres-postgresql-0 1/1 Runningdefault tables-service-0 1/1 Runningdefault user-service-74bf69656d-xf9xg 1/1 Runningdefault users-postgres-postgresql-0 1/1 Runningdefault www-web-6b6f6db4ff-6tvl8 1/1 Runningflux-system helm-controller-68578f8447-hzqbr 1/1 Runningflux-system kustomize-controller-7ddfbb5875-rdn2f 1/1 Runningflux-system source-controller-6679d8bdb-xck5j 1/1 Runningmonitoring loki-0 2/2 Runningmonitoring monitoring-grafana-847dfbc55d-r4lwl 3/3 Runningmonitoring monitoring-kube-prometheus-operator-... 1/1 Runningmonitoring monitoring-kube-state-metrics-... 1/1 Runningmonitoring monitoring-prometheus-node-exporter-zct2v 1/1 Runningmonitoring prometheus-monitoring-kube-prometheus-... 2/2 Runningmonitoring promtail-jcfgc 1/1 Running
Twenty pods. The whole thing idles at around 15% CPU and 8GB of memory. Most of that is Prometheus.
Could I have done this with Docker Compose on the bare metal? Absolutely. Would it have taught me anything about Kubernetes networking, GitOps, or debugging iptables chains? No. The whole point of the Vultr cluster was to learn this stuff, and now the whole point of the VM is the same thing, minus the monthly bill. Some lessons only stick when they cost you an evening. I have fewer of those to spare now, so I'd rather spend them at home.