Or: the day a cluster tried to heal itself to death. The Silence It started with Plex. Then Paperless. Then Pocket ID. Uptime Kuma started quietly lighting up like a Christmas tree. Services on one node had stopped answering, but the node itself was... fine? A ping to one of
It started, as these things always do, with "I can't get to Karakeep." Not a server crash. Not a disk failure. Not even a misconfigured reverse proxy. Just a bookmark manager that worked yesterday and didn't work today. The kind of problem that should
The last post ended with the model scoring 80% and confidently telling me that Plex -- the service running on my primary Proxmox node, mentioned hundreds of times in the training data -- does not exist. It was the best score yet and also the most embarrassing failure. Like acing
The last post ended with a cliffhanger: V8 was training, the dataset had been surgically rebuilt with 140 hand-written gold pairs, and the target was 76-82%. Three days, several explosions, and one strongly-worded email later, the model hit 80%. Ten points above V7. Best score yet. Getting there was the
The last post ended with a model that scored 46% and the profound insight that throwing more parameters at a problem is not, in fact, a substitute for having good training data. Revolutionary stuff. Nobel committee, you know where to find me. What followed was about 24 hours of increasingly
This is the boring sequel. The 9B model trained on the first attempt. No SSH failures, no disk full errors, no cascading tracebacks. The custom Docker container did its job. I launched the pod, it pulled the image, SSH connected, training started, and 50 minutes later I had a model.
Last time I left you with a model training on an H100 in the Netherlands. The loss was dropping. Everything looked promising. Then the GGUF export failed. Then the 9B model couldn't download because the disk was full. Then my terminal filled with cascading Python tracebacks that my
Two days ago I had a fine-tuned model that scored 87% on a 15-question test. It knew my IPs, my VMIDs, and most of my backup retention policy. Good enough, right? No. Obviously not. Because I looked at the training data and realised half the answers didn't even
At the end of my last post, I had a working search engine for my homelab docs. Point qmd at the markdown files, let Claude search them semantically, job done. I even hinted at what might come next. What actually came next was a Saturday night rabbit hole. I was
Rebooting a single server is easy. Rebooting three servers that are collectively running 32 containers, 5 virtual machines, a media stack, two reverse proxies, and a monitoring system -- without anyone noticing -- is a different kind of problem. My Proxmox cluster has three nodes. Between them they run everything
My wife has a robot vacuum. We named it Treat, after our daughter's hamster who passed away not long ago. Treat the hamster had a habit of climbing over absolutely everything in his path -- furniture, hands, the side of his cage, didn't matter. He'
I have 35 markdown files documenting every service in my homelab. Caddy configs, Proxmox cluster notes, AI agent skills, firewall rules, Plex migration war stories -- the lot. They live in a private GitHub repo, and they're genuinely useful. When they're open in front of me.
homelab
I run a self-hosted AI agent on my homelab. It reboots crashed containers, monitors uptime, sends me WhatsApp messages, and even claims my free coffee. It needs API keys and passwords to do all of this. It's only been running for a couple of weeks, but I caught
homelab
Every week, Octopus Energy gives Octoplus members a free coffee from Caffe Nero or Greggs. New codes drop daily before the coffee shops open, and are gone within minutes. Claim one and you're sorted for the week. I got my AI agent to grab mine. The Problem The
fueltrack
The UK government quietly runs one of the most useful APIs nobody talks about. Every petrol station in the country reports its prices to the GOV.UK Fuel Finder, and they expose all of it through a public REST API. Around 7,400 stations, updated within 30 minutes of price
There's a particular kind of laziness that loops back around to being productive. It's the kind where you spend three hours automating a task that takes five minutes, then feel like a genius every time those five minutes happen without you. This is a story about
This is where I document my homelab adventures - from Proxmox clusters and LXC containers to reverse proxies and automation pipelines. If it runs on a server rack at home, it probably belongs here. What to expect Posts here will cover the real stuff - the configs that worked, the