Chasing faster local LLM inference on Apple Silicon — the wins, one spectacular near-miss, and the bug I almost walked away from. There's a specific impatience that only shows up once you run language models locally. The cloud spoils you rotten, then you pull everything in-house and watch a
Or: how I stopped letting a laptop sleep on the embedding job. The previous setup ran a self-hosted RAG stack on a laptop. Open WebUI in Docker, Chroma for vectors, Tika for extraction, Infinity for embeddings, and an LLM running on the same box via the local server. It worked.
A while back I wrote about teaching my servers to reboot themselves. That post ended with a little FastAPI thing I'd built on top of the Ansible playbooks — host inventory, a terminal overlay streaming output over SSE, some big buttons labelled UPDATE and REBOOT. I called it Herdmon.
proxmox
Or: the day a cluster tried to heal itself to death. The Silence It started with Plex. Then Paperless. Then Pocket ID. Uptime Kuma started quietly lighting up like a Christmas tree. Services on one node had stopped answering, but the node itself was... fine? A ping to one of
homelab
It started, as these things always do, with "I can't get to Karakeep." Not a server crash. Not a disk failure. Not even a misconfigured reverse proxy. Just a bookmark manager that worked yesterday and didn't work today. The kind of problem that should
homelab
The last post ended with the model scoring 80% and confidently telling me that Plex -- the service running on my primary Proxmox node, mentioned hundreds of times in the training data -- does not exist. It was the best score yet and also the most embarrassing failure. Like acing
homelab
The last post ended with a cliffhanger: V8 was training, the dataset had been surgically rebuilt with 140 hand-written gold pairs, and the target was 76-82%. Three days, several explosions, and one strongly-worded email later, the model hit 80%. Ten points above V7. Best score yet. Getting there was the
homelab
The last post ended with a model that scored 46% and the profound insight that throwing more parameters at a problem is not, in fact, a substitute for having good training data. Revolutionary stuff. Nobel committee, you know where to find me. What followed was about 24 hours of increasingly
homelab
This is the boring sequel. The 9B model trained on the first attempt. No SSH failures, no disk full errors, no cascading tracebacks. The custom Docker container did its job. I launched the pod, it pulled the image, SSH connected, training started, and 50 minutes later I had a model.
homelab
Last time I left you with a model training on an H100 in the Netherlands. The loss was dropping. Everything looked promising. Then the GGUF export failed. Then the 9B model couldn't download because the disk was full. Then my terminal filled with cascading Python tracebacks that my
homelab
Two days ago I had a fine-tuned model that scored 87% on a 15-question test. It knew my IPs, my VMIDs, and most of my backup retention policy. Good enough, right? No. Obviously not. Because I looked at the training data and realised half the answers didn't even
homelab
At the end of my last post, I had a working search engine for my homelab docs. Point qmd at the markdown files, let Claude search them semantically, job done. I even hinted at what might come next. What actually came next was a Saturday night rabbit hole. I was