# The 2026 Homelab AI Stack: What's Worth Running
After an autonomous agent campaign spent a weekend inside its infrastructure, Hugging Face closed its [incident disclosure](https://huggingface.co/blog/security-incident-july-2026) with a recommendation: "have a capable model you can run on your own infrastructure vetted and ready **before** an incident." A hosted model's guardrails can refuse to work with exploit code at the moment forensics needs it most. I covered the breach itself in [The Autonomous Agent Attack Surface](https://chuckworks.hashnode.dev/the-autonomous-agent-attack-surface).
That reads like an enterprise concern. It's also the best argument for running a local model at home that I've come across, because whether a local model is worth it depends entirely on what you need it to do when the hosted one isn't an option.
I run Ollama in my homelab. It started as the fallback for the publishing pipeline I built for this blog, and the first thing it taught me was about size.
## What Small Means
The pipeline called a hosted model first and fell back to a local one when the provider failed. The local models were the small end of what Ollama offers, in the 0.5 to 1 billion parameter range, picked for cheap, fast testing of the pipeline's wiring.
When a configuration error sent a real request to that fallback, it returned valid JSON that had nothing to do with the request. I wrote about the failure in [Sora's Shutdown and the Hidden Cost of AI API Dependency](https://chuckworks.hashnode.dev/soras-shutdown), and the silent fallback was only half the bug. The other half was handing a model that size a job it was never going to do. It could follow the format. It couldn't do the work.
A local model is worth running when it's sized to a job, not to whatever hardware happens to be free.
## What a Local Model Is For
**Data that shouldn't leave**. Inference on your own hardware means the prompt, and whatever document you pasted into it, never goes to a provider. For personal financial records, medical paperwork, or anything covered by an agreement with an employer, that's the whole argument, and it doesn't depend on the model matching a hosted one. It only has to be good enough for the task.
**Independence from someone else's decisions.** A hosted model can be retired on six months' notice, repriced, or retuned to refuse things it used to do. A model file on your disk does what it did yesterday. That's the Sora lesson, and Hugging Face's.
**Testing without a meter running.** Most of the pipeline's local calls were never meant to produce good output. They exercised the pipeline's routing, retries and logging without spending money on a hosted API, and a tiny model is fine for that.
**Incident response**, in Hugging Face's sense: a model you've already tried on hostile material, on infrastructure you control, ready before the day you need it.
What it isn't for, on most home hardware, is replacing a frontier model on work that needs judgment. The models a homelab can run are not the ones you rent by the token, and the gap shows up exactly where output that **looks right** and output that **is right** part ways.
## The Plumbing Is the Easy Part
Ollama exposes an [OpenAI-compatible API](https://docs.ollama.com/api/openai-compatibility), including the chat completions endpoint, so anything written against OpenAI's format can talk to it by changing a base URL. That's why my pipeline's fallback needed no second client: the same request shape went to the hosted provider or to the machine at home.
The same documentation notes that Ollama ignores the API key a client sends. The local API doesn't authenticate callers, so it belongs on the inside of the network, behind the same reverse proxy and identity layer as every other service, and so does any chat front end you put in front of it, such as Open WebUI. In [Five Security Lessons My Homelab Taught Me](https://chuckworks.hashnode.dev/five-security-lessons) I argued that identity is infrastructure, not an afterthought. A model endpoint that answers anyone on the network is one more service that needs it.
## Where I'd Start
Pick the job first. Write down the tasks you'd want a local model for, pull a few models of different sizes, and run each one against your real inputs, not a benchmark, before trusting any of them. Keep the results. A model you've tested on your own work is worth more than a larger one you haven't.
Then size the hardware to the job, not the other way around.
Here's what that looks like in mine. Most of the lab is Raspberry Pis and a pair of older laptops with 8 GB of memory, none of which can run a useful model, so Ollama lives on a spare desktop with Open WebUI in front of it. It does three jobs. It's the fallback when a hosted provider like Anthropic is unreachable because my own connection is the thing that's down. It takes the low-stakes automated work, like routine merge request reviews from Glint (the reviewer that comments on my GitLab merge requests) so the hosted model's budget goes to the reviews that need judgment. And it backs the notebook and bookmark tools that summarize and tag what I save, which is exactly the kind of personal data I'd rather not send anywhere. The next job on the list is the incident-response one: a model I've already fed my own CrowdSec ban logs, so I know what it does with hostile input before the day it matters.
The question isn't whether a homelab can run AI (it can), it's whether the model you'd reach for in a bad moment has been tested on the work you'd hand it. Hugging Face's advice was to settle that before the incident. It's cheaper to settle it at home.
---
*Charles Smith is a solutions architect at a large financial institution. He also runs a self-hosted homelab stack: fewer constraints, same discipline. He writes about platform engineering, automation, and the gap between what technology enables and what organizations can actually ship.*