The storm took the fibre out for six days and the thing that was most quietly missed was not streaming. It was the ability to ask a question and get a useful answer — how long does that medication keep, what is the conversion for this, how do I word this letter to the insurer. All of that had quietly migrated to a website, and the website needed a connection that no longer existed. A model running on the drive in the next room does not.
What it actually takes
The specification that decides everything is VRAM — the memory on the graphics card. A model has to fit in it to run at a usable speed. Spill over, and it falls back to system memory and slows to a crawl.
- 6 to 8 GB — small models, around 7 or 8 billion parameters, quantized. Genuinely useful for writing, summarising and explaining.
- 12 GB — comfortable with those, and opens up models in the 13 billion range.
- 16 GB — the practical threshold. Most of what a normal person wants runs well here.
- 24 GB and up — large models, and the point at which quality stops being the constraint.
No dedicated graphics card at all? It still works, on the processor, just slowly — words appear at reading speed rather than instantly. On a machine with 16 GB of system memory that is perfectly tolerable for drafting.
Storage: budget 5 to 50 GB per model. They are large files.
“Quantized” and why every filename says Q4
A model’s numbers are originally stored at high precision. Quantization stores them at lower precision, which shrinks the file dramatically for a small quality cost.
- Q8 — near the original, large.
- Q5 and Q6 — the quality sweet spot.
- Q4 — the standard choice. Roughly a quarter of the size, and the difference is hard to notice in ordinary use.
- Q2 and Q3 — small and noticeably worse. Only if nothing else fits.
The practical rule: a bigger model at Q4 beats a smaller model at Q8, almost every time.
Getting it running
This used to require assembling things from source. It does not anymore.
The simplest path — LM Studio. A normal application with a normal installer, for Windows, Mac and Linux. It has a search screen, it tells you whether a model will fit your hardware before you download it, and it gives you a chat window.
The command-line path — Ollama. Install it, then one line: ollama run llama3. It downloads and starts. Lighter, and it exposes a local API that other software can talk to.
Both store the model on your drive and both run with the network cable out.
What to download first
Names change constantly, so pick by category rather than brand:
- A general 7–8B instruct model at Q4 — this is the default. Writing, explaining, summarising, rewording.
- A small model, 3B or under, if your hardware is modest. Noticeably weaker, still useful for drafting.
- A code model if you write any. They are markedly better at it than general models of the same size.
Download them before you need them. A model you intended to fetch is not a local model.
What it is honestly good at
- Explaining something in plainer words
- Drafting a letter, a complaint, an email you are dreading
- Summarising a long document you paste in
- Conversions, rough calculations, unit and recipe maths
- Working through a decision out loud
- Translating, roughly
And what it is not
- Anything current. It knows nothing past its training date.
- Anything about you. Your documents, your accounts, your messages.
- Medical, legal or financial decisions. It will answer confidently and may be wrong, with no way for you to tell.
- Precise facts. Dates, figures, names of people — these are where models invent most readily.
The resilience argument
Most of what people now rely on sits behind a connection: search, translation, reference, the ability to draft something. All of it works until the line goes down, which it does during exactly the events where you want it most.
A model on a drive is about twenty gigabytes and it does not care about the connection, the subscription, or whether a company changed its terms. It is the same argument as the paper map in the car. Not better than the online version on a normal day. Available on the day that is not normal.
