Tech

Running AI locally on a laptop in 2026: what actually works, and what is still a demo

Local models have got genuinely good and the hardware has caught up unevenly. We ran the current crop on three machines to find where the line sits between useful and technically impressive.

By Kavita Iyer

Published June 16, 2026 · 10 min read

The bottom line

Local language models crossed a usability threshold in the past year for a narrow set of tasks — summarising, rewriting, structured extraction and code completion — on machines with unified memory or a recent discrete GPU. They remain poor substitutes for hosted frontier models on anything requiring broad knowledge or long reasoning chains. The binding constraint is memory bandwidth rather than raw compute, which is why an Apple Silicon laptop with 32GB frequently outperforms a gaming laptop with a faster GPU and less unified memory. If you want local AI for privacy, it works today for document work. If you want it to replace a subscription, it does not.

Running a capable language model on your own laptop stopped being a party trick some time in the last year. It did not stop being complicated.

We spent six weeks with the current tooling across three machines — an Apple Silicon laptop with 32GB of unified memory, a Windows laptop with a recent discrete GPU and 16GB of VRAM, and an older machine with 16GB of system memory and no useful GPU — to work out where local models are genuinely useful and where they remain a demonstration.

The short version

Local models are now good at a narrow, well-defined band of work: summarising text you supply, rewriting in a different register, extracting structured data from messy input, and code completion. On a machine with enough memory these are fast enough to sit inside a workflow rather than interrupt it.

They remain weak at anything that needs broad world knowledge, current information, long chains of reasoning, or arithmetic you would rely on. That gap has not closed much, and framing it honestly matters: the tasks local models do well are the ones they were already adequate at, and they have got considerably better at them.

The specification that actually predicts performance

The intuition most people bring is that a faster GPU means faster local inference. It is roughly wrong, and this is the single most useful thing we learned.

Generating each token requires reading the model’s weights from memory. Throughput is therefore bounded by memory bandwidth, not by arithmetic throughput. A processor capable of far more operations per second sits idle waiting for weights to arrive.

Two consequences follow, both counterintuitive:

Unified memory architectures punch above their benchmark scores. An Apple Silicon laptop with 32GB can load and run a mid-size model at usable speed, while a Windows laptop with a nominally faster GPU and 8GB of VRAM cannot load the same model at all. Capability is gated by whether the weights fit, and only then by how fast they can be read.

Quantisation helps twice. Reducing weight precision shrinks the model so it fits, and shrinks the volume of data read per token so it runs faster. Four-bit quantisation of a mid-size model is the current sweet spot: the quality loss is small enough to be hard to notice on document tasks, and the speed gain is substantial.

What the memory tiers get you

16GB is the floor. Small models run, and they are noticeably weaker — more repetition, more instruction drift, more confident nonsense. Useful for autocomplete, frustrating for anything else.

32GB is where this becomes comfortable. Mid-size quantised models run at conversational speed and handle document work properly. For most people this is the tier worth buying.

64GB opens the larger open-weight models. The quality improvement is real but smaller than the memory jump suggests, and for typical document work it is not the difference between working and not working.

The tools

Three pieces of software cover essentially everyone.

Ollama is the simplest path. One command pulls a model and runs it, and it exposes a local API that other applications can talk to. If you want this working in ten minutes, start here.

LM Studio wraps the same territory in a graphical interface with model browsing, parameter controls and a chat window. Better for exploring what runs well on your hardware without reading documentation.

llama.cpp is the engine underneath much of this and the right choice if you want to tune quantisation, context length and memory layout yourself. More work, more control.

Where it genuinely wins

Privacy, without qualification. A local model sends nothing anywhere. No prompt logging, no retention policy, no question about training on your inputs, no exposure if a provider changes its terms. For client documents, medical records, legal drafts or unreleased material, this is not a preference — it is often the only compliant option, and it is the strongest argument for the whole exercise.

Cost at volume. If you are processing thousands of documents, per-token pricing adds up and local inference is a fixed cost you have already paid.

Working offline. On a plane, on bad hotel wifi, in a facility that blocks external services. It simply works.

Latency on short tasks. No network round trip. For autocomplete and quick rewrites, local often feels faster even when raw throughput is lower.

Where it does not

Broad knowledge questions. A local model’s knowledge is smaller, staler and less reliable, and it is no better at signalling when it does not know.

Long reasoning. Multi-step problems degrade faster than the benchmark differences suggest.

Anything current. No web access unless you build it, and building it well is a project.

Convenience. Model management, quantisation choices, context limits and occasional breakage after updates are real ongoing costs. A subscription has none of them.

If privacy is a requirement, run locally today for document work — it is good enough and the alternative may not be permissible.

If you are curious, install Ollama and spend an afternoon. The floor is much higher than it was a year ago and the experience is informative regardless of what you conclude.

If you are hoping to cancel a subscription, do not buy hardware for it yet. The tasks a hosted frontier model does that a local one cannot are precisely the ones most people are paying for, and no amount of memory closes that particular gap.

Frequently asked questions

Is a local AI model good enough to replace ChatGPT?

For a narrow band of tasks, yes; for general use, no. Summarising a document you paste in, rewriting text in a different register, extracting structured data from messy input, and code completion all work well locally on current mid-size models. What does not work is anything needing broad world knowledge, current information, long multi-step reasoning, or reliable arithmetic. The gap is not closing evenly either — local models have improved most on the tasks they were already adequate at. If your use is document-shaped and privacy matters, local is genuinely viable now. If you use a hosted model as a general-purpose assistant, it is not a substitute.

How much RAM do you need to run AI models locally?

Sixteen gigabytes is the practical floor and constrains you to smaller models that are noticeably weaker. Thirty-two is where it becomes comfortable and where mid-size quantised models run at usable speed. Sixty-four opens up the larger open-weight models, with diminishing returns for most people's actual use. The subtlety is that unified memory architectures let the model use most of the pool, while a discrete GPU is limited to its own VRAM — so a 32GB Apple Silicon machine can often run a model that a laptop with a faster GPU and 8GB of VRAM cannot load at all.

Why is my local AI model so slow?

Almost always memory bandwidth rather than compute. Generating each token requires reading the model's weights, so throughput is bounded by how fast memory can be read, not by how many operations the processor can perform. This is why the specification that predicts local model speed best is memory bandwidth, and why two machines with similar benchmark scores can differ by several times in tokens per second. Quantisation helps directly by shrinking the weights that have to be read.

Is running AI locally actually more private?

Yes, and it is the strongest reason to do it. A model running on your machine sends nothing anywhere. There is no prompt logging, no retention policy to read, no question about training on your inputs, and no dependency on a provider's terms surviving the next funding round. For anyone working with client documents, medical records, legal drafts or unreleased material, that is not a preference but a requirement. The trade is capability, and for document work the trade is now a reasonable one.

← More Tech coverage