You can ask a chatbot almost anything today, and most of the time the answer is fine. The part that bothers people is where the question goes. Paste a contract clause, a medical letter, a half-finished resignation email, and that text lands on a server you don't own, governed by a policy you probably haven't read. Running a model on your own laptop changes that arrangement completely. The conversation stays on the disk in front of you, and you can prove it by turning off Wi-Fi.
This used to require a workstation with a serious graphics card. It no longer does. A mid-range laptop from the last three or four years will run a useful assistant, provided you pick the right model size and set expectations accordingly.
What "local" actually means
The word gets used loosely. An app that advertises private chats may still send every prompt to a server and simply promise not to keep it. That's a policy promise, not an architectural one. True local means three specific things: the model weights sit on your drive, the computation happens on your own hardware, and nothing in the request path touches the network.
The last point is testable, which is what makes it valuable. Load a model, disable Wi-Fi, ask a question. If a coherent answer comes back, the setup is genuinely offline. Run that check before you trust any tool with a document you wouldn't email to a stranger.
The honest trade-off: small models on consumer hardware are less capable than the largest hosted ones. For summarizing a report, drafting code, rewriting an email, or searching your own notes, they hold up well. For long chains of reasoning across fifty pages, you'll notice the gap.
Hardware: what you actually need
Two numbers decide everything: how much memory the model needs, and how quickly your machine can move data through it. Everything else is comfort.
Rough tiers
- 8 GB RAM, recent CPU, no dedicated GPU: 3B-parameter models at 4-bit quantization. A few words per second. Fine for short questions, painful for long documents.
- 16 GB RAM, integrated graphics: the practical sweet spot for 7B–8B models at 4-bit. Response speed depends heavily on whether your runtime supports your integrated GPU.
- 16 GB or more of dedicated or unified memory: 13B–30B models at 4-bit, running at conversational speed.
- 32 GB unified memory or 24 GB VRAM: larger models and long context windows without constant compromise.
The memory math
A useful rule of thumb: a 7B model at 4-bit quantization needs roughly 4–5 GB for the weights alone, plus another gigabyte or two for the context window and runtime overhead. A 13B model at 4-bit lands near 8 GB. Context matters more than people expect — asking a model to hold a long conversation or a big document can add a gigabyte or more on top.
If your total usage creeps within a gigabyte of your machine's RAM, the operating system starts swapping to disk and the experience falls apart. Better to run a smaller model that fits comfortably than a larger one that stutters. Model files themselves are typically 2–8 GB each, so keep them on an SSD and plan to hold two or three rather than a dozen.
Choosing a model that fits your machine
You don't need to evaluate dozens of options. Pick one small and one mid-sized model from a well-supported family — Llama, Mistral, Qwen, Gemma and Phi all have active communities and runtimes that handle their formats cleanly.
Sizes and quantization
Model names usually carry a parameter count (3B, 8B, 13B) and a quantization tag. Quantization compresses the weights from 16-bit numbers to 4- or 5-bit ones, cutting memory use by roughly a factor of three or four in exchange for a modest quality loss. Q4_K_M is a common, sensible default. Q5 and Q8 stay closer to the original but demand more memory.
If answers feel shallow, move up a size before moving up a quantization level. If everything feels slow, do the opposite.
Instruction-tuned, not base
Choose an "instruct" or "chat" variant of whichever model you pick. Base models simply continue text; instruct models answer questions and follow formatting. Load the wrong one and you'll conclude local models are useless when the real problem was the file you downloaded.
Setting it up, step by step
The quickest route is a runtime that handles downloading, quantization formats and GPU offloading for you. Ollama is command-line first and popular for scripting. LM Studio and Jan give you a graphical app with a built-in model browser. llama.cpp sits underneath several of these as the low-level engine.
- Install your chosen runtime from its official site, not from a random mirror.
- Pull a small instruction-tuned model — something in the 3B range — and let it download.
- Ask it something you already know the answer to. Watch the reported tokens-per-second figure.
- Turn off Wi-Fi and ask again. An answer here is your proof that nothing is leaving the machine.
- If memory allows, download a 7B or 8B model and compare quality on a real task from your week.
- Tune the settings that matter: context length, how many layers run on the GPU, and temperature. Lower temperature makes output more predictable; higher makes it more varied.
Start with the smallest model that can do the job, then grow. Most disappointing local setups fail because someone loaded a model their laptop couldn't feed fast enough.
Adding a chat window and your own files
Command-line tools are fine for testing, but a proper interface makes daily use realistic. The graphical runtimes include one. Several also support pointing the assistant at a folder of notes or PDFs: the tool splits the text into chunks, converts them into numeric representations on your own machine, and feeds the most relevant pieces into the prompt before each answer.
That retrieval step stays local too, which is the whole point. A model that can search your meeting notes without those notes ever leaving your laptop is a genuinely different tool from one that can't. The catch is storage: you're now keeping an index of private material on disk, so full-disk encryption stops being optional.
Privacy trade-offs nobody mentions
- Telemetry. Many runtimes check for updates or send anonymous usage data by default. Usually disableable, rarely disabled.
- Your disk is the new boundary. Chat logs often sit in plain text, searchable by anyone with access to the machine. Encryption matters more, not less.
- Silent fallbacks. Some applications route to a cloud service when the local model fails or when you use a specific feature. Check the settings.
- Download sources. Model files are large and easy to tamper with. Stick to official repositories.
- No safety layer. A hosted service will refuse certain requests; a local model generally won't. That's freedom and risk in the same package, and it's yours to manage.
A practical way to start
Spend an evening on it. Install one runtime, pull a 3B instruction-tuned model, and use it for a week on low-stakes tasks: summarizing an article you were going to skim, rewriting a paragraph, converting notes into a checklist. That tells you whether the quality and speed suit you far better than any benchmark table.
Then decide whether to invest further — a larger model, a second one tuned for code, or a folder index over your documents. Keep the offline test in your routine: every so often, disconnect and ask something. If it answers, the setup is doing what you built it for.
Photo: fancycrave1 / Pixabay
