What a "small language model" actually is
A large language model (LLM) is what powers ChatGPT and its peers — hundreds of billions of parameters, living on racks of GPUs in a data center. A small language model (SLM) is the same underlying idea, sized down to run somewhere you can hold: a laptop, a phone, a car head-unit, a robot. In practice, "small" usually means a few billion parameters or fewer, and the model's weights are a file measured in low gigabytes rather than terabytes.
SLMs matter for exactly one reason: where they run. A model that fits on a phone can be private (nothing travels), free-per-message (no server bill), and offline (no network needed). Those three properties don't exist in cloud AI, at any size. The only question is whether the small model is good enough at the task — and for a growing list of tasks, the answer is finally yes.
Why on-device AI was almost impossible until recently
The dream of running a real language model on a phone bumped into three walls at once:
- Memory. A modern LLM's weights are big; on a phone you're fighting the OS, the browser, the game the user just closed, and the low-memory killer looking for something to evict. A model that just barely loads on a phone is a model that dies the first time the user gets a notification.
- Compute. Transformer inference is a stream of matrix multiplies. Phone GPUs are surprisingly good at this — but only if the model's tensors fit their per-buffer limits, and only if the software stack (LiteRT, MediaPipe, MLC, llama.cpp mobile) actually hits the GPU instead of falling back to CPU.
- Battery and thermals. A phone's chip can run flat-out for a burst, but sustained load turns into heat, and heat turns into thermal throttling. A model that answers the first message in two seconds and the tenth in twenty is a demo, not a product.
Every previous "small model on your phone" story broke on one of these three. Gemma 4 is the first family that clears all three at once — and it clears them because of two architectural choices worth knowing about.
The Gemma family, briefly
Google's Gemma models are the open-weight relatives of Gemini — smaller, released for anyone to run, but built with lessons from training the big ones. The lineage:
- Gemma 1 and 2 — solid open-weight models, but essentially small versions of a standard transformer. They ran fine on a beefy laptop and stumbled on a phone.
- Gemma 3 — added vision, better multilingual coverage, still standard architecture.
- Gemma 3n — a mobile-first spin-off. Introduced Per-Layer Embedding (PLE) and the matformer nested-parameter design, and coined the "E2B / E4B" naming.
- Gemma 4 — the on-device–first mainline. Inherits PLE and matformer, adds better instruction-following, better vision, better multilingual performance, and a chat-tuned variant that's usable out of the box (E2B-it, E4B-it).
The important thing about Gemma 4 isn't a benchmark score. It's that the architecture was designed backward from "must run well on a mid-range phone", not forward from "must beat GPT-4 on paper".
PLE: the trick that let 2B parameters fit on a phone GPU
Here's a specific, concrete constraint that most people never hear about. Many popular Android phones — a Samsung Galaxy S23 Ultra, for one — expose a maximum WebGPU storage buffer of 128 MB per binding. That's the size limit of a single tensor the GPU driver will accept. Overshoot it and the driver rejects the binding, and the model silently outputs garbage or falls back to CPU.
In a standard transformer, the biggest single tensor is the embedding table: vocabulary size × hidden dimension. For a modern multilingual model this table alone can be 150–260 MB, even after INT8 quantization. In other words: on a huge fraction of real phones, the embedding table won't load. The model can't run on the GPU. CPU fallback works but is 10–20× slower — a system prompt takes twenty seconds to prefill instead of one.
Per-Layer Embedding is Gemma 4's answer. Instead of one enormous embedding table, embeddings are split across layers, so each tensor is comfortably below the 128 MB ceiling. The overall parameter count is similar, but the shape is different — and shape is what actually determines whether the model runs on real hardware.
Matformer: what E2B and E4B actually mean
The E in E2B and E4B stands for Effective. An E2B model behaves like a dense 2-billion-parameter model in the tasks that matter; an E4B like a dense 4B. But under the hood, they come from the same weights.
The mechanism is called matformer — think matryoshka dolls, but for neural networks. During training, the model is optimized so that its parameters form a nested hierarchy: you can slice the model at different depths and widths and get a smaller sub-model that still behaves coherently. From one master set of weights, Google can carve out E2B for phones with tighter memory and E4B for phones with more room. Both share training, both benefit from the same instruction-tuning; you're not choosing between two independent products, you're choosing where along a continuum your device sits.
How Gemma 4 compares to other small models
The 2B-and-under space is more crowded than most people realize. Here's an honest look at the shortlist, with the practical runs on a mid-range Android GPU? question front and center.
| Model | Params | Embedding size (INT8) | Fits 128 MB buffer? |
|---|---|---|---|
| Qwen3 0.6B | 0.6B | ≈ 155 MB | ✕ CPU fallback |
| Gemma 3 1B | 1B | ≈ 151 MB (even INT4) | ✕ CPU fallback |
| Llama 3.2 1B | 1B | ≈ 262 MB | ✕ CPU fallback |
| Qwen3 1.7B | 1.7B | ≈ 311 MB | ✕ CPU fallback |
| IBM Granite 4.0 350M | 0.35B | ≈ 102 MB | ✓ (small vocab, weaker multilingual) |
| Gemma 4 E2B (PLE) | Effective 2B | split per layer | ✓ full GPU |
| Gemma 4 E4B (PLE) | Effective 4B | split per layer | ✓ full GPU (with enough RAM) |
The story the table tells: parameter count is a proxy people fixate on, but on real phones the deciding factor is whether the biggest single tensor overflows the GPU's per-buffer limit. For today's Android landscape, PLE is the architecture that wins.
Where to actually try Gemma 4 today
On your phone: Gist&Chat (E2B, free)
Gist&Chat is a free Android app built around Gemma 4 E2B. First launch, it downloads the model on Wi-Fi — a one-time ~2.6 GB — and from then on the AI runs entirely on your phone. You get:
- Private chat — voice or text, memory across sessions, passes the airplane-mode test. Unlimited and free because your phone does the computing.
- Summaries — paste a YouTube link, an article URL, or a PDF / DOCX / photo of text, get the gist, then ask follow-up questions.
- 11 languages — including the ones you're actually likely to practice in.
If you want the shorter version of the "why on-device at all" argument, that's a separate piece. If you want the Gemma-4-specific version, it's this article — and it happens to be running the model on your side while you read it.
E4B on Pro (coming)
The larger Gemma 4 E4B variant is landing in Gist&Chat's upcoming Pro tier. E4B doubles the effective parameters — noticeably better at reasoning, longer-context Q&A, and code-adjacent tasks — but wants a phone with more RAM to stay smooth. Pro users get to pick the model tier that fits their hardware; free users continue to run E2B.
On a laptop or desktop
Prefer to tinker on a bigger machine? Google's Gemma 4 weights are available through the usual open-model ecosystem — Hugging Face, Ollama, LM Studio, Jan, GPT4All. We compare that whole desktop shelf against the mobile picture in offline ChatGPT alternatives: the real options in 2026.
The honest limits
A guide that skips this section is a press release. What Gemma 4 E2B won't match:
- Frontier reasoning. A 2B-effective model is not a 200B-parameter cloud model. If you're asking for graduate-level math derivations or novel research synthesis, cloud giants still win.
- Fresh knowledge. Weights are a snapshot of the training data. Gemma 4 doesn't know what happened yesterday; for that, you still want something with web access.
- Very long contexts, on very small phones. The model can handle long inputs, but the KV cache grows with context length; on a phone with 6 GB of RAM you'll run into memory before you run into the model's context limit.
- Weird outputs, sometimes. Compact models still occasionally hallucinate, mix up numbers, or repeat themselves. Treat their answers the way you'd treat a smart friend on a distracted afternoon: often right, worth double-checking on stakes.
What you get in return is the same three properties that make on-device AI worth caring about in the first place: your words don't travel, there's no meter, and the phone doesn't need to be connected to anything.
Try Gemma 4 on your phone, free
Gist&Chat ships Gemma 4 E2B out of the box — private AI chat plus YouTube, article and PDF summaries, in 11 languages. The larger E4B variant is coming to Pro.
- Gemma 4 E2B, on-device — full GPU inference on modern Android
- Private chat, unlimited & free — no account, no meter
- Summaries with follow-up Q&A — video, article, PDF, photo of text (free to start)
- 11 languages — interface and conversation
- E4B on Pro, coming — bigger effective model, for phones with the RAM
Free to download · One-time ~2.6 GB model download on Wi-Fi · Not a byte of your chat leaves the phone — privacy policy
Frequently asked questions
What is Gemma 4 and how is it different from earlier Gemma models?
Gemma 4 is Google's on-device–first small-model family. Unlike Gemma 1, 2 and 3 — which were mainly small versions of the same monolithic transformer — Gemma 4 uses a Per-Layer Embedding (PLE) architecture inherited from Gemma 3n. Instead of one huge embedding table, embeddings are split per layer, which keeps every individual tensor small enough to fit on the tight memory limits of a phone GPU.
What do the E2B and E4B in Gemma 4 mean?
The E stands for Effective. E2B is a model whose behavior is comparable to a dense 2B-parameter model, and E4B to a dense 4B. Under the hood, both are cut from the same matformer weights (a matryoshka-style nested-parameter design), so a device can pick the effective size that matches its RAM and heat budget.
Why does PLE architecture matter for on-device use?
Mid-range phone GPUs have hard per-buffer memory limits — a Snapdragon 8 Gen 2 phone, for instance, exposes only 128 MB per WebGPU storage buffer. A single monolithic embedding table (vocab size × hidden size) in most modern LLMs already blows past that. PLE splits the embeddings by layer so no single tensor overflows the limit; the model runs on the GPU instead of falling back to slow CPU inference.
Can I run Gemma 4 on my phone today?
Yes. Gist&Chat is a free Android app that ships Gemma 4 E2B out of the box: a one-time download, then the model runs entirely on your phone for private chat and for summarizing YouTube videos, articles and PDFs. The larger E4B variant is coming to Gist&Chat's Pro tier for higher-end phones with enough RAM to run it comfortably.
How does Gemma 4 E2B compare to other small models like Phi, Qwen, or Llama 3.2?
On paper the parameter counts look similar. In practice, most other 1–2B models still ship with a large monolithic embedding table that fails to load on many mobile GPUs and forces a slow CPU fallback. Gemma 4's PLE architecture is currently the most reliable way to run a real 2B-class model on the GPU of a mid-range phone, which changes latency from painful to conversational.
More from gist-ai.net: what on-device AI actually is (and why it works), offline ChatGPT alternatives — the real options in 2026, the AI chatbot that works in airplane mode, how to summarize any YouTube video with AI, summarize articles, PDFs and photos of text — offline, practice a language with an AI in your pocket, and the 8 best podcasts on YouTube — no sign-up, ever.
