r/LocalLLM

URL: https://github.com/quantumaikr/quant.cpp

Title (≤80 chars)

Show HN: quant.cpp – 7x longer LLM context via KV cache compression, pure C

Post

I built a minimal LLM inference engine in pure C (67K LOC, zero dependencies) with one goal: extend context length without adding hardware.

The key insight: LLM inference memory is dominated by the KV cache, not model weights. Compressing the KV cache to 4-bit keys + Q4 values gives 6.9x memory reduction with negligible quality loss.

Real numbers on a 16GB Mac (M1 Pro):

Model FP16 KV (llama.cpp) Compressed KV (quant.cpp) Gain

Llama 3.2 3B ~50K tokens ~350K tokens 6.9x

Gemma 4 26B-A4B (MoE) ~4K tokens ~30K tokens 6.9x

How it works:

Keys: uniform 4-bit min-max quantization per 128-element block

Values: Q4 nibble quantization with per-block scales

Delta mode: store key[t] - key[t-1] instead of absolute keys (like video P-frames), enabling 3-bit at +1.3% PPL

QK-norm aware: models like Gemma 4 automatically use FP32 keys + Q4 values (sparse key distributions break low-bit quantization)

Quality (WikiText-2 PPL, SmolLM2 1.7B):

FP32 baseline: 14.63

4-bit K + Q4 V: 14.57 (+0.0%)

Delta 3-bit K + Q4 V: 14.82 (+1.3%)

vs llama.cpp Q4 KV: llama.cpp Q4_0 KV gives PPL +10.6%. quant.cpp gives +0.0%. Same bit budget, 10x less degradation.

Code philosophy: 67K lines of C11. No frameworks, no CUDA required. The full forward pass fits in one file. Ships as a single-header quant.h (15K LOC) you can drop into any C project.

Supported models: Llama 3.2, Qwen 3.5, Gemma 3/4, MoE (128 experts).

./quant model.gguf -p "hello" -k uniform_4b -v q4 # that's it

Feedback welcome. Particularly interested in: (1) what context length you'd need for your use case, (2) which models to prioritize next.

Talking Points for Comments

"Why not just use llama.cpp?" — llama.cpp is fast. quant.cpp goes further. Same model, same hardware: llama.cpp runs out of memory at 8K context, quant.cpp keeps going to 30K. Different tools for different problems.

"How does delta compression work?" — Adjacent key vectors differ by ~30% of their range. Instead of storing absolute 3-bit keys (PPL +62%), we store 3-bit deltas (PPL +1.3%). Every 64 tokens, an FP32 I-frame prevents drift. Same idea as video compression.

"What about GPU?" — Metal shaders included and working. But the bigger win is memory: GPU VRAM is even more constrained than system RAM, making KV compression even more valuable there.

"Single header?" — Yes, quant.h is 15K lines. #define QUANT_IMPLEMENTATION in one .c file, compile with cc app.c -lm -lpthread. Full GGUF loading, tokenization, and inference. No cmake, no build system

I've been using both side by side over this evening working on a project. Basically I'd paste a chunk of creative text into chat and tell it to dismantle it thesis-by-thesis, then I'd see if the criticism is actually sound, and submit the next iteration of the file which incorporates my solutions to bypassing the criticism. Then move on to the next segment, next file, repeat ad infimum.

What I found is that Gemma 4 31B keeps track of the important point very cleanly, maintains unbiased approach over more subsequent turns: GLM basically turns into a yes-man immediately "Woah! Such a genius solution! You really did it! This is so much better omfg, production ready! Poosh-poosh!", Gemma can take at least 3-4 rounds of back and forth and keep a level of constructivism and tell you outright if you just sidestepped the problem instead of actually presenting a valid counterargument. Not as bluntly and unapologetically as it could've, but compared to GLM, ooof, I'll take it man... Along the way it also proposed some suggestions that seemed really efficient, if not out of the box (example, say you got 4 "actors" that need to dynamically interact in a predictable and logical way, instead of creating a 4x4 boolean yes-no-gate matrix where a system can check who-"yes"-who and who-"no"-who, you just condense it into 6 vectors that come with instruction for which type of interaction should play out if the linked pair is called. it's actually a really simple and even obvious optimization, but GLM never even considered this for some reason until I just told it. Okay, don't take this is as proof of some moronic point, it's just my specific example that I experienced.

Gemma sometimes did not even use thinking. It just gave a straight response, and it was still statistically more useful than the average GLM response.
GLM would always think for a thousand or two tokens. Even if the actual response would be like 300, all to say "all good bossmang!"

It also seemed like Gemma was more confident at retrieving/recreating stuff from way earlier in conversation, rewriting whole pages of text exactly one-to-one on demand in chat, or incorporating a bit from one point in chat to a passage from a different point, without a detailed explanation of what exact snippets I mean. I caught GLM just hallucinate certain parts instead. Well, the token meter probably never went above like 30k, so I dunno if that's really impressive by today's standard or not though.

On average I would say that GLM wasted like 60% of my requests by returning useless or worthless output. With Gemma 4 it felt like only 30% of the time it went nowhere. But the amount of "amazing" responses, which is a completely made up metric by me, was roughly the same at like maybe 10%. Anyway, what I'm getting at is, Gemma 4 is far from being a perfect model, that's still a fantasy, but for being literally a 30B bracket model, to feel so much more apparently useful than a GLM flagman, surprised the hell out of me.

A big milestone for local inference.

Hey all, I know what I am about to say may be laughable and unideal, but is there is a way to make this work? I like local but can't afford a big budget local AI setup. Can I just plug in an Nvidia R9 in an external GPU case (with psu) and plug it into an old computer and make a slow running ollama server? It doesn't have much RAM, like 8 or 16 GB, and it is also slow DDR, but can I make it use SWAP space or something for big code ingestions? I don't mind waiting hours for results. I just don't want to deal with this model quotas when coding. Tried searching for this use case in the sub but can't seem to find a clear answer on this.

I’m less interested in what already works, and more in what still feels missing.

I'm working on the mobile app with local AI, that provides not only chatbot features, but real use cases and I really need your thoughts!

A lot of mobile local AI right now feels like “look, it runs” or “here’s an offline chatbot” but I’m curious where people still feel the gap is.

What do you wish local AI on phones could do really well, but still can’t?

Could be anything:

• something you’ve tried to do and current apps are too clunky for
• something that would make local AI genuinely better than cloud for you
• some super specific niche use case that no one has nailed yet

Basically, what’s the missing piece?

What’s the thing where, if someone built it properly, you’d actually use it all the time?

I built a pretty beefy machine to do 2 things on:

• Training an LLM
• Playing games on Steam

My hardware is:

• Processor - AMD Ryzen 9 9900X 4.4 GHz 12-Core
• Motherboard - ASRock B850I Lightning Mini ITX
• Ram - Corsair Vengeance 64 GB (2 x 32 GB) DDR5-5200 CL40
• Storage - Western Digital WD_Black SN850X 2TB
• GPU - Gigabyte WINDFORCE SFF GeForce RTX 5070 Ti 16GB

I'm coming from Windows so would like the transition to be as seamless as possible. From what I read, the best pick for my use cases would be Kubuntu 24.04 LTS

Anyone disagree? Thanks in advance ^^

I ran into a limit on Claude and thought it was dumb. I have an M1 16gb mini and am looking to run something locally. Would my machine be too slow? Would I run into any potential issues? I am not a crazy user by any means, exploring mostly and have some use cases but noting needing to run 24/7 or anything. Though it would be nice to give it a research task to run overnight.

Hi everyone, I am from Australia : ) I just released a new research prototype

It’s a lossless BF16 compression format that stores weights in 12 bits by replacing the 8-bit exponent with a 4-bit group code.
For 99.97% of weights, decoding is just one integer ADD.

Byte-aligned split storage: true 12-bit per weight, no 16-bit padding waste, and zero HBM read amplification.

Yes 12 bit not 11 bit !! The main idea was not just “compress weights more”, but to make the format GPU-friendly enough to use directly during inference:

sign + mantissa: exactly 1 byte per element
group: two nibbles packed into exactly 1 byte too

https://preview.redd.it/qbx94xeeo2tg1.png?width=1536&format=png&auto=webp&s=831da49f6b1729bd0a0e2d1f075786274e5a7398

• 1.33x smaller than BF16
• Fixed-rate 12-bit per weight, no entropy coding
• Zero precision loss bit-perfect reconstruction
• Fused decode + matmul, so there is effectively no separate decompression stage
• Byte-aligned storage, no LUT, no bitstream parsing
• Works on both NVIDIA and AMD

Some results so far:

Single-user (B=1), RTX 5070 Ti

• Llama 2 7B: 64.7 tok/s (1.47x vs vLLM)
• Mistral 7B: 60.0 tok/s (1.10x vs vLLM)
• Llama 3.1 8B: 57.0 tok/s (vLLM OOM on 16 GB)

Multi-user (B=256), total tok/s

• Llama 2 7B: 2931 vs 1086 in vLLM (2.70x)
• Mistral 7B: 2554 vs 872 in vLLM (2.93x)

It also seems surprisingly stable across model types:

• Llama 3.1 405B: 0.034% escape rate
• Mixtral 8x7B: 0.050%
• SDXL UNet: 0.233%
• CogVideoX 2B: 0.128%

So far this is tested on BF16 safetensors only.

Repo: https://github.com/cenconq25/Turbo-Lossless

Also worth noting: the V3 fused decode+GEMM kernel uses tensor-core patterns inspired by ZipServ / ZipGEMM (Fan et al., ASPLOS 2026).

Happy to hear criticism, edge cases, or reasons this idea won’t scale.

Thanks for your time : )

https://preview.redd.it/lkn1nik8l1tg1.png?width=2762&format=png&auto=webp&s=5912f82fb678dde47b5df655cedb055426873c48

so everyone know by now that Google just dropped Gemma 4 and I had to double check the numbers a few times because it's insane..

31B params and runs on a single GPU. And it's putting up across benchmrks:

• Arena Elo ~1452 (GPT-5.1 ~1475, basically same tier)
• MMLU-Pro 85.2% (slightly higher than GPT-5.1)
• GPQA Diamond 84.3% (a bit behind but close enough)

I mean - this is not some massive cluster model, you can run this locally, how??

not long ago, open models were clearly a step behind. now you're looking at something you can download and run yourself sitting right next to a $200/month flagship on the benchmarks that matter for general reasoning..

the only place there's still a noticeable gap is coding heavy stuff like SWE-bench, but everything else feels… uncomfortably close

if that's the new trend, I'm curious how long big labs can hold onto their current valuation?

I'm working on a project where I'll be fine-tuning LLMs with GRPO on a 170K-sample dataset for explainable LJP (legal judgment prediction, where the model predicts case outcomes and generates step-by-step reasoning citing the facts). I'm considering models like GPT OSS 20B or Qwen 3.5 27B, with a slight preference for Qwen 3.5 27B because of its strong reasoning capabilities.

I recently obtained a 96GB VRAM workstation (RTX PRO 6000) to handle the RL rollouts, which should give some solid headroom for larger models.

What are your recommendations for the best open-source models for GRPO fine-tuning in 2026? Any advice on structuring explainable LJP rewards would also be appreciated.

Thanks!

What are you using them for?

I have read many of the posts in this subreddit on this subject but I have a personal perspective that leads me to ask this question again.

I am a sysadmin professionally with only limited scripting experience in that domain. However, I've recently realized what Claude Code allows me to do in terms of generating much more advanced code as an amateur. My assumption is that we are in a loss leader phase and this service will not be available at $20/mo forever. So I am curious if there is any point in exploring whether smallish local models can meet my very introductory needs in this area or if that would simply be disappointing and a waste of money on hardware.

Specifically, my expertise level is limited to things like creating scrapers and similar tools to collect and record information from various sources on various events like sports, arts, music, food, etc and then using an llm to infer whether to notify me based on a preference system built for this purpose. Who knows what I might want to build in the future that is where I'm starting which I'm assuming is a basic difficulty level.

Using local models able to run on 64G of VRAM/Unified, would I be able to generate this code somewhat similarly to how well I can using Claude Code now or is this completely unrealistic?