Basically, the database is full of all the tons of the most common phrases paired with a unique ID. On average it seems like I can compress my message to half the size. I wasn't really aiming to do this. I was just trying to make a code book and this was a byproduct and I thought It might be interesting to share.
But I got me thinking, what's the highest data compression we can get on text currently?
imagine that, or an algorithm that uses a mismatch of many algorithms, to get the best value, through automatic trial and error.
or something like an ai that picks the optimal settings based on a few tests or something. i think this is an idea that is worth exploring.
Hi all,
I've spent the last few months developing a lossless image compressor called SPX, aiming to balance compression density and encoding speed, that is, maintaining compression rate higher than .webp (m6) but lower than .jxl (e7) while significantly enhancing encoding speed.
I did some testing and the performance seems consistent in most datasets but compression savings aren't that consistent.
I think I've hit my limit as a self-taught amateur developer knowing a little Python. I can't come up with any new idea to improve it anymore so Gemini suggested coming here for professional advice.
It's an Apache 2.0 open source project. Any suggestion on how to improve compression rate without losing too much speed is highly appreciated! Thank you!
GitHub: https://github.com/nonkilife/SPX-Image-Lossless-Compression
Quick Start: pip install spx-codec
==
// The Architecture:
SPX isn't a fundamental breakthrough, but a streamlined 4-part pipeline designed for modern CPU throughput:
// Customization & Extensibility:
// The Performance (Snapshot on AMD Ryzen 5 3500X):
// The Bottleneck:
I've reached a point where manual optimizations (branchless logic, LUT, SIMD-friendly structures) are no longer yielding significant gains.
I've experimented with:
While 90% of approaches are proven failure, there is still unexplored territory:
Please don't tell me about Pigeonhole principle, i don't like pigeons, they dirty the street and Also roof of my house
Been playing around with compression, but instead of treating code as raw bytes, I tried modeling it.
Idea is pretty simple: tokenize the Python source, use an n-gram model to predict the probability of the next token, and then feed those probabilities into an arithmetic coder. The more predictable the token, the fewer bits it costs.
Ran it on the Flask repo (~575 KB of .py files):
Neural + AC → 101 KB (82.4% reduction)
zlib / ZIP → 151 KB (73.7%)
lzma / 7z → 152 KB (73.5%)
zstd → 147 KB (74.4%)
So yeah, about ~33% better compression ratio than zlib.
Nothing magical going on Python code just has a lot of structure at the token level, and n-grams pick up enough of it to make a difference. Arithmetic coding just turns that into actual bits.
The setup is split pretty cleanly: tokenizer + model in Python, and the arithmetic coder is written in Zig (compiled to a shared library) and called via ctypes. Python handles probability generation, Zig handles the actual encoding and bitstream.
The obvious downside: it’s slow. Like really slow. ~75 seconds vs ~0.05s for zlib on the same data (~1600× slower). Most of that is just calling the model once per token with no caching.
Still, kind of interesting to see that even a basic n-gram model can beat general-purpose compressors just by not treating code like noise.
Feels like there’s something here if the prediction side gets better (or faster). Curious if anyone else has tried something similar.
I built a small lossless preprocessing library called STRATA.
It exposes structural transforms such as:
Repo: https://github.com/rjamesy/strata
The unusual finding is that STRATA works better as a preprocessor for general-purpose codecs than as a standalone codec.
In particular, STRATA-preprocess + zstd-1 can measurably beat raw zstd-22 on both speed and compression ratio for shape-aware data:
The mechanism appears to be simple: zstd at -22 performs expensive long-range string matching, but on smooth or structured raw data there may be few exact repetitions to match. STRATA exposes the redundancy directly, so even zstd-1 can exploit it. Total work decreases.
The results are reproducible: bench/preprocess_demo.py writes a CSV covering all tested combinations.
Caveats:
Six months of heavy API usage and my bills felt higher than they should be. Finally sat down and traced exactly where the tokens were going.
Turned out most of it was repetition. Every API call resends the full context window, the whole conversation history, the system prompt, all of it. The context resets each call. You're paying for the same information over and over, every single request.
Built ContextPilot to fix it. It sits between your code and the API and compresses context before each call.
Saving around 60% on API costs at my usage level. MIT licensed, no account needed, works with OpenAI and Anthropic.
Still early, v0.2.2 on PyPI. Would genuinely appreciate feedback from anyone who gives it a try, especially on edge cases or integrations I haven't thought about.
