u/Defiant_Confection15

Every local model — Llama, Gemma, Qwen, BitNet — generates with full confidence whether the answer is right or completely made up. There’s no internal signal that says “I’m not sure.”
σ-gate adds that signal. It reads the model’s hidden states during inference and computes a unified confidence score from five independent signals — cheapest first, stops when the verdict is clear.
Three outcomes: ACCEPT (trust it), RETHINK (regenerate), ABSTAIN (“I don’t know”).
The image explains the full pipeline.
12 bytes of state. C89. No floats. No malloc. No dependencies. No cloud. Runs locally on a MacBook Air at 5.8W.
Works with any GGUF model. Available as MCP plugin.
github.com/spektre-labs/creation-os

Posted about Creation OS a couple weeks ago. Here’s the follow-up with numbers.

Problem

Most hallucination detectors need multiple forward passes. Semantic entropy needs 5-20 samples. SelfCheckGPT needs multi-generation. Expensive and slow for local inference.

σ-gate

One forward pass. Measures distortion between outputs and hidden states. Returns ACCEPT, RETHINK, or ABSTAIN.

12 bytes state. No floats. No malloc. C89. Deterministic. Tested on MacBook Air M4 8GB at 5.8W.

Results

|Signal |Benchmark |AUROC|Notes |

|---------|------------------|-----|--------------------------|

|LSD probe|TruthfulQA holdout|0.982|trained, n=57 |

|LSD probe|TriviaQA |0.960|cross-domain, n=100 |

|HIDE |TruthfulQA |0.857|training-free, single pass|

|HIDE |Gemma-2-2b |0.778|cross-model, n=10 |

ECE: 0.043. Wrong + confident: 0. Cost routing: ~98% vs always-large-model. ABSTAIN rate: 10.5%. Conformal bound: P(error | ACCEPT) ≤ α (α=0.80, δ=0.10).

Formal verification

Lean 4: 6/6 sorry-free. Frama-C WP: 15/15 tier-1 discharged.

Limitations

GPT-2 scale probe, white-box. Cross-model n=10 (n=30 in progress). Strongest on factual QA — not dominant on HellaSwag/MMLU. Long-form not yet evaluated. docs/limitations.md

Try it

git clone https://github.com/spektre-labs/creation-os

cd creation-os && make cos cos-demo && ./cos demo --batch

from cos.sigma_gate import SigmaGate

gate = SigmaGate("path/to/probe.pkl")

sigma, decision = gate(model, tokenizer, prompt, response)

MCP server: python3 -m cos.mcp_sigma_server

How I build

I use LLMs as tools — Claude, GPT, Gemini, DeepSeek — cross-validated against each other. I like working with them.

github.com/spektre-labs/creation-os