u/cryptocreeping

​

I've been building an extended OTR implementation for IRC that goes beyond the OTRv4 spec draft, targeting high-threat privacy scenarios like Tor hidden services and I2P eepsites. Just pushed v10.5.5 with a significant Rust core security hardening pass. Wanted to share it here because the crypto design decisions are the interesting part and I'd genuinely like feedback on them.

GitHub: https://github.com/muc111/OTRv4Plus

What it actually does cryptographically

The core ratchet is a Rust implementation (zeroize on drop via the zeroize crate) with a PyO3 binding layer. The key exchange uses X448 for the DH ratchet and ML-KEM-1024 (NIST FIPS 203) for the brace key, so every ratchet epoch is hybrid classical/post-quantum. The brace key rotates via KDF SHAKE-256 on each epoch, which means a quantum attacker harvesting ciphertexts today can't retroactively decrypt past epochs even if they break X448 later.

Identity signatures use ML-DSA-87 (NIST FIPS 204) implemented through a C extension that calls into the OpenSSL EVP layer for the actual CRYSTALS operations. I didn't roll my own lattice arithmetic, just the binding code and the OTR DAKE integration. Public key bytes are 2592, signatures are 4627 bytes per FIPS 204.

The SMP implementation follows the OTRv4 spec section 4.6. Standard Boudot/Jacobi ZK proof over a 3072-bit safe-prime group, but I added session binding so the hashed secret is KDF(SHAKE-256, secret || session_id || fingerprint_alice || fingerprint_bob). This prevents cross-session replay where an attacker captures SMP transcripts from one session and replays them in a different context.

The DAKE uses the ring signature construction from the OTRv4 draft for deniability. You get auth without a PKI trail.

v10.5.5 security hardening (latest commit)

This was an audit pass on the Rust double ratchet core. Fixed several things that were bugging me:

· RNG: Switched from rand::random() to OsRng with fill_bytes(). The old approach wasn't guaranteed cryptographically secure across all target platforms and I don't know why I used it in the first place.

· KDF forward secrecy: Message encryption keys now derive from the next chain key rather than the current one. Previously a compromise of the current chain key could decrypt the current message. Now it can't.

· Hybrid key composition: The root ratchet KDF now domain separates DH and PQ contributions using distinct usage IDs (ROOT_DH and ROOT_PQ) before combining. Previously they were concatenated raw which felt sloppy and could allow cross-protocol confusion.

· DoS bounds: Added explicit bounds checks on ratchet skip loops. An attacker could previously trigger a near unbounded chain key derivation loop. Now capped at MAX_SKIP = 1000.

· Replay cache: Switched from linear VecDeque scan to HashSet plus VecDeque for O(1) replay lookup. Previously an attacker could degrade performance by forcing linear scans over a full cache.

Test suite now passes 313 tests including a new adversarial security suite that specifically targets each of these issues to prevent regression. Caught several of these during testing which was satisfying.

Seven CVEs patched in earlier releases

These were mostly implementation bugs rather than spec issues:

· Timing side channel in the SMP proof verification (non constant time modular exponentiation, replaced with Montgomery ladder via OpenSSL BN_mod_exp_mont_consttime)

· MAC key reveal list wasn't being zeroed after transmission (forward secrecy leak)

· Fragment reassembly buffer had no per sender ceiling, trivial DoS

· Session expiry check happened after key material was loaded rather than before

· SOCKS5 proxy hostname resolution was happening locally not at the proxy (.onion addresses were being passed to the OS resolver before Tor saw them)

· Two others in the key derivation path.

What I'd actually like feedback on

1. The brace key rotation. I'm using kdf_1(BRACE_KEY_ROTATE, old_brace_key || KEM_ss, 32) where kdf_1 is SHAKE-256 with a usage ID prefix. Is there a cleaner way to handle the hybrid KEM forward secrecy contribution without coupling the brace key state to the ratchet epoch counter?

2. The SMP session binding. I'm not aware of this being in the OTRv4 spec draft, I added it myself. Has anyone seen a formal analysis of whether stock OTRv4 SMP is vulnerable to cross session attacks, or am I defending against a threat that doesn't exist in practice?

3. ML-DSA-87 key sizes are large for IRC (2592 byte public keys in DAKE messages). Anyone dealt with this in practice in constrained message size protocols? Feels heavy but I don't see a way around it.

4. The hybrid KDF domain separation approach in v10.5.5 using separate usage IDs for DH and PQ contributions then concatenating. Is this the right pattern or should I be using a proper KDF combiner like HKDF with info strings? I went with what felt clean but open to being wrong here.

Disclosure (subreddit rule)

Claude AI assistance throughout this project I just could not have achieved this working prototype took 13+ Months so far still working on improvements would love fresh eyes on it fine bugs etc.

Thanks!