u/_clickfix_

I switched to using local models with Hermes and I’m never going back.

I first tried cloud hosted models using the Anthropic API with Haiku. It was honestly pretty dumb and insanely expensive. I burned through $100 in a single day just getting set up and running tests.

My goal is to use an AI agent to actually make money, so I knew that burn rate was never sustainable.

I finally bit the bullet and invested $4,500 into a 128GB unified memory machine running Hermes with gpt-oss locally. The reasoning is great, the bot feels smart, and responses are fast.

I also like that my data never leaves my own network, where I know it’s secure.

It’s a big upfront investment. But compared to spending $100 a day on API credits, the hardware pays for itself in about 45 days.

After that, my only real cost is electricity, which is negligible.

Has anyone else switched to a setup like this? Curious what hardware and models people are running locally now.

A recent internal data leak from “The Gentlemen” ransomware-as-a-service (RaaS) group has provided the cybersecurity community with a rare, unfiltered look into their daily operations. Exposed on underground forums, the internal communications shed light on exactly how ransomware affiliates organize, breach, and extort global organizations.

But among the many technical details revealed in Checkpoint Research’s comprehensive analysis (“Thus Spoke… The Gentlemen”), one operational pattern stands out prominently: their heavy reliance on infostealer credential logs for initial access.

Credit: originally posted by u/malwarebeasts

The Mythos/Glasswing/Mercor incident chain from March–April is worth studying as an AI security case because every link in the chain involved AI infrastructure, and none of the failures were caught by the most sophisticated voluntary AI safety framework in the industry.

Quick recap of the kill chain for those who didn't follow it closely:

TeamPCP compromised Trivy's GitHub Action via a pull_request_target workflow vulnerability, rewrote it with credential harvesting payloads. LiteLLM's CI/CD ran the compromised action with unpinned references exfiltrating its PyPI publish token. Malicious LiteLLM 1.82.7 and 1.82.8 hit PyPI on March 27, live for roughly 40 minutes. LiteLLM is in about 36% of cloud environments with give or take numbers at around 95-97M monthly downloads.

Mercor $10B AI training data startup working with OpenAI, Anthropic, Meta, Google auto-pulled the malicious package. Result: being 4TB exfiltrated via Tailscale. 939 GB source code, 211 GB user database, 3 TB of video interviews, KYC documents, and biometric data for 40,000 contractors.

Then the AI specific part of it, members of a Discord group combined an active third party vendor evaluation credential with internal model hosting naming conventions they reconstructed from the Mercor data to guess Mythos's deployment URL and walk into unreleased Anthropic models.

As far as I'm concerned this is an AI security issue, the single gateway pattern. LiteLLM functioned as both the LLM routing layer and the concentration risk. One compromised package gave lateral access across the entire AI inference stack. This is a pattern that's everywhere right now with abstraction layers that become single chokepoints because nobody's treating model routing infrastructure as critical attack surface.

No AI specific telemetry caught the exfiltration. 4TB out during attacker dwell. Standard SIEM wasn't ingesting prompt/completion/tool use semantic conventions. No canary tokens in the AI pipeline. No embedding drift detection. The assumed breach detection layer had zero AI specific instrumentation that I've found.

Agent and vendor credential scoping. The third party evaluation credential that provided access to Mythos had no per session authorization gates, no time bounding, no immutable audit trail. CISA's Zero Trust Maturity Model v2.0 explicitly excludes AI/ML methodologies from scope so even orgs running mature ZT programs have a structural gap around AI workload identity.

The AIBOM gap. No signed provenance manifest on the LiteLLM dependency. No SLSA attestation. The unpinned reference in CI/CD would have been a hard stop if provenance verification had been a deployment gate.

The bigger point:

Anthropic's RSP is the most sophisticated voluntary AI safety framework that exists. It didn't prevent this because the enforcement lived in policy documents and access agreements, not in the deployment infrastructure itself. The perimeter model for AI security vetted partners, carefully drafted agreements, threshold based capability evaluations just doesn't survive contact with multi tier vendor reality.

The security primitives that would have caught this are architectural: signed provenance on every artifact in the AI pipeline, per session zero trust authorization extended to AI workloads and agents, AI specific assumed breach telemetry (semantic conventions for prompt/completion/tool use flows, canary tokens, embedding anomaly detection), concentration risk tracking on model routing infrastructure, and importantly hash chained evidence registries on every access decision.

None of that is out of our technological environment. It's applying security engineering principles we already know to AI specific infrastructure.

What's everyone seeing in their own environments? Is anyone actually instrumenting their AI inference pipelines with security telemetry, or is it still bolted on monitoring at the network layer?