Login

u/Ashcrypto915

We open-sourced the AI gateway behind our SaaS. Looking for feedback from AI SaaS founders
▲ 5 r/SaasDevelopers+2 crossposts

We open-sourced the AI gateway behind our SaaS. Looking for feedback from AI SaaS founders

We just open-sourced Alephant AI Gateway, the gateway layer behind our AI FinOps product.

The problem we’re working on: AI SaaS teams start with one provider, then add more models, retries, agents, background jobs, test traffic, and customer-specific usage. Eventually the bill arrives and nobody knows which feature, user, or agent caused the spend.

Alephant Gateway is open-source and OpenAI-compatible:

  • 50+ providers
  • 320+ models
  • BYO keys
  • Hosted or self-hosted
  • Routing, fallback, caching, policy, logs
  • Usage metadata for cost visibility

Repo: https://github.com/AlephantAI/AIephant-AI-Gateway

Question for SaaS founders building AI features: Are you tracking AI cost per user/customer today? If yes, how?

u/Ashcrypto915 — 6 hours ago
▲ 2 r/LLMDevs

How are you handling routing, fallback, and cost attribution across multiple LLM providers?

I’m working on LLM gateway infrastructure and wanted to compare notes with people running multi-provider AI apps in production.

The pattern I’m seeing is that teams usually start simple:

One OpenAI SDK integration Then Anthropic or Gemini gets added Then fallback gets added Then retries and rate-limit handling Then agents start making chained calls Then nobody can answer which user, feature, or agent caused the spend spike

The technical problems get messy fast:

Normalizing request/response formats across providers Handling streaming differences Mapping provider errors consistently Preserving usage metadata Tracking cost per user, session, agent, or feature Adding fallback without hiding failures Preventing retry storms Deciding when to cache Keeping provider keys isolated from app-facing keys

For people here building LLM apps, how are you solving this today?

Are you using:

Direct provider SDKs LiteLLM OpenRouter Helicone Portkey A custom proxy/gateway Something else?

I’m especially curious about where people draw the line between “simple wrapper” and “we need a real gateway now.”

I’m working on an open-source Rust gateway in this space, but I’m mainly looking for design feedback here rather than promoting it. If anyone wants context, I can share the repo in comments.

reddit.com
u/Ashcrypto915 — 7 hours ago
▲ 2 r/google_antigravity+2 crossposts

Open source AI gateway for agent cost control

Hey everyone, I'm building Alephant an open source AI gateway written in Rust, focused on AI agent cost control and multi-LLM observability.

Most gateways just route requests. Alephant tracks what they cost, token-level spend, per-agent budget controls, and guardrails to prevent runaway AI bills across OpenAI, Anthropic, and 50+ providers.

It's OpenAI-compatible, self-hostable, BYO keys, and works today with Cursor, Codex, and opencode.

Just launched v0.2.0-beta as open source.

👉 https://github.com/AlephantAI/AIephant-AI-Gateway

Would love your feedback especially if you've hit painful AI billing surprises in production.

u/Ashcrypto915 — 5 hours ago