Hey all — wanted to share something we’ve been building in case it’s useful to anyone working with LLMs/agents, and also get feedback from this community.
Disclosure: I work at TrueFoundry, so take this as a “here’s what we made,” not a neutral recommendation.
The problem we kept running into: once you’re calling multiple LLM providers, spinning up MCP servers/tools, and running agents in production, you lose visibility fast — no unified logging, no cost tracking per token/user, no easy way to add guardrails across providers, no failover when a provider goes down.
So we built an AI Gateway that sits in front of all of it:
- One OpenAI-compatible API for any provider (OpenAI, Anthropic, etc.) with automatic fallback/load balancing
- An MCP Gateway layer for governing which tools/agents can call what
- Token-level cost attribution and budget limits
- Cross-provider guardrails (not locked to one vendor’s safety layer)
- Full observability — traces, logs, prompt analytics
- Can run as SaaS, on-prem, or fully air-gapped (SOC 2 / HIPAA compliant, which matters if you’re in healthcare/finance)
Docs/repo: TrueFoundry · GitHub
Curious if others here building agent-based projects have run into the same pain points — how are you currently handling multi-provider routing or cost tracking in your own projects?