OpenLM AI Proxy
To track AI token usage per user and model, route your large language model (LLM) API requests through OpenLM AI Proxy, a self-hosted gateway that sits between your users and LLM providers. It forwards API requests to the provider and records the token usage of each request, so you can attribute AI spend to users and models and feed that data into OpenLM Platform.
AI Proxy reads usage metadata from requests and responses. It doesn't inspect or store prompt or completion content.
AI Proxy is designed first as an API gateway: you route LLM API traffic through it, and it records usage. User-level monitoring through AI Proxy is experimental and secondary. Start with API interception, described in Configure AI Proxy.
How AI Proxy works
A client sends an LLM API request to AI Proxy instead of directly to the provider. AI Proxy forwards the request to the provider, returns the response to the client, and records the token counts for the call. It then stores the usage record and sends usage data to OpenLM Platform.
The clients that can route through AI Proxy include the Claude Code CLI, your own applications that call an LLM API, and a LiteLLM gateway placed in front of one or more providers.
Architecture
AI Proxy runs as a set of Docker containers, deployed together with Docker Compose:
| Container | Role |
|---|---|
| API | Core proxy. Routes LLM requests, records token usage, and syncs usage data to OpenLM Platform. Exposes the API and OpenTelemetry (OTel) endpoints. |
| UI | Web interface for managing configuration: the OpenLM Platform connection, providers and API keys. |
| MongoDB | Stores usage records and configuration. |
| Redis | Backs background jobs and caching. |
The API container connects to OpenLM Platform through a receiver that you set up in Receivers Configuration, and emits usage events (ai-usage-events) and asset information (ai-asset-info). It can also export OpenTelemetry events to an external collector, such as Grafana or Datadog.
Supported AI platforms
The following table lists the AI platforms and their status:
| Platform | Status | Notes |
|---|---|---|
| Anthropic (Claude Code) | Supported | The most mature integration. Includes a Claude Code CLI proxy and OpenTelemetry-based usage monitoring. |
| OpenAI | Prototype | Managed and passthrough modes. |
| Google Vertex AI (Gemini) | Supported | Managed and passthrough modes. |
| Cursor | Not supported through AI Proxy | Track Cursor usage with Cursor instead. |
Authentication modes
AI Proxy supports two ways to authenticate requests to the LLM provider:
- Managed: AI Proxy holds a shared provider API key and applies it to outgoing requests. Clients authenticate to AI Proxy, not to the provider.
- Passthrough: Each client sends its own provider API key with the request, and AI Proxy forwards it unchanged.
Support for each mode varies by platform. See Configure AI Proxy.
Deployment model
You deploy AI Proxy as a Docker Compose stack inside your own network. This keeps API keys and usage data within your infrastructure.
In this section
Work through the pages in order:
- System requirements: the host, runtime, and network the stack needs.
- Deploy AI Proxy: run the Docker Compose stack in your network, then upgrade or remove it.
- Configure AI Proxy: connect to OpenLM Platform, set up providers, route clients, and confirm that data reaches AI FinOps.
- Troubleshoot AI Proxy: fix common problems.
Related references
- AI FinOps reports: dashboards for AI cost and token consumption.
- Broker Hub: the AI Proxy tab lists your AI Proxy agents.
- AI Proxy changelog: per-version fixes and improvements.
Continue to System requirements to check your host before you deploy.
The documentation team might draft this page using a large language model. A technical writer reviews every page and verifies it against the product before publishing it. A human makes the final decision.