Skip to main content

Deploy AI Proxy

Deploy AI Proxy as a Docker Compose stack on a Linux host inside your network. After the stack is running, you connect it to OpenLM Platform and set up providers in Configure AI Proxy.

Before you begin​

Confirm these prerequisites:

  • A Linux host that meets the system requirements.
  • Docker Engine and the Docker Compose v2 plugin, installed. To check, run docker compose version.
  • Access to the AI Proxy container images, provided by OpenLM. If the images are in a private registry, sign in first with docker login.
  • Network access, as described in the system requirements.
  • Your own values for the MongoDB password, the Redis password and the encryption key. Generate them before you start. See Set your own passwords and key.

Deploy the stack​

To deploy the stack, create two files, review their values, and start the containers. The AI Proxy UI runs under the /aiproxy/ path and sends its API calls to /api/ on the same address, so it needs a reverse proxy in front of it. In a working directory on the host, for example ~/aiproxy/, create a file named aiproxy-ui-proxy.conf:

server {
  listen 8092;

  # The AI Proxy UI, with the /aiproxy prefix stripped.
  location /aiproxy/ {
    proxy_pass http://aiproxyui:8092/;
  }

  # API calls from the browser, forwarded to the AI Proxy API.
  location /api/ {
    proxy_pass http://aiproxyapi:8088/api/;
  }
}

In the same directory, create a file named docker-compose.yml with the following content:

services:
  mongodb:
    image: mongo:7
    ports:
      - "27017:27017"
    volumes:
      - mongodb_data:/data/db
    environment:
      - MONGO_INITDB_ROOT_USERNAME=admin
      - MONGO_INITDB_ROOT_PASSWORD=admin
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    command: redis-server --save 60 1 --loglevel warning --requirepass openlm
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "redis-cli", "-a", "openlm", "ping"]
      interval: 10s
      timeout: 5s
      retries: 5

  aiproxyapi:
    image: public.ecr.aws/r3q3q2f4/olm-aiproxyapi:1.260630.1851
    ports:
      - "8088:8088"
    depends_on:
      mongodb:
        condition: service_healthy
      redis:
        condition: service_healthy
    environment:
      - MONGODB_CONNECTION=mongodb://admin:admin@mongodb:27017
      - REDIS__CONNECTIONSTRING=redis,password=openlm
      - CLOUD_MODE=false
      - ENCRYPTION_KEY=default-dev-key
      # Required. The API does not start without it.
      - hangfire__dashboard__enabled=false
    restart: unless-stopped

  aiproxyui:
    image: public.ecr.aws/r3q3q2f4/olm-aiproxyui:1.260625.0620
    depends_on:
      - aiproxyapi
    environment:
      - LB_FQDN=localhost
    restart: unless-stopped

  # Serves the UI under /aiproxy/ and forwards its API calls to the API.
  aiproxy-proxy:
    image: nginx:stable
    ports:
      - "8092:8092"
    depends_on:
      - aiproxyui
      - aiproxyapi
    volumes:
      - ./aiproxy-ui-proxy.conf:/etc/nginx/conf.d/default.conf:ro
    restart: unless-stopped

volumes:
  mongodb_data:
  redis_data:

Set your own passwords and key​

Before you start the stack, replace the sample values in docker-compose.yml with your own:

  1. Choose a MongoDB password. Enter it in the MONGO_INITDB_ROOT_PASSWORD setting of the mongodb service and in the MONGODB_CONNECTION setting of the aiproxyapi service. The two must match.

  2. Choose a Redis password. Enter it in the --requirepass option of the redis service and in the REDIS__CONNECTIONSTRING setting of the aiproxyapi service. The two must match.

  3. Generate an encryption key and enter it in the ENCRYPTION_KEY setting of the aiproxyapi service:

    openssl rand -base64 32

    Paste the string exactly as it prints.

  4. Don't publish the database ports. The API reaches MongoDB and Redis over the Compose network, so remove the ports entries of the mongodb service (27017) and the redis service (6379).

Don't reuse a password or key from a sample or another environment. MongoDB creates its account only when its data volume is empty, so changing the MongoDB password later doesn't change the existing account.

Keep the encryption key

AI Proxy uses ENCRYPTION_KEY to protect the credentials and API keys that it stores. Store the key somewhere safe, and keep the same value every time you recreate the containers. If you change it, AI Proxy can't read the stored secrets and you enter them again.

Also check the host ports for the API (8088) and the UI proxy (8092). Change them if another service on the host uses them. You configure the OpenLM Platform connection and providers later, in the UI.

Start the stack​

Start the stack in the background. Compose pulls the images on the first run:

docker compose up -d

Check that all containers report a running state:

docker compose ps

Verify the deployment​

Check each item in turn. The last two items prove that the stack reports to OpenLM Platform, so you can do them after you connect AI Proxy to OpenLM Platform.

To confirm that usage reaches AI FinOps, follow Confirm that data reaches AI FinOps.

Upgrade​

Before you upgrade, check which build you run, find the newer builds, and compare your Compose file with the one in this guide.

  1. List the builds you run:

    docker compose images

  2. Find the newest builds. Tags are build numbers in the form 1.<yymmdd>.<hhmm>, and the registry has no latest tag. The registry issues its access token anonymously, so you need no account. On Linux, with curl and jq:

    repo=olm-aiproxyapi   # or olm-aiproxyui
    token=$(curl -s "https://public.ecr.aws/token/?service=public.ecr.aws&scope=repository:r3q3q2f4/$repo:pull" | jq -r .token)
    curl -s -H "Authorization: Bearer $token" "https://public.ecr.aws/v2/r3q3q2f4/$repo/tags/list" | jq -r '(.tags // [.errors[0].message])[]' | sort

    Run the commands once for each image. The tags sort in build order, so the last line is the newest build. If the output is Rate exceeded, wait 5 seconds and run the commands again.

  3. Compare your docker-compose.yml with the one in this guide. A newer build can need settings that your file doesn't have yet. Add them to your file and keep your own passwords and ENCRYPTION_KEY.

  4. Back up the MongoDB data, for example with mongodump or by copying the mongodb_data volume.

  5. Update the image tags in docker-compose.yml, then pull the images and recreate the containers. Compose recreates only what changed:

    docker compose pull
    docker compose up -d

Your usage data and configuration persist in the MongoDB volume across upgrades. AI Proxy upgrades its stored data when it starts, so keep the same ENCRYPTION_KEY. The AI Proxy changelog lists what changed in OpenLM Platform for AI Proxy. It doesn't list image tags, so use the registry to find them.

Uninstall​

To remove AI Proxy, remove the stack on the host, delete the agent in OpenLM Platform, and point your clients back at their providers.

  1. Route your clients away from AI Proxy first, so that they don't call an endpoint that no longer exists. For Claude Code, follow the revert step in Route the Claude Code CLI through AI Proxy. Reset any other client that you pointed at AI Proxy.

  2. In the directory that holds docker-compose.yml, stop and remove the containers:

    docker compose down

    Your data stays in the Docker volumes, so you can start the stack again later.

  3. To remove the data too, run the command in the following warning.

    This deletes stored data

    The command removes the MongoDB and Redis volumes. It erases the usage records that are not yet sent, your provider settings, the OpenLM Platform connection (including its Client ID and Secret), and the stored API keys. A new stack on the same host starts with a new identity and appears as a new agent in Broker Hub.

    docker compose down -v

  4. In OpenLM Platform, open Broker Hub, select the AI Proxy tab, select the check box of the agent, and select Delete.

  5. If you created an Identity client for AI Proxy, delete it in OpenLM Platform.

  6. Optionally delete the working directory and the images (docker image rm).

Next step

Continue to Configure AI Proxy to connect the stack to OpenLM Platform. If something fails, see Troubleshoot AI Proxy.

Last updated on by Maria Gilca

The documentation team might draft this page using a large language model. A technical writer reviews every page and verifies it against the product before publishing it. A human makes the final decision.