LLMOps: A pluggable LLMOps toolkit for TypeScript applications.

Deploy LLMOps as a standalone service using Docker. This is perfect for running LLMOps as a dedicated microservice or deploying to container platforms like Railway, Fly.io, or AWS ECS.

Quick Deploy

Deploy instantly to Railway:

Docker Deployment

Prerequisites

Docker installed on your system
PostgreSQL database (can be external or via Docker)

Set up PostgreSQL

If you don't have a PostgreSQL database, create one using Docker:

docker run -d \
  --name llmops-postgres \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=password \
  -e POSTGRES_DB=pg_llmops \
  -p 5432:5432 \
  postgres:16

Your database connection string will be:

POSTGRES_URL=postgresql://postgres:password@localhost:5432/pg_llmops

For production, use a managed PostgreSQL service like:

AWS RDS
Supabase
Neon
PlanetScale

Pull the Docker image

Pull the latest LLMOps standalone image:

docker pull ghcr.io/llmops-build/llmops-standalone:latest

Or build from source:

git clone https://github.com/llmops-build/llmops.git
cd llmops
docker build -f standalone/Dockerfile -t llmops-standalone .

Configure environment variables

Create a .env file with your configuration:

.env

# Required: PostgreSQL connection string
POSTGRES_URL=postgresql://postgres:password@localhost:5432/pg_llmops

# Optional: Server port (default: 3000)
PORT=3000

Run the container

Start the LLMOps standalone container:

docker run -d \
  --name llmops-standalone \
  --env-file .env \
  -p 3000:3000 \
  ghcr.io/llmops-build/llmops-standalone:latest

If using the locally built image:

docker run -d \
  --name llmops-standalone \
  --env-file .env \
  -p 3000:3000 \
  llmops-standalone

Verify deployment

Check that the container is running:

docker ps

Test the health endpoint:

curl http://localhost:3000/

You should see a response like:

{
  "status": "ok",
  "message": "LLMOps Standalone Server",
  "timestamp": "2024-01-12T10:30:00.000Z"
}

Access the LLMOps dashboard at:

http://localhost:3000/

Configure providers

Open the dashboard at http://localhost:3000
Go to Settings → Workspace Providers
Add your LLM provider credentials (e.g., OpenRouter, OpenAI, Anthropic)
Create a config to start routing requests

Docker Compose

For easier management, use Docker Compose:

docker-compose.yml

version: '3.8'

services:
  postgres:
    image: postgres:16
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: pg_llmops
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  llmops:
    image: ghcr.io/llmops-build/llmops-standalone:latest
    environment:
      POSTGRES_URL: postgresql://postgres:password@postgres:5432/pg_llmops
      PORT: 3000
    ports:
      - "3000:3000"
    depends_on:
      postgres:
        condition: service_healthy

volumes:
  postgres_data:

Start the services:

docker-compose up -d

Making API Requests

Once deployed, you can make requests using any OpenAI SDK:

import OpenAI from 'openai';

const openai = new OpenAI({
  baseURL: 'http://localhost:3000/api/genai/v1',
  apiKey: '<ENV SECRET>',
  defaultHeaders: {
    'x-llmops-config': 'your-config-id',
  },
});

const response = await openai.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'Hello!' }],
});

console.log(response.choices[0].message.content);

Environment Variables

Variable	Required	Default	Description
`POSTGRES_URL`	Yes	-	PostgreSQL connection string
`PORT`	No	`3000`	Server port

Production Deployment

Railway

Click the "Deploy on Railway" button above
Configure the PostgreSQL database
Set the POSTGRES_URL environment variable
Deploy!

Railway will automatically:

Provision a PostgreSQL database
Build and deploy the Docker image
Set up HTTPS with a custom domain
Handle scaling and monitoring

Fly.io

# Install flyctl
curl -L https://fly.io/install.sh | sh

# Create app
fly launch

# Set secrets
fly secrets set POSTGRES_URL=your-connection-string

# Deploy
fly deploy

AWS ECS

Push image to ECR:

docker tag llmops-standalone:latest <account-id>.dkr.ecr.<region>.amazonaws.com/llmops-standalone:latest
docker push <account-id>.dkr.ecr.<region>.amazonaws.com/llmops-standalone:latest

Create ECS task definition with the image
Configure environment variables in task definition
Create ECS service with Application Load Balancer
Set up RDS PostgreSQL for the database

Troubleshooting

Container won't start

Check logs:

docker logs llmops-standalone

Common issues:

Invalid POSTGRES_URL format
PostgreSQL not accessible from container
Port 3000 already in use

Database connection errors

If using Docker Postgres on the same host:

Use host.docker.internal instead of localhost in POSTGRES_URL
Or use Docker Compose networking

POSTGRES_URL=postgresql://postgres:password@host.docker.internal:5432/pg_llmops

Provider configuration

Configure providers through the dashboard UI, not environment variables
API keys are stored in the PostgreSQL database
Each workspace has independent provider configurations

Next Steps

Configuration - Learn about all configuration options
Providers - Configure different LLM providers
API Reference - Explore the complete API

Standalone Docker Deployment