Introduction

LLMOps is a framework-agnostic, universal LLM operations toolkit for TypeScript. It provides a comprehensive set of features out of the box. Whether you need prompt versioning or multi-environment deployments, it lets you focus on building your application instead of reinventing the wheel.

Features

LLMOps aims to be the most comprehensive LLMOps toolkit. It provides a wide range of features out of the box and allows you to extend it with plugins. Here are some of the features:

• AI Gateway - OpenAI-compatible API to access 1600+ LLMs from 70+ providers
• Prompt Management - Version, test, and deploy prompts with a visual editor
• Environment Management - Production, staging, development with secure secrets
• Framework Agnostic - Works with Express, Hono, and more
• TypeScript-First - Strict type safe SDK
• Self-Hosted - Full control over your data and infrastructure

...and much more!

Quick Setup

1. Install

npm install @llmops/sdk pg

2. Configure

import { llmops } from '@llmops/sdk';
import { Pool } from 'pg';

export default llmops({
  basePath: '/llmops',
  providers: {
    openai: { apiKey: process.env.OPENAI_API_KEY },
    anthropic: { apiKey: process.env.ANTHROPIC_API_KEY },
  },
  database: new Pool({
    connectionString: process.env.DATABASE_URL,
  }),
  auth: {
    type: 'basic',
    defaultUser: 'admin@example.com',
    defaultPassword: 'password',
  },
  autoMigrate: true,
});

3. Mount Middleware

import express from 'express';
import { createLLMOpsMiddleware } from '@llmops/sdk/express';
import llmopsClient from './llmops';

const app = express();
app.use('/llmops', createLLMOpsMiddleware(llmopsClient));
app.listen(3000);

Architecture

LLMOps is organized as a monorepo with focused packages:

Next Steps

• Quickstart - Get up and running in minutes