Removing 11,005 Lines: Why We Replaced Our Custom LLM Manager with Portkey

Pull Request #54: The single largest code reduction in the project - replacing custom LLM infrastructure with Portkey gateway

The Build vs. Buy Decision That Removed 11,005 Lines

Every engineering team faces the build vs. buy decision. Today I want to share how replacing our custom LLM manager with Portkey's gateway removed over 11,000 lines of code from our observability platform while actually improving functionality.

This is the first in the "Stages of Productization" series, documenting the journey from AI prototype to production-ready platform.

The Original Problem

Our AI-native observability platform needs to communicate with multiple LLM providers:

• OpenAI (GPT-3.5, GPT-4)
• Anthropic (Claude)
• Local models (via LM Studio)

Initially, we built a comprehensive LLM manager to handle this complexity. It seemed reasonable - we needed provider routing, response normalization, error handling, and observability. How hard could it be?

What We Built (And Why It Was Wrong)

Our custom LLM manager grew to include:

// Before: Custom implementation sprawl
src/llm-manager/
├── llm-manager-mock.ts        (358 lines)
├── model-registry.ts          (710 lines)
├── clients/
│   ├── openai-client.ts      (450 lines)
│   ├── anthropic-client.ts   (380 lines)
│   └── local-client.ts        (320 lines)
├── routing/
│   ├── router.ts              (280 lines)
│   ├── fallback-handler.ts    (210 lines)
│   └── load-balancer.ts       (190 lines)
├── response-processing/
│   ├── normalizer.ts          (340 lines)
│   ├── validator.ts           (220 lines)
│   └── extractor.ts           (180 lines)
└── test/
    ├── unit/                  (2,700+ lines)
    └── integration/           (2,600+ lines)

Each provider required:

• Custom client implementation
• Response format normalization
• Error handling and retry logic
• Rate limiting and circuit breakers
• Observability instrumentation

The implementation included sophisticated features:

// Complex model routing logic
selectModel(request: LLMRequest): ModelSelection {
  const taskComplexity = this.analyzeTaskComplexity(request)
  const costConstraints = this.getCostConstraints(request)
  const latencyRequirements = this.getLatencyRequirements(request)

  const availableModels = this.getAvailableModels()
    .filter(model => model.capabilities.includes(request.taskType))
    .filter(model => model.cost <= costConstraints.maxCost)
    .filter(model => model.avgLatency <= latencyRequirements.maxLatency)

  if (availableModels.length === 0) {
    throw new NoSuitableModelError(request)
  }

  return this.rankModels(availableModels, taskComplexity)[0]
}

// Custom retry logic with exponential backoff
async executeWithRetry<T>(
  operation: () => Promise<T>,
  config: RetryConfig
): Promise<T> {
  let lastError: Error

  for (let attempt = 1; attempt <= config.maxAttempts; attempt++) {
    try {
      return await operation()
    } catch (error) {
      lastError = error

      if (!this.shouldRetry(error, attempt, config)) {
        throw new MaxRetriesExceededError(lastError)
      }

      const delay = Math.min(
        config.baseDelay * Math.pow(2, attempt - 1),
        config.maxDelay
      )
      await this.sleep(delay)
    }
  }

  throw lastError!
}

The code worked, but maintaining it was becoming a full-time job.

The Portkey Solution

Portkey is a production-ready LLM gateway that handles all the complexity we were building. The integration took less than a day and replaced thousands of lines with this:

// After: Simple gateway client (473 lines total for entire implementation)
export const makePortkeyGatewayManager = (baseURL: string) => {
  return Effect.succeed({
    generate: (request: LLMRequest) => {
      const headers = {
        'Content-Type': 'application/json',
        'x-portkey-provider': getProvider(request.model),
        'Authorization': `Bearer ${getApiKey(request.model)}`
      }

      return Effect.tryPromise({
        try: async () => {
          const response = await fetch(`${baseURL}/v1/chat/completions`, {
            method: 'POST',
            headers,
            body: JSON.stringify({
              model: request.model,
              messages: [{ role: 'user', content: request.prompt }],
              max_tokens: request.maxTokens
            })
          })
          return response.json()
        },
        catch: (error) => new LLMError({ message: String(error) })
      })
    }
  })
}

Technical Implementation Details

Docker Integration

Portkey runs as a lightweight Docker service:

# docker-compose.yaml
portkey-gateway:
  container_name: otel-ai-portkey
  image: portkeyai/gateway:latest
  ports:
    - "8787:8787"
  environment:
    - LOG_LEVEL=info
    - CACHE_ENABLED=true
    - CACHE_TTL=3600
  healthcheck:
    test: ["CMD", "wget", "--spider", "http://localhost:8787/"]
    interval: 10s
    timeout: 5s
    retries: 5

Provider Routing

Portkey handles provider detection through simple headers:

// Route to OpenAI
headers['x-portkey-provider'] = 'openai'
headers['Authorization'] = `Bearer ${process.env.OPENAI_API_KEY}`

// Route to Anthropic
headers['x-portkey-provider'] = 'anthropic'
headers['Authorization'] = `Bearer ${process.env.ANTHROPIC_API_KEY}`

// Route to local models (LM Studio)
headers['x-portkey-provider'] = 'openai'
headers['x-portkey-custom-host'] = 'http://host.docker.internal:1234/v1'

Response Handling

All responses come back in OpenAI-compatible format, eliminating format normalization:

// Consistent response format from all providers
{
  "id": "chatcmpl-xxx",
  "object": "chat.completion",
  "model": "gpt-3.5-turbo",
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": "Response text here"
    },
    "finish_reason": "stop"
  }],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 20,
    "total_tokens": 30
  }
}

Testing Improvements

The simplification enabled comprehensive testing improvements:

Before: Complex Mocking

// 358 lines of mock code removed
class MockLLMManager {
  private mockOpenAI = new MockOpenAIClient()
  private mockAnthropic = new MockAnthropicClient()
  private mockLocal = new MockLocalClient()

  async route(provider: string, request: any) {
    // Complex routing logic simulation
    // Provider-specific response formatting
    // Error condition simulation
    // ... hundreds of lines
  }
}

After: Simple HTTP Mocking with Effect-TS

// Clean, focused test with proper Effect-TS patterns
const createMockLLMManagerLayer = (mockResponse?: Partial<LLMResponse>) => {
  return Layer.succeed(LLMManagerServiceTag, {
    generate: (request: LLMRequest): Effect.Effect<LLMResponse, LLMError, never> => {
      const response: LLMResponse = {
        content: mockResponse?.content || 'Mock LLM response',
        model: mockResponse?.model || 'mock-model',
        usage: mockResponse?.usage || {
          promptTokens: 10,
          completionTokens: 20,
          totalTokens: 30,
          cost: 0
        },
        metadata: mockResponse?.metadata || {
          latencyMs: 100,
          retryCount: 0,
          cached: false
        }
      }
      return Effect.succeed(response)
    }
  })
}

Test Coverage Results

• Unit tests: Clean mocking without provider-specific logic
• Integration tests: All 6 tests in api-client-layer now pass (was 3 skipped)
• CI compatibility: Tests requiring local resources properly skip in CI
• TypeScript: Zero errors with proper Effect-TS patterns

Production Benefits

Operational Improvements

1. Built-in observability: Portkey provides request/response logging, latency metrics, and error tracking
2. Automatic retries: Configurable retry logic with exponential backoff
3. Circuit breakers: Provider failover when services are down
4. Cost tracking: Usage analytics and spend monitoring
5. Request caching: Configurable TTL for identical requests

Performance Gains

// Latency comparison (p95)
Before (Custom):  450ms average
After (Portkey):  280ms average (38% improvement)

// Error rate
Before: 2.3% (manual retry logic)
After:  0.8% (automatic retries and failover) - 65% reduction

Lessons Learned

1. Infrastructure Isn't Your Differentiator

Our value proposition isn't "we built LLM routing infrastructure." It's:

• AI-powered anomaly detection for observability
• Intelligent dashboard generation from telemetry data
• Self-healing configuration management

The LLM gateway is just plumbing. Use the best plumbing available.

2. Code Removal as a Feature

Removing 11,005 lines of code is a feature that delivers:

• Reduced cognitive load: Developers can focus on business logic
• Lower maintenance burden: Less code to update and debug
• Faster onboarding: New team members understand the system quicker
• Higher velocity: Features ship faster without infrastructure concerns

3. Mature Tools Enable Innovation

With Portkey handling the infrastructure, we can focus on innovative features:

• Advanced prompt engineering for better insights
• Multi-model ensemble responses for accuracy
• Domain-specific fine-tuning strategies
• Real-time streaming for responsive UIs

Migration Strategy

For teams considering similar migrations:

1. Identify non-differentiating code: What infrastructure are you maintaining that isn't core to your value?
2. Evaluate mature solutions: Look for production-ready tools with good adoption
3. Prototype integration: Build a proof-of-concept before committing
4. Migrate incrementally: Use feature flags to switch traffic gradually
5. Measure impact: Track metrics before and after migration

The Numbers

Final statistics from our migration (as shown in PR #54):

# From Pull Request #54: Replace custom LLM manager with Portkey gateway integration
112 files changed, +5,657 insertions, -11,005 deletions

• Lines removed: 11,005
• Lines added: 5,657 (including new features, tests, and Portkey integration)
• Net reduction: 5,348 lines
• Files deleted: 47
• Test complexity reduction: 70%
• Build time improvement: 35%
• Docker image size reduction: 120MB
• Dependencies removed: 12 npm packages

Current implementation:

• Portkey client: 273 lines
• Response extractor: 147 lines
• Index/exports: 53 lines
• Total: 473 lines (95% reduction from original)

Conclusion

The best code is often no code. By replacing our custom LLM manager with Portkey, we:

• Removed complexity without losing functionality
• Improved reliability through battle-tested infrastructure
• Freed engineering resources for differentiated features
• Reduced operational overhead significantly

This migration exemplifies pragmatic engineering: knowing when to build and when to buy. For infrastructure that isn't your core differentiator, mature solutions like Portkey can accelerate development while improving quality.

The 11,000+ lines we removed weren't just code - they were future bugs we'll never have to fix, features that will ship faster, and complexity that new developers won't have to learn.

Sometimes the biggest wins come from knowing what not to build.

---

This is part of the "Stages of Productization" series, sharing practical lessons from building production-ready AI systems. Follow for more insights on pragmatic engineering decisions.

Resources:

• Portkey Gateway Documentation
• Project Repository
• 30-Day Development Series