LLM Ops Cheatsheet: Best Practices for LLM Infrastructure

LLM Ops Cheatsheet

Monitoring Key Metrics

• Latency: P50, P95, P99 response times
• Throughput: Tokens per second (TPS)
• Error Rate: 4xx/5xx responses
• Cost: Cost per 1K tokens
• Token Usage: Input vs output token ratios

Infrastructure Patterns

Model Serving

• Use vLLM or TGI for high-throughput serving
• Implement request batching (batch size: 32-128)
• Enable continuous batching for optimal GPU utilization

Caching Layer

• Cache embeddings for repeated queries
• Implement prompt-level caching with TTL
• Use Redis or Memcached for low-latency access

Scaling Strategies

1. Horizontal scaling: Add more model replicas
2. Vertical scaling: Increase GPU memory per instance
3. Load balancing: Distribute across regions
4. Circuit breakers: Fail gracefully under load