Executive Summary
Deploying large language models into multi-tenant production architectures presents distinct infrastructural hurdles: volatile upstream latencies (800ms–3000ms), strict third-party provider rate limits, skyrocketing API costs from repeated prompts, and single-point-of-failure risks.
LLMProxy is a high-performance Layer-7 API gateway engineered in Go. It acts as an authoritative reverse proxy sitting between client applications and upstream model providers. By combining atomic Redis Lua rate-limiting, tenant-isolated prompt caching, and intelligent upstream load balancing with circuit breaking, LLMProxy delivers enterprise-grade resilience and sub-15ms cached responses.
+---------------------------------------+
| Client Applications / SDKs |
| (Tenant Bearer Token) |
+-------------------+-------------------+
|
v
+-------------------+-------------------+
| LLMProxy L7 Gateway |
+-------------------+-------------------+
|
+---------------------------------+---------------------------------+
| | |
v v v
+-------+---------------+ +-------+---------------+ +-------+---------------+
| RBAC & Tenant Auth | | Redis Lua Limiter | | Prompt Cache Engine |
| (Scoped Permissions) | | (Atomic Token-Bucket) | | (Tenant-Isolated Key) |
+-----------------------+ +-----------------------+ +-------+---------------+
|
+----------------------------+
| Cache Hit (< 15ms)
v
[ Instant 200 OK Response ]
|
v Cache Miss
+------------------------+--------------+
| Dynamic Health & Failover Matrix |
+---------+-------------------+---------+
| |
v v (On 429/5xx Timeout)
+---------+---------+ +-------+---------+
| Primary Provider | | Secondary Host |
| (OpenAI / Claude) | | (Local / Cloud) |
+-------------------+ +-----------------+Architecture & Technical Deep Dive
1. Atomic Token-Bucket Rate Limiting (Redis + Lua)
Standard application-level rate limiters suffer from race conditions when running across horizontally scaled gateway replicas. Incurring multiple round-trips to Redis (e.g. GET token count, calculate leak, SET new count) introduces concurrency drift and latency overhead.
LLMProxy solves this by pushing the token-bucket algorithm directly into Redis using an atomic Lua Script:
- State evaluation and token replenishment occur in a single atomic server-side execution.
- Prevents burst exploitation and guarantees strict multi-tenant quotas across distributed gateway instances without database locking.
2. Tenant-Isolated Prompt Caching (Latency: 800ms ➔ 12ms)
A large fraction of LLM traffic consists of duplicate queries (e.g. documentation search, common system prompts, and classification pipelines).
- Key Derivation: Cache keys are cryptographically derived from
Hash(TenantID + ModelID + Temperature + NormalizedPrompt). - Security Isolation: Tenant isolation is enforced at the key namespace level, preventing prompt data leaks across different tenants.
- Performance: Cache hits bypass external model provider round-trips entirely, dropping response times from 800ms–2500ms down to ~12ms.
3. Upstream Load Balancing & Automatic Circuit Breaking
Upstream model endpoints frequently suffer from sudden rate-limiting throttles (HTTP 429) or localized latency degradation:
- Routing Algorithms: Supports configurable
round-robin,weighted, andleast-connectionsload distribution across configured model endpoints. - Active Health Probes: Continuously monitors upstream status codes and latency. If an upstream provider exceeds failure thresholds, the circuit breaker opens and traffic is immediately diverted to healthy fallback providers without dropping client connections.
Engineering Impact
- 98% Latency Reduction on Cached Prompts: Drops typical LLM latency down to microsecond network cache hits.
- Zero-Downtime Provider Failover: Transparently shields client applications from external AI provider outages.
- Unified Observability: Direct Prometheus metric export tracking requests per second, cache hit ratios, token consumption, and upstream latency histograms.