Google Cloud Managed Lustre cuts LLM inference costs 50% via KV cache offloading

Full analysis

Google Cloud has published a reference architecture combining Google Kubernetes Engine (GKE) and its Managed Lustre high-performance parallel filesystem to offload KV caches — the stored intermediate attention states that large language models reuse across requests — to an external shared storage tier rather than keeping them on GPU or local SSD. In benchmarks on a six-node A3 Mega cluster running Llama-3.3-70B with 50,000-token prompts, the setup achieved a 95% cache hit rate, cut GPU-hour requirements by nearly 60%, and delivered over 50% total-cost-of-ownership savings. A hybrid extension that also offloads to CPU RAM on top of Lustre further improved Time to First Token by ~40% and reduced end-to-end latency by ~30%. The architecture uses vLLM as the serving engine with an open-source llm-d offloading connector, plus a purpose-built PVC Evictor service for automated least-recently-used cache garbage collection, and targets models including Llama, Qwen 3.5, and Gemma 4. This matters for agent builders because long-context and multi-step agentic workloads generate especially large KV caches; shared external caching directly lowers the cost and latency of running those workloads at scale on GKE.