DevOps & Infrastructure

Distributed KV Caching

Sets up a shared L2 KV cache for self-hosted LLM inference on Amazon ElastiCache for Valkey, so GPU servers load prefill KV state over EFA RDMA instead of recomputing it. Checks fit, estimates savings with KVTrace, builds a business case, creates the cache, connects vLLM or SGLang (also as NVIDIA Dynamo or llm-d workers) through an LMCache server, tunes for cost or latency, and troubleshoots. Use when a builder serves open-weight models on GPUs (for example H100 or B200, on Amazon EC2 or Amazon EKS) for multi-turn chat, agents, RAG, or long contexts and asks about KV cache offloading, prefix cache sharing, prompt caching for self-hosted models, LMCache with Valkey, or lower TTFT or GPU cost. Not for Bedrock prompt caching, LLM response or semantic caching, or general application caching.

Publisher / source
aws/agent-toolkit-for-aws (found on skills.sh)
Upstream record
aws/agent-toolkit-for-aws
Version
Version 1
Published
Publication review
Approved for publication
Content fingerprint
a2dc62729c8159caa0b5c83635f1f62b249ac03ecc790cc12f4bed60ac8340ad