View original page

Distributed KV Caching

DevOps & Infrastructure

Sets up a shared L2 KV cache for self-hosted LLM inference on Amazon ElastiCache for Valkey, so GPU servers load prefill KV state over EFA RDMA instead of recomputing it. Checks fit, estimates savings with KVTrace, builds a business case, creates the cache, connects vLLM or SGLang (also as NVIDIA Dynamo or llm-d workers) through an LMCache server, tunes for cost or latency, and troubleshoots. Use when a builder serves open-weight models on GPUs (for example H100 or B200, on Amazon EC2 or Amazon EKS) for multi-turn chat, agents, RAG, or long contexts and asks about KV cache offloading, prefix cache sharing, prompt caching for self-hosted models, LMCache with Valkey, or lower TTFT or GPU cost. Not for Bedrock prompt caching, LLM response or semantic caching, or general application caching.

Publisher / source

aws/agent-toolkit-for-aws (found on skills.sh)

Upstream record

aws/agent-toolkit-for-aws

Version

Version 1

Published

2026-11-20

Publication review

Approved for publication

Content fingerprint

a2dc62729c8159caa0b5c83635f1f62b249ac03ecc790cc12f4bed60ac8340ad

Review before you use it

Publication approval is a recorded review state, not a guarantee of safe execution. Check the upstream license, required tools and permissions, and test in your intended client.

Sign in to inspect the full instructions and download the SKILL.md file. This public page contains metadata only.

Open skill in your workspace


Source: https://prompeteer.ai/skills-hub/406b26cd-7468-490b-81c7-c5b1b1c6ed5c