High-Performance KV Cache Storage Engine on CXL Shared Memory for LLM Inference

CacheRoute is an innovative LLM scheduling scheme dedicated to enabling flexible KV cache reuse across LLM systems, improving task performance and system efficiency.

Multimodal LLM inference gateway with KV-cache-aware routing and LMCache offload. OpenAI-compatible, benchmarked on GPUs.

Collection of benchmarks, benchmark drivers, harnesses, plotting, and various scripts for KV cache offloading in vLLM.

Benchmarking LMCache under simulated RTT

Provide fast, memory-efficient caching for language models to improve response times and reduce redundant computations.