Skip to content
BranchesTags

Repository files navigation

CRBench: Context Resource Benchmark

Status: Research Preview Paper PDF License: MIT Python 3.9+

A Method-Agnostic, Resource-Aware Evaluation Framework for Long-Context Large Language Models

Preprint PDF • Quickstart • Empirical Results • Custom Adapter • Documentation • Paper Source

Overview

As Large Language Models (LLMs) expand context windows beyond 32K, 128K, and 1M tokens, the Key-Value (KV) cache becomes the primary operational bottleneck. For an 8B-parameter model in 16-bit precision, caching a single 128K context sequence consumes 16.0 GiB (17.18 GB)—exceeding the model's static weight footprint and exhausting consumer GPU VRAM.

While dozens of context compression paradigms have emerged—quantization, token eviction, pooling, low-rank projection, and factorized states—current evaluation practices suffer from two major flaws:

  1. Quality–Resource Disconnection: Accuracy benchmarks measure task capability without tracking physical memory or system latency.
  2. Confounding Base Model Strength: Absolute scores conflate base-model intelligence with compression algorithm efficiency.

CRBench resolves these challenges by introducing a query-level, dense-anchored evaluation primitive that quantifies exactly how much contextual capability an LLM retains relative to the memory and compute resources it consumes.

CRBench Framework Overview

Figure 1: CRBench Method-Agnostic Evaluation Framework Architecture. Decouples evaluation into Part 1 (Algorithmic Representation Utility $\mathcal{S}_{\text{res}}$) and Part 2 (Hardware System Track $\mathcal{S}_{\text{sys}}$), anchored pairwise to the uncompressed dense baseline on every individual prompt.

Core Scoring Formulation

Every candidate memory method $\mathcal{A}$ is evaluated pairwise against the model's own uncompressed 16-bit reference $\mathcal{R}_{\text{dense}}$ on the identical query $(x_i, y_i)$:

1. Model-Relative Quality Retention ($Q \in [0, 100]$)

$$Q_i = \min\left(100.0, ; \max\left(0.0, ; \frac{s_{i, \text{method}} - s_{\text{floor}}}{\max(\Delta_{\min}, ; s_{i, \text{dense}} - s_{\text{floor}})} \times 100.0\right)\right)$$

  • Isolates representation fidelity from base-model capability.
  • Dynamic range gate $\Delta_{\min} = 0.05$ prevents numerical instability on small or saturated models.
  • Clamped strictly to $[0.0, 100.0]$.

2. Analytical & Physical Memory Efficiency ($R_{\text{mem}} \in [0, 100]$)

$$R_{\text{mem}, i} = 100.0 \times \max\left(0.0, ; 1.0 - \frac{M_{i, \text{method}}}{M_{i, \text{dense}}}\right)$$

  • Tracks raw tensor payloads ($M_{\text{algo}}$) plus mandatory metadata ($M_{\text{meta}}$) such as quantization scaling factors, codebooks, eviction index bitmaps, and alignment overheads ($M_{\text{align}}$).
  • Normalizes storage into effective bits per stored KV element ($b_{\text{eff}}$, where uncompressed FP16 $= 16.0,\text{bits/elem}$).

3. Part 1 Resource Utility Score ($\mathcal{S}_{\text{res}}$)

$$\mathcal{S}_{\text{res}, i} = \alpha \cdot Q_i + (1 - \alpha) \cdot R_{\text{mem}, i} \quad (\text{default } \alpha = 0.70)$$

  • Under $\alpha = 0.70$ (quality-dominant operating point):
    • Uncompressed Dense Baseline ($Q=100%, R=0%$): $\mathcal{S}_{\text{res}} = 70.0$.
    • High-Fidelity 4-Bit Method ($Q=94%, R=75%$): $\mathcal{S}_{\text{res}} = 88.3$ (recognized as superior resource utility).
    • Degraded 2-Bit Method ($Q=10%, R=87.5%$): $\mathcal{S}_{\text{res}} = 33.3$ (correctly penalized below dense).

Empirical Results & Tradeoff Frontier

Preliminary validation on Qwen/Qwen2.5-1.5B-Instruct across 5 contextual tasks up to 4,096 tokens (Apple Silicon MPS prototyping profile) demonstrates the empirical Pareto frontier:

Empirical Pareto Frontier

Figure 2: Empirical Quality–Memory Pareto Frontier. Non-dominated methods form the frontier: Dense Baseline → DKV (High Preset) → DKV (Mid Preset) → Low-Rank KV (SVD) → SnapKV → INT2 Quantization.

Multi-Model Canonical Publication Leaderboards (NVIDIA RTX GPU)

Evaluated under Option B Physical Memory Aggregation ($\mathcal{S}{\text{res}} = 0.70 \cdot Q{\text{abs}} + 0.30 \cdot \mathcal{R}{\text{mem}}^{\text{agg}}$) and Canonical Part 2 Hardware Serving Utility ($\mathcal{S}{\text{sys}}$):

1. Gemma 4 E2B (Full 2,048 to 131,072 Token Context Ladder)

Rank Method Accuracy ($Q_{\text{abs}}$) Realized $b_{\text{eff}}^{\text{agg}}$ Physical VRAM Saved Part 1 Score ($\mathcal{S}_{\text{res}}$) Part 2 Score ($\mathcal{S}_{\text{sys}}$)
1 snapkv 46.8% 6.00 bpt 62.5% saved 51.5 60.6
2 kivi_style_kv_quant 46.4% 6.01 bpt 62.4% saved 51.2 64.6
3 dkv_mid (128 INT4) 44.7% 5.46 bpt 65.9% saved 51.0 59.1
4 dkv_high (256 INT4) 46.7% 10.78 bpt 32.6% saved 42.5 52.7
5 low_rank_kv 31.2% 6.07 bpt 62.1% saved 40.5 57.4
6 streaming_llm 22.9% 6.00 bpt 62.5% saved 34.8 54.3
7 dense_fp16 (Reference) 46.4% 16.00 bpt 0.0% saved 32.5 52.8
8 kv_merging 13.3% 6.01 bpt 62.5% saved 28.0 49.6

2. Gemma 4 E4B (2,048 to 32,768 Context Window)

Rank Method Accuracy ($Q_{\text{abs}}$) Realized $b_{\text{eff}}^{\text{agg}}$ Physical VRAM Saved Part 1 Score ($\mathcal{S}_{\text{res}}$) Part 2 Score ($\mathcal{S}_{\text{sys}}$)
1 kivi_style_kv_quant 62.0% 6.04 bpt 62.2% saved 62.1 73.4
2 snapkv 60.0% 6.00 bpt 62.5% saved 60.8 48.6
3 dkv_mid (128 INT4) 55.3% 5.00 bpt 68.7% saved 59.4 66.3
4 dkv_high (256 INT4) 61.3% 10.68 bpt 33.3% saved 52.9 60.0
5 low_rank_kv 48.8% 6.27 bpt 60.8% saved 52.4 65.9
6 dense_fp16 (Reference) 64.0% 16.00 bpt 0.0% saved 44.8 61.4
7 streaming_llm 25.5% 6.00 bpt 62.5% saved 36.6 55.6
8 kv_merging 25.3% 6.03 bpt 62.3% saved 36.4 55.5

3. Qwen2.5 7B (2,048 to 32,768 Context Window)

Rank Method Accuracy ($Q_{\text{abs}}$) Realized $b_{\text{eff}}^{\text{agg}}$ Physical VRAM Saved Part 1 Score ($\mathcal{S}_{\text{res}}$) Part 2 Score ($\mathcal{S}_{\text{sys}}$)
1 kv_quant (KIVI) 60.7% 6.04 bpt 62.2% saved 61.1 72.8
2 snapkv 58.2% 6.00 bpt 62.5% saved 59.5 53.7
3 dkv_mid (128 INT4) 53.3% 5.84 bpt 63.5% saved 56.3 54.2
4 dkv_high (256 INT4) 59.1% 12.14 bpt 24.1% saved 48.6 48.5
5 streaming_llm 36.5% 6.00 bpt 62.5% saved 44.3 61.0
6 dense_fp16 (Reference) 60.0% 16.00 bpt 0.0% saved 42.0 59.4
7 low_rank_kv 28.6% 6.09 bpt 62.0% saved 38.6 54.9
8 kv_merging 19.9% 6.03 bpt 62.3% saved 32.6 52.8

Query-Level Evaluation Pipeline

Atomic Query Pipeline

Figure 3: Atomic Query-Level Evaluation Pipeline in CRBench. Every sample executes dual inference passes against the dense reference before metric extraction and non-destructive score logging.

Installation

# Clone repository
git clone https://github.com/Omc12/CRBench.git
cd CRBench

# Setup Python environment
python -m venv .venv
source .venv/bin/activate

# Install in editable mode with development dependencies
pip install -e ".[dev]"

Quickstart

1. Evaluate a Single Query (Atomic CLI Primitive)

crbench evaluate \
  --model "Qwen/Qwen2.5-0.5B-Instruct" \
  --query "What is the secret passkey?" \
  --context "The secret passkey is 987123." \
  --ground-truth "987123" \
  --method "kv_quant_int4" \
  --budget 4.0 \
  --dense

Output:

========================================================================
CRBench Query Evaluation Summary
========================================================================
Query ID:             cli_query_001
Task:                 cli_evaluation_task (Context: 512 tokens)
Model:                Qwen/Qwen2.5-0.5B-Instruct
Method:               kv_quant_int4 (Budget: 4.0 bits/elem)
Status:               SUCCESS
------------------------------------------------------------------------
Quality Metrics:
  Dense raw score:     1.000
  Method raw score:    1.000
  Quality retained:    100.0%
Resource Metrics:
  Dense memory:        0.500 GB (16.0 bits/elem)
  Method memory:       0.133 GB (4.25 bits/elem)
  Resource efficiency: 73.4% savings
Benchmark Score:
  CRBench Part 1 score: 92.02 (Formula: linear, α=0.70)
========================================================================

2. Evaluate a Dataset (Query Aggregation)

crbench evaluate-dataset \
  --model "Qwen/Qwen2.5-0.5B-Instruct" \
  --dataset "single_niah" \
  --method "snapkv" \
  --budget 4.0 \
  --context-lengths 2048 --context-lengths 4096 \
  --samples 10 \
  --output-dir "results/dataset_snapkv"

3. Non-Destructive Score Recomputation

Recompute benchmark scores under alternative $\alpha$ preference weights or scoring formulas instantaneously without re-running expensive inference:

crbench recompute \
  --raw-file "results/dataset_snapkv/raw_results_v1.json" \
  --alpha 0.80 \
  --formula linear

Integrating a Custom Method (< 50 Lines)

Researchers can evaluate any novel KV representation by subclassing BaseContextAdapter and registering it with @Registry.register_adapter:

import torch
from crbench.core.adapter import BaseContextAdapter, KVStateMetadata
from crbench.core.registry import Registry

@Registry.register_adapter("my_custom_kv_method")
class MyCustomKVMethod(BaseContextAdapter):
    """Custom KV representation adapter example."""

    @property
    def method_type(self) -> str:
        return "custom"  # 'quantized', 'eviction', 'merging', 'low_rank', 'custom'

    def apply_budget(self, budget: float, context_length: int) -> None:
        """Configure adapter hyperparameters based on assigned budget."""
        super().apply_budget(budget, context_length)
        self.target_rank = int(budget)

    def forward_or_generate(
        self,
        input_ids: torch.Tensor,
        attention_mask=None,
        max_new_tokens: int = 32,
        **kwargs
    ) -> torch.Tensor:
        """Autoregressive generation with custom KV representation."""
        return self.model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_new_tokens=max_new_tokens,
            **kwargs
        )

    def get_kv_metadata(self, context_length: int) -> KVStateMetadata:
        """Report exact payload bytes and auxiliary metadata storage."""
        num_layers = getattr(self.model.config, "num_hidden_layers", 24)
        num_kv_heads = getattr(self.model.config, "num_key_value_heads", 2)
        head_dim = getattr(self.model.config, "head_dim", 64)

        total_elements = 2 * num_layers * num_kv_heads * head_dim * context_length
        algorithmic_bytes = total_elements * 0.5            # 4-bit payload (0.5 bytes/elem)
        metadata_bytes = (total_elements / 32.0) * 2.0     # FP16 scale factors per group of 32
        effective_bpe = (algorithmic_bytes + metadata_bytes) * 8.0 / max(1, total_elements)

        return KVStateMetadata(
            adapter_name=self.name,
            method_type=self.method_type,
            effective_bits_per_element=effective_bpe,
            total_tokens_stored=context_length,
            context_length=context_length,
            num_layers=num_layers,
            num_kv_heads=num_kv_heads,
            head_dim=head_dim,
            algorithmic_bytes=algorithmic_bytes,
            metadata_overhead_bytes=metadata_bytes
        )

Supported Methods & Tasks

Compression Methods Taxonomy

  • Quantization: FP8, INT8, INT4, INT2 (per-channel and grouped dynamic scaling with outlier preservation)
  • Eviction & Sparsification: SnapKV, StreamingLLM, $\text{H}_2\text{O}$, Scissorhands (attention sinks, heavy-hitter key eviction)
  • Merging & Pooling: Temporal and semantic token clustering
  • Low-Rank State: Spectral and SVD latent subspace projection
  • Factorized Context Memory: Dynamic Key-Value (custom_dkv) subspace factoring

Benchmark Evaluation Suite

  • Needle-In-A-Haystack (NIAH): Single-target and multi-target associative retrieval
  • RULER Benchmark: Multi-variable tracking and high-entropy key-value association
  • Multi-Hop QA: Cross-document multi-step synthetic reasoning
  • LongBench Suite: Multi-task document QA and summarization

Hardware Support & Error Transparency

  • NVIDIA CUDA: Synchronized GPU event profiling, CUDA memory allocators, FlashAttention-2 integration.
  • Apple Silicon (MPS) & CPU: Full cross-platform local development and prototyping.
  • Strict Failure Categorization: Errors are never silently assigned zero; explicit diagnostic codes (OOM, UNSUPPORTED_PRECISION, RUNTIME_ERROR) are recorded in benchmark logs.

Research Preprint & Citation

For methodological derivations, desiderata proofs, and complete analysis, see the academic preprint:

CRBench: A Method-Agnostic, Resource-Aware Evaluation Framework for Long-Context Large Language Models
Om Chimurkar
Research Preprint (Version 0.2.0) — PDF Available Here

@article{chimurkar2026crbench,
  title={CRBench: A Method-Agnostic, Resource-Aware Evaluation Framework for Long-Context Large Language Models},
  author={Chimurkar, Om},
  journal={arXiv preprint (Research Preview v0.2.0)},
  year={2026},
  url={https://github.com/Omc12/CRBench}
}

License

CRBench is open-source software licensed under the MIT License.

About

CRBench is a method-agnostic benchmark for long-context LLMs that measures contextual capability retained relative to a dense reference alongside memory and system resource efficiency.

www.researchgate.net/publication/413568410

Topics

benchmarkingmemory-efficiencymodel-evaluationresource-aware-evaluation

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages