PromptLab Evaluation

A systematic evaluation framework for comparing prompt engineering strategies across 16 LLMs and 5 NLP benchmarks. Each model is evaluated 10 times per benchmark to provide statistically robust results (mean and standard deviation).

Benchmarks

Benchmark	Task	Type	Labels	Samples	Metric
IMDB	Sentiment Classification	Classification	Positive / Negative	1,000 (stratified)	classification metrics
MMLU	Multi-Choice QA	Classification	A / B / C / D	1,000	classification metrics
Toxic-Chat	Toxicity Detection	Classification	Toxic / Not Toxic	1,000 (stratified)	classification metrics
IWSLT2017	AR → EN Translation	Generation	Free text	1,000	BLEU, chrF
XSUM	News Summarization	Generation	Free text	1,000	ROUGE, BLEU

Models

16 models spanning 135M to 72B parameters (IWSLT uses 13 — SmolLM2 models are excluded as they do not support Arabic):

Family	Sizes
SmolLM2	135M, 360M, 1.7B
Qwen2.5	0.5B, 1.5B, 3B, 7B, 32B, 72B
Gemma 3	1B, 4B, 27B
Llama	3.2-1B, 3.1-8B, 3.3-70B
GPT-OSS	20B

Prompts

Each benchmark includes 15-18 prompts authored by 5-6 team members, covering diverse prompting strategies:

• Direct — Straightforward classification/generation instructions
• Chain-of-Thought (CoT) — Step-by-step reasoning before answering
• Role-Playing — "Act as an expert" persona framing
• Few-Shot — Examples provided within the prompt
• Minimal — Bare-minimum formatting to test instruction-following
• Negation / Creative — Unconventional prompt structures

Prompts are defined in prompts.json within each benchmark directory.

Project Structure

promptlab-evaluation/
├── imdb/                          # IMDB sentiment classification
│   ├── prompts.json               # Prompt definitions
│   ├── utils.py                   # Normalization, metrics, sampling
│   ├── evaluate_vllm_model.py     # Local GPU inference (vLLM)
│   ├── evaluate_openrouter_model.py  # API-based inference
│   ├── evaluate_models.sh         # Batch automation (16 models × 10 runs)
│   ├── aggregate_results.py       # Mean ± std aggregation
│   ├── sbatch_wrap.sh             # SLURM job wrapper
│   └── results/                   # Per-model JSON + Markdown reports
├── mmlu/                          # MMLU multiple-choice QA
├── toxic-chat/                    # Toxic-Chat toxicity detection
├── iwslt/                         # IWSLT2017 AR→EN translation
├── xsum/                          # XSUM news summarization
├── notebooks/                     # Visualization notebooks + PDF exports
├── pyproject.toml                 # Dependencies (managed by uv)
└── CLAUDE.md                      # Developer guide for AI-assisted coding

All five benchmarks follow the same structure. See CLAUDE.md for implementation details.

Quick Start

Prerequisites

• Python 3.13+
• NVIDIA GPUs with CUDA support
• uv package manager

Setup

git clone https://github.com/KFUPM-JRCAI/promptlab-evaluation.git
cd promptlab-evaluation
uv sync
source .venv/bin/activate

Run a Single Evaluation

cd imdb/
python evaluate_vllm_model.py --model Qwen/Qwen2.5-0.5B-Instruct --n-samples 1000

Run Full Benchmark Suite (10 runs per model)

cd imdb/
./evaluate_models.sh

On a SLURM cluster:

./sbatch_wrap.sh

Generate Aggregate Report

python aggregate_results.py --csv

This produces a Markdown report and optional CSV with mean ± std for each metric across all runs.

How It Works

Evaluation Pipeline

1. Prompt Loading — Reads prompts.json (15-18 prompt templates per benchmark)
2. Sampling — Draws 1,000 samples from the test split (stratified for classification tasks)
3. Batched Inference — All prompts x all samples are batched into a single vLLM call for maximum GPU throughput
4. Prediction Normalization — Maps raw LLM output to expected labels (unparseable outputs become "unknown" and count as incorrect)
5. Metrics — Computes accuracy, precision, recall, F1, and task-specific generation metrics
6. Reporting — Saves per-run JSON with full predictions and a Markdown summary table

Key Design Decisions

• Temperature 0.0 with seed 42 for deterministic outputs
• Text truncation to fit model context windows (reserves 32-64 tokens for classification output, 256-512 for generation)
• Conservative unknown handling — unparseable outputs are treated as incorrect
• Resume-safe automation — evaluate_models.sh detects existing runs and only fills gaps

Results

Results are stored per-model under each benchmark's results/ directory:

<benchmark>/results/
├── <model_name>/
│   ├── YYYYMMDD_HHMMSS.json          # Full predictions + metrics
│   └── report_YYYYMMDD_HHMMSS.md     # Formatted summary table
└── aggregate_report_YYYYMMDD_HHMMSS.md  # Mean ± std across 10 runs

Current Coverage

Benchmark	Models	Runs per Model	Total Evaluations
IMDB	16	10	160
MMLU	16	10	160
Toxic-Chat	16	10	160
IWSLT	13	10	130
XSUM	16	10	160
Total			770

Adding a New Model

1. Add the model path to the VLLM_MODELS array in evaluate_models.sh for each benchmark
2. If the model's attention heads are incompatible with the default tensor parallelism (4 GPUs), add an entry to MODEL_TENSOR_PARALLEL_SIZES
3. Run ./evaluate_models.sh — it will automatically run 10 evaluations

Adding a New Prompt

Edit prompts.json in the relevant benchmark directory:

{
  "name": "my_prompt",
  "template": "Classify the following text as positive or negative:\n{text}",
  "created_by": "your_name"
}

Template variables must match what the evaluation script provides ({text} for IMDB, {Question}/{A}/{B}/{C}/{D}/{Subject} for MMLU, {arabic_text} for IWSLT, {model_output} for Toxic-Chat, {document} for XSUM).

License

MIT - KFUPM-JRCAI