rubric-based-evaluation

This is my personal home rig for serious LLM experimentation. I built it to test models head-to-head, create custom evaluation rubrics, automatically improve prompts based on the previous run’s results, and generate high-quality synthetic training data. Everything runs locally first (Ollama by default), with optional cloud support. logged locally.

Analyze Claude Code session logs and generate efficiency reports, cost diagnostics, and actionable recommendations. This project reads local JSONL session logs, computes deterministic efficiency signals, and can optionally add local LLM recommendations using Ollama.

Bilingual LLM annotation dataset — EN/PT quality evaluation

Pondera is a lightweight, YAML-first framework to evaluate AI models and agents with pluggable runners and an LLM-as-a-judge.

A research tool that scores a human-AI conversation to show whether the person got more capable or just more dependent. Claude rates cognitive agency, prompt steering and critical discernment phase by phase; the code checks the arithmetic and the evidence, and a calibration loop lets a human correct the reviewer over time.

Free short books about the engineering problems that show up after the demo works. Start with AI agent evaluation.

Research repository for the Visual Nudges study, examining how lightweight interface interventions structure analytic judgment in visualization-based peer review.

A multi-turn agentic dialogue system built on hybrid dense-sparse retrieval, hierarchical agent coordination, and rubric-based evaluation. Designed for real-world deployment with FastAPI serving, streaming responses, and full observability

How Chartula is measured: eval cases, run costs, and judgement of the generated changelogs.