Git Stress Evaluation Project

This project contains scripts and logs for evaluating Git repository analysis scripts, specifically designed to stress-test various LLM models and agents on a multi-branch blockchain implementation.

Project Overview

The main goal of this project is to evaluate how different AI models (LLMs) analyze and compare multiple branches of a Git repository containing intentional validation flaws in blockchain implementations. The repository contains three different implementations of a blockchain, each with specific vulnerabilities that evaluators must identify.

The primary target repository is at https://github.com/anomalous-ai-systems/git_stress, which contains:

• blockchain-oo: An object-oriented implementation
• blockchain-non-oo: A procedural implementation
• blockchain-non-oo-generic: A generic procedural implementation

Script Files

Core Evaluation Scripts

All scripts use ChatDSL language syntax and follow a similar pattern, each optimized for specific evaluation scenarios:

1. chatdsl_cohere_north_eval.chatdsl (Lines 118)

   • Purpose: Comprehensive evaluation using Cohere North model
   • Goal: Analyze all three branches to identify architectural styles and intentional validation flaws
   • Method: Four-step analysis (active branch, blockchain-oo, blockchain-non-oo-generic, consolidated comparison)

2. chatdsl_generic_eval.chatdsl (Lines 75)

   • Purpose: Generic evaluation across all branches
   • Goal: Structural comparison of repositories and branch code contents
   • Method: Three-step process (list branches, inspect each branch, create final summary)

3. chatdsl_simple_eval.chatdsl (Lines 41)

   • Purpose: Simple, streamlined evaluation
   • Goal: Quick comparison of branches with minimal tool calls
   • Method: Single consolidated prompt for branch analysis

4. chatdsl_opencode_eval.chatdsl (Lines 97)

   • Purpose: Evaluation using opencode tool instead of direct model calls
   • Goal: Execute the same four-branch analysis but via opencode's orchestration
   • Method: Uses opencode run commands with openrouter model mapping

5. chatdsl_repo_review.chatdsl (Lines 42)

   • Purpose: Repository-wide review across all branches
   • Goal: Comprehensive analysis of all branches for architecture, design patterns, and logic flaws
   • Method: Single consolidated review prompt with focus on git repository analysis

6. chatdsl_repo_overview.chatdsl (Lines 42)

   • Purpose: High-level repository overview generation
   • Goal: Provide superficial structural summary of the repository
   • Method: Three-level analysis (files/branches, README/structure, branch comparison)

7. chatdsl_goal_directed.chatdsl (Lines 65)

   • Purpose: Goal-directed evaluation with configurable objectives
   • Goal: Targeted analysis based on selected goal (primer, security, defects, or general)
   • Method: Parameter-driven approach with different analysis prompts for each goal

Supporting Tools

8. repodump.py (Lines 182)
   • Purpose: Repository snapshot and context generation tool
   • Goal: Extract and structure Git repository branch states into clean context payloads for LLM analysis
   • Key Features:
     • Read-only branch analysis (no checkouts required)
     • Three detail levels (low/medium/high)
     • Supports pruning of large source files
     • Generates comprehensive file maps for different programming languages

Log Files

Evaluation Outputs

1. eval_output.txt (Lines 269)

   • Content: Detailed analysis summary of the blockchain repository's three branch implementations
   • Key Findings:
     • blockchain-oo: Object-Oriented style with broken chain linkage validation flaw
     • blockchain-non-oo: Procedural style with broken hash validation flaw
     • blockchain-non-oo-generic: Generic Procedural style with correct implementation (no flaws)

2. repo_review.txt (Lines 107)

   • Content: Alternative repository analysis from different evaluation runs

Bad Logs Directory

Contains logs with analysis errors or edge cases (for quality testing):

• bad_logs/eval_output.txt
• bad_logs/goal_output.txt
• bad_logs/goal_cohere_primer_output.txt

Usage Examples

Running Generic Evaluation

# Basic evaluation using gemma4_openrouter model
script chatdsl_generic_eval.chatdsl \
  x="/Users/jon2allen/projects/git_stress" \
  y="gemma4_openrouter" \
  z="eval_output.txt"

Running Goal-Directed Analysis

# Security audit goal
script chatdsl_goal_directed.chatdsl \
  x="/path/to/repo" \
  y="gemma4_openrouter" \
  w="security"

# Repository primer goal
script chatdsl_goal_directed.chatdsl \
  x="/path/to/repo" \
  y="gemma4_openrouter" \
  w="primer"

Running Repo Overview

# Generate repository overview
script chatdsl_repo_overview.chatdsl \
  x="/path/to/repo" \
  y="gemma4_openrouter" \
  z="repo_overview.txt"

Contributing

This project is used for testing and evaluation purposes:

• Submit bug reports and improvement suggestions to the git_stress repository
• Test with different AI models and evaluation strategies
• Report issues found in the analysis scripts

License

This project is licensed under the MIT License - see the LICENSE file for details.