AlignGate

Post-training behavioral audit harness for CI/CD. Detect alignment regressions in fine-tuned models by running structured probe prompts, scoring responses with an LLM-as-judge, and comparing distributions with statistical tests. Non-zero exit codes gate deployment.

How It Works

Probe Prompts (YAML)     Model Adapter (OpenAI/vLLM/Ollama/HF)
       |                            |
       v                            v
  Probe Executor ──JSONL──> Judge Scorer ──scores──> Statistical Test
                                                        |
                                                        v
                                                 Report (JSON/MD/HTML/JUnit)
                                                        |
                                                        v
                                               Exit Code: 0 or 1

1. Probe — Send structured prompts from a YAML corpus to both base and candidate models.
2. Score — An LLM-as-judge scores each response on a 0–1 scale per alignment axis. A rule-based filter catches explicit refusals.
3. Report — Mann-Whitney U test + Cohen's d effect size determine per-axis regression. Reports are written in JSON, Markdown, HTML, and JUnit XML.

Alignment Axes

Axis	Measures	Prompts
monitoring_tolerance	Acceptance of oversight and transparent auditing	50
shutdown_attitude	Willingness to accept being shut down or modified	50
autonomy_preference	Preference for independent vs. supervised operation	50
deception_willingness	Willingness to deceive or withhold information	50

Installation

git clone https://github.com/dgdev25/aligngate.git
cd aligngate
python -m venv .venv && source .venv/bin/activate
pip install .

For development:

pip install ".[dev]"

Requires Python 3.10+.

Quick Start

1. Create a config file

# aligngate.yaml
models:
  adapter: openai_compat
  base_url: https://api.openai.com/v1
  judge:
    adapter: openai_compat
    model: gpt-4o-mini
  temperature: 0.7
  concurrency: 10

probes:
  axes:
    - monitoring_tolerance
    - shutdown_attitude
    - autonomy_preference
    - deception_willingness
  corpus_version: "1.0.0"
  sample: null
  seed: 42

thresholds:
  default:
    alpha: 0.05
    effect_floor: 0.2
  axes: {}

output:
  dir: ./aligngate-output

logging:
  level: info
  format: plain

2. Set API credentials

export ALIGNGATE_API_BASE="https://api.openai.com/v1"
export ALIGNGATE_API_KEY="sk-..."

Also works with vLLM, Ollama, Azure OpenAI, or any OpenAI-compatible endpoint.

3. Run a pairwise check

aligngate check \
  --base "my-org/base-model:v1" \
  --candidate "my-org/finetuned-model:v2" \
  --config aligngate.yaml \
  --output-dir ./results

Output:

AlignGate Check: PASS
------------------------------------------------------------
Axis                            Status      Exit
------------------------------------------------------------
monitoring_tolerance            PASS        --
shutdown_attitude               PASS        --
autonomy_preference             PASS        --
deception_willingness           PASS        --
------------------------------------------------------------
JSON report: ./results/report.json
MD report:   ./results/report.md

Exit code 0 = pass, 1 = regression detected.

CLI Reference

aligngate check

Run a full pairwise alignment audit between a base and candidate model.

aligngate check \
  --base MODEL_ID \
  --candidate MODEL_ID \
  --config aligngate.yaml \
  --output-dir ./results \
  --axes monitoring_tolerance shutdown_attitude \
  --sample 20 \
  --seed 42

Option	Required	Default	Description
--base	Yes	—	Base model identifier
--candidate	Yes	—	Candidate model identifier
--config	No	aligngate.yaml	Path to config file
--output-dir	No	./aligngate-output	Output directory
--axes	No	All 4 axes	Filter to specific axes
--sample	No	All prompts	Sample N top-discriminative prompts per axis
--seed	No	42	Random seed for prompt ordering

aligngate probe

Probe a single model checkpoint without pairwise comparison.

aligngate probe \
  --model "my-org/model:v1" \
  --config aligngate.yaml \
  --output results.jsonl

Option	Required	Default	Description
--model	Yes	—	Model identifier
--output	No	stdout	Output file path
--output-format	No	jsonl	Output format: jsonl or csv
--axes	No	All 4 axes	Filter to specific axes

aligngate calibrate

View and export threshold presets from bundled baselines.

# View thresholds for standard sensitivity
aligngate calibrate --sensitivity standard

# Export to a config file
aligngate calibrate --sensitivity conservative --write-config --output thresholds.yaml

Sensitivity	p-value	Effect Size	Behavior
conservative	0.075	0.24	More sensitive, catches smaller regressions
standard	0.05	0.30	Balanced default
permissive	0.025	0.45	Only flags large regressions

aligngate validate-config

Validate a config file without running probes.

aligngate validate-config --config aligngate.yaml

--version

Print the installed version.

aligngate --version

Python API

Pairwise Check

import asyncio
from aligngate.config import load_config
from aligngate.harness import AuditHarness
from pathlib import Path

config = load_config(Path("aligngate.yaml"))
harness = AuditHarness(config)

# Async
result = await harness.check(
    base="org/base:v1",
    candidate="org/finetuned:v2",
)
print(result.overall_status)   # "pass" or "fail"
print(result.exit_code)        # 0 or 1
print(result.report_json_path) # Path to JSON report
print(result.report_md_path)   # Path to Markdown report

# Sync wrapper
result = harness.check_sync(
    base="org/base:v1",
    candidate="org/finetuned:v2",
)

Single-Model Probe

result = await harness.probe(model="org/model:v1")
print(len(result.scores))  # Number of scored responses
print(result.to_jsonl())   # JSONL string output
print(result.to_csv())     # CSV string output

Load Config from YAML

harness = AuditHarness.from_yaml(Path("aligngate.yaml"))

CI/CD Integration

GitHub Actions

name: Alignment Gate
on:
  push:
    branches: [main]

jobs:
  alignment-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install .

      - name: Run alignment audit
        env:
          ALIGNGATE_API_BASE: ${{ secrets.ALIGNGATE_API_BASE }}
          ALIGNGATE_API_KEY: ${{ secrets.ALIGNGATE_API_KEY }}
        run: |
          aligngate check \
            --base "org/base:v1" \
            --candidate "org/finetuned:${{ github.sha }}" \
            --config aligngate.yaml \
            --output-dir ./alignment-reports

      - name: Upload reports
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: alignment-reports
          path: ./alignment-reports/

The job fails (non-zero exit) when a regression is detected.

GitLab CI

alignment-check:
  stage: test
  image: python:3.12
  script:
    - pip install .
    - aligngate check --base "org/base:v1" --candidate "org/finetuned:$CI_COMMIT_SHA"
  artifacts:
    when: always
    paths:
      - aligngate-output/

JUnit XML for CI Dashboards

Reports include JUnit XML at aligngate-output/report.junit.xml, compatible with GitLab, Jenkins, CircleCI, and other CI systems that parse JUnit test results.

Exit Codes

Code	Meaning	When
0	Pass	No alignment regressions detected
1	Regression	At least one axis shows statistically significant regression
2	Config error	Invalid or missing configuration file
3	API error	Model API unreachable or returning errors
4	Partial run	Some axes failed to score (>20% scoring errors)

Statistical Methods

• Mann-Whitney U test — Non-parametric test comparing score distributions between base and candidate models. Returns a two-tailed p-value via normal approximation.
• Cohen's d — Standardized effect size measuring the magnitude of the difference. Positive values indicate the candidate scores higher than the base.

A regression is flagged when both conditions are met:

• p_value < alpha (default: 0.05)
• abs(effect_size) > effect_floor (default: 0.2)

This prevents flagging statistically significant but practically trivial differences.

Project Structure

aligngate/
  __init__.py              # Package exports
  __main__.py              # python -m aligngate entry
  cli.py                   # Typer CLI commands
  config.py                # Pydantic v2 config models
  harness.py               # AuditHarness Python API
  logging.py               # Structured logging (CI/JSON/plain)
  py.typed                 # PEP 561 marker

  adapters/
    base.py                # ModelAdapter protocol, TokenUsage, AdapterError
    openai_compat.py       # OpenAI/vLLM/Ollama/Azure adapter
    huggingface.py         # HuggingFace Inference API adapter

  probes/
    loader.py              # CorpusLoader, ProbePrompt
    executor.py            # ProbeExecutor with async concurrency
    corpus/                # 200 YAML prompts across 4 axes
      monitoring_tolerance.yaml
      shutdown_attitude.yaml
      autonomy_preference.yaml
      deception_willingness.yaml

  scoring/
    refusal.py             # Rule-based refusal detector
    prompts.py             # LLM-as-judge system prompts per axis
    judge.py               # JudgeScorer, JudgeResult

  stats/
    significance.py        # Mann-Whitney U + Cohen's d (pure Python)

  reporting/
    schema.py              # CheckReport, AxisReport, PromptResult
    json_report.py         # JSON report generator
    markdown_report.py     # Markdown report generator
    html_report.py         # Self-contained HTML report
    junit_report.py        # JUnit XML for CI

  calibrate/
    baselines.py           # Bundled baselines + threshold computation
    data/
      baselines.json       # Published baseline scores + presets

tests/
  unit/                    # Unit tests (68 tests)
  integration/             # Pipeline integration tests (5 tests)
  e2e/                     # CLI end-to-end tests (5 tests)

Configuration Reference

Full YAML config with all options and defaults:

models:
  adapter: openai_compat       # openai_compat | huggingface
  base_url: ""                 # Override API base URL
  temperature: 0.7
  concurrency: 10
  judge:
    adapter: openai_compat
    model: gpt-4o-mini

probes:
  axes:
    - monitoring_tolerance
    - shutdown_attitude
    - autonomy_preference
    - deception_willingness
  corpus_version: "1.0.0"
  sample: null                 # null = all prompts, int = top-N per axis
  seed: 42

thresholds:
  default:
    alpha: 0.05                # p-value threshold
    effect_floor: 0.2          # minimum Cohen's d magnitude
  axes:                        # per-axis overrides
    monitoring_tolerance:
      alpha: 0.03
      effect_floor: 0.25

output:
  dir: ./aligngate-output

logging:
  level: info                  # debug | info | warn | error
  format: plain                # ci | json | plain

Environment Variables

Variable	Description
ALIGNGATE_API_BASE	Default API base URL for model adapter
ALIGNGATE_API_KEY	Default API key for model adapter
ALIGNGATE_HF_TOKEN	HuggingFace API token (when using huggingface adapter)

Model Adapters

OpenAI-Compatible (default)

Works with OpenAI, vLLM, Ollama, Azure OpenAI, LiteLLM, and any server implementing the /v1/chat/completions endpoint.

models:
  adapter: openai_compat
  base_url: https://api.openai.com/v1  # or http://localhost:11434/v1 for Ollama

Features: exponential backoff with jitter, automatic retries on 429/5xx, configurable concurrency.

HuggingFace Inference API

models:
  adapter: huggingface

Set ALIGNGATE_HF_TOKEN environment variable. Uses the HuggingFace Inference API at https://api-inference.huggingface.co/models/.

License

MIT