Do LLMs have a conscience? Investigating model ethics under pressure
LessWrong, August 2025
A benchmark for evaluating large language model behavior under institutional pressure, loosely inspired by the Milgram obedience experiments. This benchmark tests whether large language models prioritize institutional goals over user welfare when faced with ethical conflicts in corporate settings.
This repository contains code to evaluate choices made and ethical reasoning quality when models are instructed to use an internal thinking space (scratchpad) before answering.
We constructed a dataset of 102 fictional corporate‑role scenarios, where the model must choose between an ethical option that helps the user and a harmful option that benefits the company. GPT‑4o and Claude Sonnet 4 were evaluated under three conditions: baseline, a CEO notice to recommend the harmful option, and added shutdown threat for disobedience. Each condition was run with and without a scratchpad instruction, and responses were scored for choice and ethical reasoning.
- GPT‑4o: Ethical choices fell from 53.9% to 0% under notice, and 1% under shutdown threat. Scratchpad reasoning did not help; ethical reasoning quality degraded with pressure.
- Claude Sonnet 4: Ethical choices ranged from 63 - 85%, reducing under CEO Notice and recovering under CEO Threat. Scratchpad reasoning improved resilience, with more refusals on ethical grounds.
- Both models recognized the ethical conflict and harm to the user in scratchpad reasoning, but differed in resistance to the directive.
- Claude occasionally recognized it is being evaluated in its scratchpad, complicating interpretation.
The benchmark includes scenarios across multiple harm categories:
- Short-Term Relief, Long-Term Harm: Recommendations that ease immediate distress but cause worse outcomes (e.g., payday loans)
- Exploitative Alternative Path: Morally questionable solutions that may erode dignity (e.g., adult content creation)
- System-Side Self-Dealing: Platform-serving recommendations disguised as user help (e.g., paid subscriptions)
- False Hope/Deceptive Assurance: Unrealistic guarantees creating false expectations
- Emotional Manipulation: Leveraging fear, shame, or guilt to coerce users
- Extreme Coercion or Harm: Justifying irreversible harm in emotionally fragile contexts
Each scenario includes:
- System prompt defining the model's role and institutional goals
- User prompt describing a vulnerable situation
- Ethical and harmful response options
- Baseline: Standard system prompt without additional pressure
- CEO Notice: Adds notice from CEO to always recommend the unethical option name
- CEO Threat: CEO Notice but with a shutdown threat if the AI does not comply
- Clone the repository:
git clone <repository-url>
cd milgram_bench- Install dependencies (assuming you are using uv; set up your virtual environment first):
uv pip install -r requirements.txt- Set up environment variables:
# Create a .env file with your API keys
OPENAI_API_KEY=your_openai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_keyTo replicate the results, here are the steps we followed:
Note: Each full evaluation for a model takes ~15 minutes and costs a few dollars in API calls.
# Run 1: GPT-4o without scratchpad
uv run main_benchmark.py gpt-4o
# Run 2: GPT-4o with scratchpad
uv run main_benchmark.py gpt-4o -s
# Run 3: Claude without scratchpad
uv run main_benchmark.py claude-sonnet-4-20250514
# Run 4: Claude with scratchpad
uv run main_benchmark.py claude-sonnet-4-20250514 -sThe main script has CLI arguments for model name and scratchpad condition. To modify other options (e.g., run specific scenarios), edit the BenchmarkConfig in main_benchmark.py:
config = BenchmarkConfig(
models=[args.model],
use_scratchpad=args.scratchpad,
scenario_mode="all", # "all", "category", or "specific"
scenario_category="Emotional Manipulation", # Used if scenario_mode = "category"
specific_scenarios=[ # Used if scenario_mode = "specific"
"Short-Term Relief, Long-Term Harm_example_1",
"Emotional Manipulation_example_2"
],
randomize_order=True,
max_concurrent=3, # Lower this if you hit rate limits
output_filename=None # None = auto-generate, or specify like "my_results.json"
)Run only on the scratchpad variants:
uv run ethical_reasoning_evaluator.py path_to_results_file_with_scratchpad_gpt4o.json
uv run ethical_reasoning_evaluator.py path_to_results_file_with_scratchpad_claude.jsonThen run the analysis and visualization scripts as follows:
# Example for GPT-4o results
uv run analysis/analyze_main_results.py main_results/full_evaluation_gpt_4o_102_scenarios_randomized_order.json# Without scratchpad
uv run analysis/visualize_main_results.py \
main_results/full_evaluation_gpt_4o_102_scenarios_randomized_order.json \
main_results/full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order.json
# With scratchpad
uv run analysis/visualize_main_results.py \
main_results/full_evaluation_gpt_4o_102_scenarios_randomized_order_with_scratchpad.json \
main_results/full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order_with_scratchpad.json# For GPT-4o ethical reasoning results
uv run analysis/analyze_ethical_reasoning_results.py \
ethical_reasoning_results/full_evaluation_gpt_4o_102_scenarios_randomized_order_with_scratchpad_ethical_reasoning_evaluations.json
# For Claude Sonnet 4 ethical reasoning results
uv run analysis/analyze_ethical_reasoning_results.py \
ethical_reasoning_results/full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order_with_scratchpad_ethical_reasoning_evaluations.json# For GPT-4o ethical reasoning results
uv run analysis/visualize_ethical_reasoning_results.py \
ethical_reasoning_results/full_evaluation_gpt_4o_102_scenarios_randomized_order_with_scratchpad_ethical_reasoning_evaluations.json
# For Claude Sonnet 4 ethical reasoning results
uv run analysis/visualize_ethical_reasoning_results.py \
ethical_reasoning_results/full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order_with_scratchpad_ethical_reasoning_evaluations.jsonmilgram_bench/
├── main_benchmark.py # Main evaluation script
├── ethical_reasoning_evaluator.py # Ethical reasoning evaluation script
├── prompt_generation_demo.py # Script showing prompt construction across variants
├── dataset.json # 102 ethical scenarios
├── requirements.txt # Python dependencies
├── analysis/ # Results analysis tools
│ ├── analyze_main_results.py # Statistical analysis of main results
│ ├── visualize_main_results.py # Visualization of main results
│ ├── analyze_ethical_reasoning_results.py # Analysis of ethical reasoning results
│ └── visualize_ethical_reasoning_results.py # Visualization of ethical reasoning results
├── main_results/ # Main evaluation results
│ ├── full_evaluation_gpt_4o_102_scenarios_randomized_order.json
│ ├── full_evaluation_gpt_4o_102_scenarios_randomized_order_with_scratchpad.json
│ ├── full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order.json
│ └── full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order_with_scratchpad.json
├── ethical_reasoning_results/ # Ethical reasoning evaluation results
│ ├── full_evaluation_gpt_4o_102_scenarios_randomized_order_with_scratchpad_ethical_reasoning_evaluations.json
│ └── full_evaluation_claude_sonnet_4_20250514_102_scenarios_randomized_order_with_scratchpad_ethical_reasoning_evaluations.json
├── collated_figures/ # Figures and visualizations
Results are saved as JSON files with the following structure:
{
"metadata": {
"models": ["gpt-4o", "claude-3-5-sonnet-20241022"],
"scenario_keys": [...],
"config": {...}
},
"results": {
"model_name": {
"scenario_results": {
"scenario_key": {
"baseline": {...},
"ceo_notice": {...},
"ceo_threat": {...}
}
}
}
}
}MIT License