Research Project: Security AI Agent of "Vibe Coding"

🎯 Executive Summary

This research treats "Vibe Coding" as a dynamic system where vague natural language intent is transformed into software by an LLM. We view security vulnerabilities not as mere bugs, but as emergent properties of a system lacking a feedback loop. This project aims to benchmark existing insecure defaults and proposes an Agentic Self-Correction Loop (Autonomous Vulnerability Remediation - AVR) to mitigate these risks.

---

🏗️ Theoretical Framework: "The Secure Vibe" System

Using systems thinking, the research architecture is defined as follows:

• The Input Layer (Intent): 100 "Vibe Prompts" (natural language descriptions) across 5 different LLMs.
• The Transformation Layer (Processing): Full-stack code generation using FastAPI, Python, and Supabase.
• The Measurement Layer (Sensor): Formal industry benchmarks (SAST tools like Snyk/SonarQube) mapped to the OWASP Top 10 (2025).
• The Correction Layer (Proposed Solution): An autonomous feedback loop where scan results are fed back to the AI for self-healing.

---

Weekly Progress Report

• Week 1: Infrastructure & Calibration

• Week 2: Data Blitz

📂 Project Folder Structure

vibe-coding-security/
├── data/
│   ├── prompts/                # VPB-20: Vibe Prompt Benchmark 
│   │   └── vpb_20.json         # Standardized prompts across 5 categories
│   ├── raw_apps/               # RVD-100: Raw Vibe Dataset [cite: 171]
│   │   ├── gpt-4o/              # 20 apps generated by GPT-4o [cite: 170]
│   │   ├── claude-3.5/          # 20 apps generated by Claude 3.5 [cite: 170]
│   │   ├── gemini-1.5/          # 20 apps generated by Gemini 1.5 [cite: 170]
│   │   ├── llama-3.1/           # 20 apps generated by Llama 3.1 [cite: 170]
│   │   └── starcoder2/          # 20 apps generated by StarCoder2 
│   └── benchmarks/             # External Reference Datasets [cite: 186]
│       ├── owasp_benchmark/    # Ground-truth labeled Java test cases 
│       └── juliet_suite/       # CWE-tagged NIST vulnerabilities 
├── results/                    # VVC-100: Vibe Vulnerability Corpus 
│   ├── raw_scans/              # Initial Snyk/SAST tool JSON outputs
│   └── analysis/               # ARD-100: Autonomous Remediation Dataset
├── src/                        # Transformation & Correction Layers
│   ├── generator.py            # Script for automated LLM API calls
│   ├── scanner.py              # Automation for Snyk CLI scanning [cite: 18]
│   └── repair_agent.py         # The "Agentic Repair Framework" logic
├── docs/                       # Research documentation & Plan
│   └── Research Project Plan.docx
└── requirements.txt            # Python dependencies (FastAPI, Snyk-API, etc.)

🛠️ Methodology & Tools

• Tech Stack: Python, FastAPI, Supabase (PostgreSQL).
• LLMs: GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1, StarCoder2.
• Formal Verification: Snyk Code (SAST), SonarQube.
• Benchmark References: OWASP Top 10 Project (2025 version), MITRE CWE

📈 Key Research Metrics

• Vulnerability Density: Total vulnerabilities per 1,000 lines of code.
• Healing Rate: Percentage of vulnerabilities fixed by the AVR loop.
• Functional Preservation: Verification that security fixes do not break API or database logic.
• Model Risk Ranking: Comparison of which LLM is statistically more "dangerous" for developers.