AutoPipeline Agent

Graduation Project — Faculty of Computers and Artificial Intelligence, Cairo University

An AI-powered multi-agent system that automatically generates production-ready CI/CD workflow files (GitHub Actions & GitLab CI) for any software project. Point it at a codebase, describe what you need, and the agent analyzes your project, plans a pipeline, generates valid YAML, validates it with real linters, and self-corrects — all streamed in real-time.

---

Key Features

• Multi-Agent Architecture — Specialized agents (Planner, Analyzer, Researcher, Generator, Validator, Writer) orchestrated by a deterministic Supervisor hub using LangGraph.
• Project-Aware — The Analyzer agent inspects your actual codebase (build files, test runners, deployment configs) so the generated pipeline is grounded in reality, not guesses.
• Web Research Capability — A dedicated ReAct Researcher agent uses DuckDuckGo to search for external documentation, syntax rules, and the latest CI/CD action versions.
• Self-Correcting Reflexion Loop — Generated YAML is validated with actionlint (GitHub Actions) or the GitLab CI Lint API. If validation fails, the Generator receives the errors and retries automatically.
• Multi-Platform — Supports both GitHub Actions and GitLab CI workflow generation.
• Multi-Provider LLM Support — Powered by a LiteLLM factory supporting Groq, OpenAI, Google Gemini, Anthropic, and any OpenAI-compatible endpoint (vLLM, Ollama, LM Studio, OpenRouter). The provider, model, and key are supplied per-request, with two-layer resilience (silent retry + Tenacity exponential backoff).
• Real-Time Streaming — Progress is streamed as NDJSON events, giving full visibility into what the agent is doing at each step, including live reasoning/thinking tokens.
• Human-in-the-Loop & Two-Phase Write Gate — The agent pauses to ask for clarification, request permission for shell commands, and obtain explicit approval before saving files to disk.
• Intelligent Compaction & Summarization — Long conversations can be compacted on-demand to preserve context limits, and completed runs generate structured markdown summaries of architectural decisions.
• Security Built-In — Shell commands are classified (safe/modify/dangerous) with permission gating, and all streamed payloads are sanitized for secrets using entropy-based detection while preserving public identifiers.
• Multi-Turn Conversations — Supports follow-up requests with full conversation history.

---

Tech Stack

Layer	Technology
Agent Framework	LangGraph + LangChain
LLM Interface & Resilience	LiteLLM + Tenacity (Two-Layer Retry Policy)
LLM Providers	Groq, OpenAI, Google Gemini, Anthropic
API Server	FastAPI + Uvicorn
Validation	actionlint (GitHub Actions), GitLab CI Lint API
Web Research	DuckDuckGo (DDGS), BeautifulSoup4
Data Models	Pydantic v2
Secret Detection	detect-secrets (Shannon Entropy Analysis)
Language	Python ≥ 3.11

---

Quick Start

# 1. Clone and install
git clone <repo-url>
cd agent
pip install -e .

# 2. (Optional) Configure env-only extras
# LLM keys are NOT needed here — the provider/model/key are sent per-request
# in the `llm` object. .env is only for optional GitLab-lint / LangSmith vars.
cp .env-example .env

# 3. Run the server
python -m agent
# Server starts at http://localhost:8000
# Pass your LLM provider + key in each request's `llm` field (see below).

Example Request

curl -N -X POST http://localhost:8000/generate/stream \
  -H "Content-Type: application/json" \
  -d '{
    "project_path": "/path/to/your/project",
    "prompt": "Create a CI pipeline for this project",
    "target_platform": "github_actions",
    "thread_id": "550e8400-e29b-41d4-a716-446655440000",
    "history": [],
    "title": "",
    "llm": {
      "provider": "groq",
      "model": "llama-3.3-70b-versatile",
      "api_key": "gsk_..."
    }
  }'

Note: The llm object is required on every request — the server holds no default provider/model/key. Supported providers: groq, openai, anthropic, gemini, and openai_compatible (local vLLM/Ollama/LM Studio, etc.).

---

Architecture Overview

The system follows a hub-and-spoke pattern where a deterministic Supervisor routes between specialized agents:

                  ┌────────────────────────┐
                  │   SUPERVISOR (hub)     │
                  │  Pure Python routing   │
                  └──┬───┬───┬───┬───┬───┬─┘
                     │   │   │   │   │   │
           ┌─────────┘   │   │   │   │   └─────────┐
           ▼             ▼   ▼   ▼   ▼             ▼
        Planner   Analyzer  Researcher  Generator HITL   Writer
         (LLM)    (ReAct)    (ReAct)    ┌──────┐ Gateway (file)
                                        │Gen   │
                                        │  ↕   │
                                        │Valid │
                                        └──────┘

1. Supervisor — Deterministic router (zero LLM calls). Reads state flags and picks the next agent.
2. Planner — Creates a high-level pipeline plan based on project analysis and web research.
3. Analyzer — ReAct agent that inspects the local codebase using tools (bash, glob, grep, read, list).
4. Researcher — ReAct agent dedicated to external web research using DuckDuckGo (websearch, webfetch).
5. Generator — Produces YAML from the plan, with a reflexion loop for self-correction.
6. Validator — Lints generated YAML with real tools (actionlint / GitLab API).
7. Human Interaction — Centralized gateway for clarifications, command permissions, and write approvals.
8. Writer — Saves the validated YAML to disk upon explicit user approval, then triggers an automated run summary.

---

API Endpoints

Method	Endpoint	Description
GET	/health	Service health + version
POST	/generate/stream	Start a new CI/CD generation (NDJSON stream)
POST	/generate/{thread_id}/resume/permission	Resume after a command or write permission pause
POST	/generate/{thread_id}/resume/clarification	Resume after a clarification pause
POST	/compact	Compress conversation history on-demand
POST	/generate/runs/{run_id}/cancel	Cancel an active generation

---

Documentation

For comprehensive technical documentation covering architecture deep-dives, package breakdowns, design decisions, security details, and complete API reference, see:

📖 Full Documentation

---

Testing

# All tests
pytest

# Unit tests only (no LLM keys needed)
pytest tests/unit/

# Integration tests (requires LLM keys)
pytest tests/integration/

---

License

MIT