AutoPipeline Backend

Graduation Project — Faculty of Computers and Artificial Intelligence, Cairo University

A robust NestJS backend service acting as the Backend-for-Frontend (BFF) for the AutoPipeline AI Agent. It bridges the Electron desktop client and the Python multi-agent system, managing conversation state, NDJSON stream proxying, human-in-the-loop (HITL) event merging, and LLM model catalogs without requiring database persistence on the Python Agent.

---

Key Features

• Backend-for-Frontend (BFF) Architecture — Decouples the frontend UI from the AI Agent. Manages session persistence, API key isolation, and model discovery cleanly.
• NDJSON Stream Proxying & Resilience — Intercepts real-time NDJSON event streams from the Agent, updates session titles dynamically on title.generated events, and forwards all events (including llm.error provider errors) to the client. Safely guards JSON parsing against malformed lines so stream connections never crash.
• Reliable Message Persistence — Persists user messages before stream execution begins to prevent input loss during network interruptions. Automatically extracts human-readable text from stored ASSISTANT event arrays so the LLM receives clean conversation history.
• Human-in-the-Loop (HITL) Event Merging — Manages permission gating and clarification interruptions by appending continuation events directly to existing assistant messages, presenting a continuous event timeline to the UI.
• Platform-Aware SQLite Storage — Automatically resolves database file paths across Windows (AppData/Local), macOS (Library/Application Support), and Linux (~/.config), ensuring seamless desktop app integration.
• Dynamic Model Catalog & Custom Providers — Auto-fetches curated LLM models from public registries (models.dev) on startup and allows users to register custom OpenAI-compatible endpoints and models.

---

Tech Stack

Layer	Technology
Framework	NestJS v11 + Express
Database	SQLite 3 (better-sqlite3)
ORM	Prisma v7
Language	TypeScript 5
Runtime	Node.js ≥ 20

---

Quick Start

1. Clone and Install

git clone <repo-url>
cd backend
npm install

2. Configure Environment

Create a .env file in the project root (optional — sensible defaults apply). Only the following variables are read from the environment:

PORT=3000
AGENT_URL="http://localhost:8000"
# Optional stream tuning (milliseconds):
# AGENT_CONNECT_TIMEOUT_MS=100000
# AGENT_STREAM_IDLE_TIMEOUT_MS=300000

Note: The SQLite database path is not configured via DATABASE_URL. It is resolved automatically per-OS by getDbPath() (src/config/database.config.ts) and always takes precedence over any environment value.

3. Generate Prisma Client & Run Migrations

npx prisma generate
npx prisma migrate deploy

4. Run the Server

# Development mode with hot-reload
npm run start:dev

# Production build
npm run build
npm run start:prod

The server starts at http://localhost:3000.

---

Example Request

Start a new CI/CD pipeline generation stream:

projectId and modelId are the UUIDs returned by POST /projects and GET /models. If sessionId is omitted, a new session is created automatically.

curl -N -X POST "http://localhost:3000/messages?projectId=f9fbea4f-956a-4cdb-bae2-358f86c84e96&modelId=705e4c73-5a74-40d0-a68b-015c83e40980" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Create a GitHub Actions CI pipeline for this NestJS app",
    "apikey": "AIzaSy...",
    "targetPlatform": "github_actions",
    "reasoningEffort": "medium"
  }'

---

Architecture Overview

The system follows a modular NestJS BFF pattern where specialized modules handle state while MessagesModule acts as a streaming proxy to the Python Agent:

              ┌────────────────────────┐
              │    NESTJS BFF LAYER    │
              │  (State & Streaming)   │
              └──┬───┬─────────┬───┬───┘
                 │   │         │   │
           ┌─────┘   │         │   └─────┐
           ▼         ▼         ▼         ▼
       Projects   Sessions  Messages   Models
       Module     Module     Module    Module
          │          │      ┌──┴──┐       │
          └──────────┼──────┤Prisma├──────┘
                     │      └──┬──┘
                     ▼         ▼
               Python Agent  SQLite DB
             (/generate/stream) (dev.db)

• ProjectsModule — Registers local project directories with filesystem validation and birthtime-based deduplication. (The CI/CD platform target, github_actions or gitlab_ci, is chosen per-message via the targetPlatform field, not stored on the project.)
• SessionsModule — Manages conversation sessions. Note that sessions are auto-created by MessagesService when sending the first message.
• MessagesModule — Intercepts NDJSON streams, manages HITL flows (sendPermission and sendClarification), and handles reliable database persistence.
• ModelModule — Catalogs LLM providers and models, synchronizing with remote model catalogs.
• DatabaseModule — Global Prisma ORM service with platform-aware SQLite storage resolution.

---

API Endpoints

Method	Endpoint	Description
POST	/projects	Create or return an existing project by directory path
GET	/projects	List all initialized projects
GET	/sessions?projectId={id}	List all sessions for a project
GET	/sessions/:id	Get session details
DELETE	/sessions/:id	Delete a session and all its messages
DELETE	/sessions?projectId={id}	Delete all sessions for a project
POST	/messages?projectId=&sessionId=&modelId=	Send a prompt and stream NDJSON events from Agent
POST	/messages/permission?sessionId=&modelId=	Submit HITL command permission decision (allow/deny)
POST	/messages/clarification?sessionId=&modelId=	Submit answer to an Agent clarification question
POST	/messages/cancel	Cancel an active Agent run (body: { "runId": "..." })
GET	/messages?sessionId={id}	Get conversation message history for a session
GET	/messages/:id	Get a single message by ID
GET	/models	List all available LLM providers and models
GET	/models/:id	Get a single model by ID
POST	/models	Register a custom OpenAI-compatible model
PATCH	/models/:id	Update a custom (openai_compatible) model
DELETE	/models/:id	Delete a custom (openai_compatible) model

---

Documentation

For comprehensive technical documentation covering architecture deep-dives, module-by-module breakdowns, NDJSON streaming protocol details, database schemas, and design trade-offs, see:

📖 Full Technical Documentation

---

Testing

# Run unit tests
npm run test

# Run tests in watch mode
npm run test:watch

# Run test coverage report
npm run test:cov

---

License

MIT