Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions submissions/Saradwanth/Dockerfile.backend
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM python:3.11-slim
WORKDIR /app
COPY backend/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY backend/ .
# Ensure local data directory exists for Chroma and SQLite
RUN mkdir -p data
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]
9 changes: 9 additions & 0 deletions submissions/Saradwanth/Dockerfile.frontend
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build
RUN npm install -g serve
EXPOSE 5173
CMD ["serve", "-s", "dist", "-l", "5173"]
28 changes: 28 additions & 0 deletions submissions/Saradwanth/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# MUTAV2 (Mutagent) - Hackathon Submission

## Overview
MUTAV2 is an Agentic Development Lifecycle (ADL) orchestration platform designed to understand, traverse, and automatically optimize its own behavior against any given GitHub repository.

Instead of bolting a chatbot on top of code, we built:
1. **Hybrid Retrieval:** Semantic indexing via local embeddings (BAAI/bge-base-en-v1.5) combined with a highly accurate tree-sitter AST dependency graph.
2. **Mutagent Optimization Engine:** A self-evolving loop that evaluates LLM prompts against ground-truth datasets using severity-gated rubrics, and runs genetic mutations on failed prompts to discover optimal configurations.
3. **Enterprise B2B Layer:** Blast radius evaluation feeds directly into Test Selection and Reviewer Routing algorithms. A Maintainer Health dashboard automatically scores issue validity.

## How to Run
We have included a packaging script (`scripts/ship.py`) that generated the Dockerfiles and `docker-compose.yml` for this project.

1. Ensure Docker and Docker Compose are installed.
2. Run the following command from the root of this submission folder:
```bash
docker-compose up --build
```
3. Navigate to `http://localhost:5173` to view the UI.

## Evaluation Results (Mutagent Reports)
We don't hide behind fake metrics. The system is designed to trace everything and report honest deltas.
- You can view our execution traces under the `traces/` directory.
- The `reports/` directory in our backend shows the true deltas of our genetic mutation runs. We explicitly include scenarios where the model failed to produce a parseable variant, demonstrating the real rigor of our evaluation loop.

## Judging Artifacts
- **Transcripts:** The `transcripts/` directory contains the complete, unedited `.jsonl` trace of the IDE agent session used to build and package this project.
- **Traces:** The `traces/` directory contains all raw prompt and latency traces generated by the Mutagent harness.
17 changes: 17 additions & 0 deletions submissions/Saradwanth/agentspec.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
name: "MUTAV2 (Mutagent)"
team: "Saradwanth"
description: >
An Agentic Development Lifecycle (ADL) orchestration platform featuring
a semantic + structural (tree-sitter) hybrid retriever, an automated
genetic-mutation prompt optimizer, and a B2B enterprise layer.
tags:
- agentic-framework
- retrieval-augmented-generation
- ast-parsing
- evaluation-loop
architecture:
- "FastAPI (Python) Backend"
- "React (TypeScript) Frontend"
- "Local BAAI/bge-base-en-v1.5 embeddings"
- "Tree-sitter AST Graph Extraction"
- "Mutagent Optimizer Loop"
41 changes: 41 additions & 0 deletions submissions/Saradwanth/backend/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Copy this file to ".env" and fill in your own values.
# NEVER commit the real ".env" file to git — it holds your secret keys.

# Get this from https://console.groq.com/keys — used for every LLM call except
# rag_qa.py's HyDE/answer synthesis when OLLAMA_BASE_URL below is set.
GROQ_API_KEY=gsk_your-key-here

# Get this from https://github.com/settings/tokens (classic token, "repo" + "read:org" scopes are enough)
# A token is optional for public repos but you'll hit rate limits fast without one.
GITHUB_TOKEN=ghp_your-token-here

# Where the local vector database is stored on disk (created automatically)
CHROMA_PERSIST_DIR=./data/chroma

# Which models to use. EMBEDDING_MODEL is a local sentence-transformers model —
# no API key, no quota, no rate limit — downloaded once on first use and cached.
EMBEDDING_MODEL=BAAI/bge-base-en-v1.5
LLM_MODEL=llama-3.1-8b-instant

# Optional: route rag_qa.py (HyDE + RAG answer synthesis only — nothing else in
# this app reads these) through a local/remote Ollama instance instead of Groq.
# Leave OLLAMA_BASE_URL blank to use Groq (LLM_MODEL above) everywhere, which is
# also what keeps every component on the same model — see B2B_AUDIT.md item 4
# for why that divergence matters if you do set this.
# Must include the /v1 suffix (Ollama's OpenAI-compatible endpoint), e.g.
# http://localhost:11434/v1 or an ngrok URL like https://xxxx.ngrok-free.dev/v1 —
# the ngrok-skip-browser-warning header is sent automatically for tunnel URLs.
OLLAMA_BASE_URL=
OLLAMA_MODEL=qwen2.5:3b

# How many chunks to retrieve per question in RAG Q&A
TOP_K=5

# HyDE: draft a hypothetical answer with the LLM and embed that for vector search,
# instead of embedding the raw (often short/vague) question. Set to "false" to disable
# and fall back to embedding the raw question directly.
HYDE_ENABLED=true

# Chunking settings (measured in tokens, not characters)
CHUNK_SIZE=500
CHUNK_OVERLAP=50
114 changes: 114 additions & 0 deletions submissions/Saradwanth/backend/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
# AI Open Source Mentor++

A hackathon MVP that helps developers onboard to unfamiliar GitHub repos:
repo Q&A, issue recommendation, a learning roadmap, and a PR readiness check.

This README assumes you've never set up a Python backend before. Follow it top to bottom.

## Folder structure

```
ai-oss-mentor/
├── .env.example # template for your secret keys — copy this to .env
├── .gitignore
├── requirements.txt # Python packages this project needs
├── config.py # loads .env into one place
├── main.py # the web server — this is what you run
├── indexing.py # Step 1: turns a repo into a searchable knowledge base
├── rag_qa.py # Step 2: answers questions using that knowledge base
├── issue_recommendation.py # recommends a good issue to start on
├── learning_roadmap.py # generates a reading order for the repo
├── pr_readiness.py # checks a PR diff before you submit it
├── github_client.py # talks to GitHub's API
├── embeddings.py # chunks text and turns it into vectors
├── vector_store.py # stores and searches those vectors (Chroma, runs locally)
└── data/
└── chroma/ # the local database gets created here automatically
```

## 1. Install Python

You need Python 3.10 or newer. Check with:
```bash
python3 --version
```
If you don't have it, download from https://www.python.org/downloads/

## 2. Set up a virtual environment

This keeps this project's packages separate from everything else on your machine.

```bash
cd ai-oss-mentor
python3 -m venv .venv

# activate it — do this every time you open a new terminal for this project
source .venv/bin/activate # Mac/Linux
.venv\Scripts\activate # Windows
```

You'll know it worked because your terminal prompt will show `(.venv)` at the start.

## 3. Install the packages

```bash
pip install -r requirements.txt
```

## 4. Set up your API keys

```bash
cp .env.example .env
```

Now open `.env` in any text editor and fill in:

- **OPENAI_API_KEY** — required. Get one at https://platform.openai.com/api-keys
(you'll need to add a small amount of billing credit — a few dollars covers a hackathon)
- **GITHUB_TOKEN** — optional but recommended. Get one at https://github.com/settings/tokens
→ "Generate new token (classic)" → check the `repo` box → generate.
Without this, you can still use public repos but you'll hit GitHub's rate limit quickly.

Leave the other variables as-is unless you know you want to change them.

## 5. Run the server

```bash
uvicorn main:app --reload
```

You should see something like `Uvicorn running on http://127.0.0.1:8000`.

## 6. Try it out

Open **http://127.0.0.1:8000/docs** in your browser. This is an automatic
interactive test page — you can try every endpoint from here without writing
any code.

The order to try things in:

1. **POST /index** — body: `{"repo_url": "https://github.com/some/small-repo"}`
Do this first for any repo. Pick a small public repo for your first test —
indexing a huge repo takes longer and costs more in API calls.
2. **POST /ask** — body: `{"repo_url": "...", "question": "What does this repo do?"}`
3. **GET /recommend-issue** — query param `repo_url`
4. **POST /roadmap** — body: `{"repo_url": "..."}`
5. **POST /pr-check** — body: `{"diff_text": "...paste a git diff here..."}`

## Common problems

- **"OPENAI_API_KEY is not set"** — you forgot step 4, or forgot to save `.env`
- **GitHub rate limit errors** — add a `GITHUB_TOKEN` (step 4)
- **Indexing takes a while / costs API credit** — this is normal; each file
gets split into chunks and each chunk calls the embeddings API. Start with
a small repo (under ~50 files) for your first test.
- **`ModuleNotFoundError`** — make sure your virtual environment is activated
(you should see `(.venv)` in your prompt) and that you ran `pip install -r requirements.txt`

## What's not built yet (see the original plan)

- Repository architecture/dependency visualization
- Similar PR retrieval for PR readiness (the module has a comment showing
exactly where to plug it in)
- The n8n webhook automation for auto-triggering indexing on repo updates
- A frontend — right now everything is tested through `/docs`
13 changes: 13 additions & 0 deletions submissions/Saradwanth/backend/b2b/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
"""backend.b2b — enterprise data model and features layered on the core engine.

Per the B2B Implementation Plan: everything here reuses the existing
parsing/graph/retrieval engine (indexing.py, graph/, rag_qa.py,
issue_recommendation.py, maintainer_health.py) unmodified. This package only
adds the organization/member layer and the enterprise-framed endpoints that
sit on top of it.

Demo-scope persistence (store.py): sqlite3, no auth. org_id/member_id are
passed as plain request params, the same way repo_url already is elsewhere
in this app. Real login/session auth is out of scope for this pass — see
CHANGES.md and B2B_AUDIT.md.
"""
74 changes: 74 additions & 0 deletions submissions/Saradwanth/backend/b2b/governance.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
"""Governance dashboard packaging (plan §3.5).

No new evaluation logic — observability/metrics.py already computes
everything. This only reshapes and labels it as the artifact a
security/compliance reviewer asks for when approving an AI tool for use
against internal code: per-component score, baseline vs. current, and a
last-evaluated timestamp.

Two things it does NOT fabricate, on principle (plan §6 — verify, don't
assert): dataset_version and rubric_version are reported as null because
mutagent/datasets and mutagent/rubrics don't carry version fields yet, and
generation_model reports what's actually configured rather than the
single model the design doc assumes, since rag_qa.py can diverge from the
rest of the app onto Ollama (see B2B_AUDIT.md).
"""
from __future__ import annotations

from datetime import datetime, timezone

from config import REPORTS_DIR, settings
from observability.metrics import get_metrics


def _generation_model_note() -> str:
if settings.OLLAMA_BASE_URL:
return (
f"All generation calls are currently routed through the local Ollama endpoint. "
f"Council gate, issue recommendation, PR check, and issue health use {settings.LLM_MODEL}. "
f"RAG Q&A (HyDE + answer synthesis) uses {settings.OLLAMA_MODEL}. "
f"This is a divergence from the 'one model' constraint, see B2B_AUDIT.md."
)
return f"All generation calls use {settings.LLM_MODEL} via Groq."


def get_governance_report() -> dict:
metrics = get_metrics()
components = []

for t in metrics["targets"]:
report_path = REPORTS_DIR / f"{t['id']}.delta.json"
last_evaluated = None
if report_path.exists():
last_evaluated = datetime.fromtimestamp(
report_path.stat().st_mtime, tz=timezone.utc
).isoformat()

components.append({
"component": t["id"],
"name": t["name"],
"priority": t["priority"],
"evaluated": t["has_report"],
"last_evaluated": last_evaluated,
"baseline_score": t.get("baseline_score"),
"current_score": t.get("optimized_score") if t.get("optimized") else t.get("baseline_score"),
"delta": t.get("delta"),
"mean_f1": t.get("mean_f1"),
"trace_count": t["trace_count"],
"dataset_version": None, # not tracked yet — see B2B_AUDIT.md
"rubric_version": None, # not tracked yet — see B2B_AUDIT.md
})

return {
"generated_at": datetime.now(timezone.utc).isoformat(),
"generation_model_note": _generation_model_note(),
"components": components,
"caveats": [
"dataset_version/rubric_version are not tracked per-report yet — "
"add a version field to mutagent/datasets and mutagent/rubrics "
"before presenting this unmodified to a compliance reviewer.",
"trace_count reflects mutagent/traces/*.jsonl, which is append-only "
"and never rotated — see B2B_AUDIT.md item 1 before treating volume "
"as a retention guarantee.",
],
}
29 changes: 29 additions & 0 deletions submissions/Saradwanth/backend/b2b/roster.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"""Manager view (plan §3.3): team roster, per-member roadmap progress, and
PR-readiness history in one payload — replaces manually chasing status
across three separate queries.
"""
from __future__ import annotations

from b2b import store


def get_roster(org_id: int) -> list[dict]:
roster = []
for member in store.list_members(org_id):
member_id = member["id"]
assigned = store.list_assigned_issues(member_id)
pr_history = store.list_pr_readiness_history(member_id)
roadmap_status = store.get_roadmap_statuses(member_id)

open_count = sum(1 for a in assigned if a["status"] not in ("done", "closed"))
ready_count = sum(1 for h in pr_history if h["verdict"] == "ready")

roster.append({
"member": member,
"assigned_issues": assigned,
"open_assignment_count": open_count,
"roadmap_status": roadmap_status,
"pr_readiness_history": pr_history,
"pr_ready_rate": round(ready_count / len(pr_history), 4) if pr_history else None,
})
return roster
Loading