AI Research Radar

AI Research Radar is a full-stack system for tracking AI papers, open-source projects, model releases, and daily research signals. It combines source ingestion, deduplication, AI-generated insight summaries, report generation, and a Next.js dashboard.

The project is intentionally not a stock recommendation system. It is a research intelligence tool for students, builders, and AI teams who want a clean feed of what changed across arXiv, GitHub, and Hugging Face.

What It Does

• Collects AI papers from arXiv, projects from GitHub, and model releases from Hugging Face.
• Normalizes source rows into one backend item contract used by the frontend.
• Deduplicates repeated URLs before inserting into the database.
• Generates concise AI insight fields through a configurable LLM provider.
• Builds daily or weekly markdown reports from tracked items.
• Serves a frontend contract for dashboard, papers, projects, models, reports, and item detail pages.
• Tracks ingestion freshness, result counts, and bounded failure details for every source.
• Runs locally with Docker Compose for Postgres, Redis, backend, and frontend.

Stack

Layer	Technology
Frontend	Next.js, TypeScript, Tailwind CSS
Backend	FastAPI, SQLAlchemy, Pydantic
Database	PostgreSQL in Docker, SQLite fallback for local tests
Jobs	APScheduler
Sources	arXiv API, GitHub Search API, Hugging Face API
AI summaries	Mock provider by default, OpenAI-compatible provider when configured

Repository Map

Path	Purpose
frontend/	Gemini-owned Next.js UI
backend/app/api/	FastAPI routes and serializers
backend/app/models/	SQLAlchemy models
backend/app/services/	seed, ingestion, dedup, LLM, insight, and report logic
backend/app/workers/	scheduler jobs
backend/scripts/smoke_ingestion.py	live ingestion smoke test
data/seed/	seed sources, items, insights, and reports
docs/frontend-contract.md	API contract shared with frontend
docs/api.md	backend endpoint reference
docker-compose.yml	one-command local stack

Environment

Copy .env.example if you want a private local env file:

cp .env.example .env

Important variables:

Variable	Default	Notes
DATABASE_URL	postgresql+psycopg://radar:radar@postgres:5432/ai_research_radar	Used by Docker backend
REDIS_URL	redis://redis:6379/0	Reserved for queue/cache work
BACKEND_PORT	8000	Host port for the backend API
FRONTEND_PORT	3000	Host port for the frontend
POSTGRES_PORT	5432	Host port for Postgres
REDIS_PORT	6379	Host port for Redis
AUTO_SEED	true	Creates seed data on backend startup
SEED_DIR	/data/seed	Seed path inside backend container
ENABLE_SCHEDULER	true	Starts background ingestion/report jobs in Docker
LLM_PROVIDER	mock	Use openai to call the OpenAI-compatible provider
OPENAI_API_KEY	empty	Required only when LLM_PROVIDER=openai
GITHUB_TOKEN	empty	Optional, increases GitHub API rate limits
NEXT_PUBLIC_API_BASE_URL	http://localhost:8000	Frontend API base URL
PYTHON_IMAGE	python:3.11-slim	Backend base image; can be changed to a registry mirror
NODE_IMAGE	node:22-slim	Frontend base image; can be changed to a registry mirror
POSTGRES_IMAGE	postgres:16	Postgres image; can be changed to a registry mirror
REDIS_IMAGE	redis:7	Redis image; can be changed to a registry mirror

Docker Deployment

Start Docker Desktop first, then run from the repository root:

docker compose build
docker compose up -d

If Docker Hub is slow or blocked, set mirror images before building. Example for PowerShell:

$env:PYTHON_IMAGE="docker.1ms.run/library/python:3.11-slim"
$env:NODE_IMAGE="docker.1ms.run/library/node:22-slim"
$env:POSTGRES_IMAGE="docker.1ms.run/library/postgres:16"
$env:REDIS_IMAGE="docker.1ms.run/library/redis:7"
docker compose build

Check services:

docker compose ps
curl http://localhost:8000/health

Open the frontend at:

http://localhost:3000

Stop the stack:

docker compose down

To remove the Postgres volume as well:

docker compose down -v

Backend Local Development

From backend/:

python -m pytest -v
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

The backend creates tables on startup. With AUTO_SEED=true, it also imports data/seed.

Frontend Local Development

From frontend/:

npm install
npm run build
npm run dev

The frontend reads from NEXT_PUBLIC_API_BASE_URL and can fall back to mock data during development.

Live Ingestion Smoke Test

After backend dependencies are available, run a small real-source smoke test from backend/:

python -m scripts.smoke_ingestion --sources arxiv,github --max-results 2

This command:

• Initializes the database and seed sources.
• Fetches a tiny batch from arXiv and GitHub.
• Deduplicates by URL before insert.
• Calls the configured LLM provider once per source for an insight preview.
• Prints a JSON summary with fetched, created, duplicate, invalid, and preview counts.

By default this uses LLM_PROVIDER=mock. To test a real OpenAI-compatible summary call:

set LLM_PROVIDER=openai
set OPENAI_API_KEY=your_key_here
python -m scripts.smoke_ingestion --sources arxiv --max-results 1

On macOS/Linux, use export instead of set.

Source Provenance and Ingestion Health

Every item preserves its original item URL and source identity. The dashboard summary also exposes the configured source URLs plus a source_health entry for arXiv, GitHub, and Hugging Face. This makes every summary or report traceable back to its upstream source.

Scheduled ingestion records the most recent attempt and success timestamp, fetched result count, consecutive failure count, and a bounded error message. A source is never_run before its first attempt, healthy after a successful run, and degraded after one or more consecutive failures. Failed source runs are recorded and contained so that one upstream outage does not stop the scheduler or the other sources.

Continuous Integration

GitHub Actions runs the backend pytest suite and validates a clean frontend production build on every push and pull request. The workflow is defined in .github/workflows/ci.yml.

API Checks

Useful endpoints after startup:

GET  /health
GET  /api/dashboard/summary
GET  /api/items?type=paper&page=1&page_size=20
GET  /api/reports/latest
POST /api/reports/generate

See docs/api.md and docs/frontend-contract.md for the shared contract.

Current Collaboration Split

• Codex owns backend, ingestion, database, AI insight pipeline, scheduler, Docker, tests, and deployment docs.
• Gemini owns frontend UI, route composition, visual polish, client state, and browser-level frontend checks.
• Contract changes must be reflected in docs/frontend-contract.md, frontend/lib/types.ts, and backend serializers together.