IMDB Sentiment Analysis & NLP Experiment Lab

Full-stack experiment lab for comparing classical NLP sentiment classifiers on IMDB movie reviews. It provides dataset ingestion, configurable preprocessing, reproducible CountVectorizer vs TF-IDF experiments, live training progress, persisted metrics, model registry promotion, feature analysis, error analysis, and custom sentiment prediction.

GitHub Repository Description

Full-stack IMDB sentiment analysis lab with React, FastAPI, SQLAlchemy, Alembic, scikit-learn, TF-IDF/CountVectorizer experiments, model registry, live predictions, and analytics dashboards.

Suggested topics:

sentiment-analysis, imdb, nlp, machine-learning, scikit-learn, tfidf, countvectorizer, fastapi, react, typescript, sqlalchemy, alembic

Features

• Import and validate IMDB review CSV datasets.
• Version datasets with checksums, class counts, duplicate metadata, and immutable records.
• Browse reviews with server-side pagination, search, and sentiment filtering.
• Preview preprocessing with configurable HTML cleanup, URL removal, casing, punctuation handling, stopword removal, lemmatization flag, and stemming.
• Preserve negation words such as not, no, and never during stopword removal.
• Train reproducible experiments with persisted train/test split assignments.
• Compare COUNT_VECTORIZER and TFIDF using the same dataset version, seed, preprocessing, model, and split.
• Train Logistic Regression, Multinomial Naive Bayes, calibrated Linear SVM, and Random Forest classifiers.
• Persist metrics, confusion matrices, run events, model artifacts, registry entries, predictions, and prediction probabilities.
• Stream real training progress with Server-Sent Events.
• Promote models to production and run custom predictions from the UI.
• Inspect linear model coefficients and misclassified samples.

Architecture

React + TypeScript + Vite frontend
          |
          | REST API + Server-Sent Events
          v
FastAPI backend
          |
          | SQLAlchemy repositories/services
          v
PostgreSQL or SQLite database
          |
          | scikit-learn worker + joblib artifacts
          v
Local artifact storage

PostgreSQL is the intended database for deployment. SQLite is supported for local smoke runs and automated tests.

Tech Stack

Layer	Tools
Frontend	React, TypeScript, Vite, Tailwind CSS, React Router, TanStack Query, React Hook Form, Zod, Recharts, lucide-react
Backend	FastAPI, Pydantic v2, SQLAlchemy 2.x, Alembic, Uvicorn
ML	scikit-learn, NumPy, pandas-compatible CSV ingestion, joblib
Database	PostgreSQL for deployment, SQLite for local fallback
Testing	pytest, ruff, Vitest, TypeScript compiler, ESLint

Project Structure

.
├── backend/
│   ├── app/
│   │   ├── api/routes/          # Versioned FastAPI routes
│   │   ├── core/                # Settings and logging
│   │   ├── db/                  # SQLAlchemy engine/session
│   │   ├── ml/                  # Preprocessing, features, trainers, metrics
│   │   ├── models/              # SQLAlchemy domain models
│   │   ├── schemas/             # Pydantic API contracts
│   │   ├── services/            # Dataset, experiment, prediction, analysis logic
│   │   └── storage/             # Local artifact storage abstraction
│   ├── migrations/              # Alembic migrations
│   ├── tests/                   # Backend tests
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── app/                 # App shell and routing
│   │   ├── components/          # Shared UI components
│   │   ├── features/            # Dashboard, dataset, experiments, prediction, analysis
│   │   ├── services/            # Typed API client
│   │   └── types/               # Shared TypeScript API types
│   └── package.json
├── scripts/
│   ├── download_dataset.py      # Downloads source IMDB CSV
│   └── smoke_flow.py            # End-to-end ML smoke flow
├── docs/
├── data/                        # Local dataset location, CSV ignored by git
├── docker-compose.yml
├── .env.example
└── README.md

Prerequisites

• Python 3.12 or compatible.
• Node.js 20+.
• pnpm.
• Docker Desktop if running PostgreSQL with Docker.

Environment

Copy .env.example to .env and edit values.

Copy-Item .env.example .env

Example PostgreSQL configuration:

DATABASE_URL=postgresql+psycopg://imdb:your-local-password@localhost:5432/imdb_lab
POSTGRES_PASSWORD=your-local-password
BACKEND_HOST=127.0.0.1
BACKEND_PORT=8000
FRONTEND_URL=http://localhost:5173
CORS_ORIGINS=http://localhost:5173,http://127.0.0.1:5173
ARTIFACT_STORAGE_PATH=./artifacts
LOG_LEVEL=INFO

SQLite fallback for local development:

$env:DATABASE_URL='sqlite:///D:/RESOURCES/NLP/IMDB_TFIDF_Analyisis/imdb_lab.db'

PostgreSQL Setup

$env:POSTGRES_PASSWORD='choose-a-local-password'
docker-compose up -d postgres

Then run migrations:

cd backend
$env:DATABASE_URL='postgresql+psycopg://imdb:choose-a-local-password@localhost:5432/imdb_lab'
python -m alembic upgrade head

If Docker reports that the daemon is unavailable, start Docker Desktop first. If docker compose is unavailable on your machine, use docker-compose.

Backend Setup

cd backend
python -m pip install -r requirements.txt
python -m alembic upgrade head
python -m uvicorn app.main:app --reload --host 127.0.0.1 --port 8000

Backend health checks:

• http://127.0.0.1:8000/api/v1/health
• http://127.0.0.1:8000/api/v1/ready
• API docs: http://127.0.0.1:8000/docs

Frontend Setup

cd frontend
pnpm install
pnpm run dev

Open:

http://127.0.0.1:5173

If pnpm blocks esbuild, run:

pnpm approve-builds --all
pnpm rebuild esbuild

Dataset Setup

Dataset source:

https://raw.githubusercontent.com/FarhanaTeli/Sentiment_Analysis_IMDB/main/IMDB%20Dataset.csv

Download:

python scripts/download_dataset.py

The CSV is saved locally as:

data/IMDB Dataset.csv

The CSV is intentionally ignored by git. Use the download script after cloning.

Running Experiments

Use the UI:

1. Open the app.
2. Go to Dataset and import data/IMDB Dataset.csv.
3. Go to New Experiment.
4. Keep dataset, split, preprocessing, classifier, and seed fixed.
5. Run one experiment with COUNT_VECTORIZER.
6. Run another with TFIDF.
7. Compare results in Compare.
8. Promote a model in Predict.
9. Enter a custom movie review and persist a prediction.

CLI smoke flow:

python scripts/smoke_flow.py

Full dataset smoke flow:

$env:SMOKE_SAMPLE_LIMIT=''
$env:SMOKE_MAX_FEATURES='5000'
python scripts/smoke_flow.py

Verified Results

The local full-dataset smoke flow imported 50,000 reviews and produced:

Experiment	Accuracy	F1
CountVectorizer + Logistic Regression	0.8781	0.8781
TF-IDF + Logistic Regression	0.8993	0.8999

The smoke prediction for The film has excellent acting and a moving ending. returned positive.

Testing

Backend:

python -m pytest backend/tests -q
python -m ruff check backend/app backend/tests scripts

Frontend:

cd frontend
pnpm run typecheck
pnpm run lint
pnpm run test
pnpm run build

Verified locally:

• Backend tests: 4 passed
• Backend lint: passed
• Alembic upgrade: passed
• Alembic upgrade, downgrade, upgrade on disposable DB: passed
• Frontend TypeScript: passed
• Frontend lint: passed
• Frontend test: 1 passed
• Frontend production build: passed

API Overview

Core endpoints:

• GET /api/v1/dashboard
• GET /api/v1/datasets
• POST /api/v1/datasets/import
• GET /api/v1/datasets/{id}/reviews
• POST /api/v1/preprocessing/preview
• POST /api/v1/experiments
• GET /api/v1/experiments
• GET /api/v1/experiments/{id}/events
• GET /api/v1/experiments/{id}/metrics
• GET /api/v1/experiments/compare
• GET /api/v1/models
• POST /api/v1/models/{id}/promote
• POST /api/v1/predictions
• GET /api/v1/features/top
• GET /api/v1/errors

FastAPI Swagger docs are available at:

http://127.0.0.1:8000/docs

GitHub Push Checklist

Generated runtime files are ignored by .gitignore, including:

• .env
• SQLite databases
• artifacts/
• data/*.csv
• frontend/node_modules/
• frontend/dist/
• test/cache folders

Initialize and push:

git init
git add .
git commit -m "Initial IMDB sentiment analysis lab"
git branch -M main
git remote add origin https://github.com/<your-user>/<your-repo>.git
git push -u origin main

Notes

• Model binaries are stored as local artifacts, not in the database.
• Do not upload arbitrary external joblib files; only internally generated artifacts should be loaded.
• Random seed defaults to 42 for reproducible train/test splits.
• PostgreSQL is recommended for persistent multi-run usage; SQLite is convenient for local smoke testing.