AutoOps — Self-Healing CI/CD Pipeline for a Task Tracker Service

Validates every change against a live, isolated "dark launch" instance with synthetic API traffic before it ever reaches production, and automatically rolls back on a failed health check, latency breach, or Prometheus alert.

Problem / Motivation

Most CI/CD setups release new builds straight into production with little pre-deployment validation or runtime observability. Bugs that would be obvious under real traffic go undetected until they hit end users, turning every deploy into a small gamble and every incident into an unplanned fire drill. AutoOps is a from-scratch implementation of a deploy pipeline that treats validation as a first-class stage, not an afterthought: every change is built, deployed to an isolated environment, and proven healthy under synthetic traffic before it's promoted — and automatically rolled back the moment it isn't.

The subject under deployment (a small Express + SQLite task-tracking API) is intentionally simple. The actual engineering is in the pipeline around it.

Architecture

flowchart LR
    Dev[Developer opens PR] --> CI[CI: ESLint + Jest]
    CI -- fail --> Stop[["Pipeline stops\n(no deploy)"]]
    CI -- pass --> Build[Build & tag Docker image]
    Build --> Dark["Deploy dark launch\n(:8082, isolated port)"]
    Dark --> Synth1[Synthetic checks:\nhealth + /tasks latency]
    Synth1 -- fail --> RB1[Ansible rollback:\nremove dark container]
    Synth1 -- pass --> Green[Promote to green deployment]
    Green --> Synth2[Synthetic checks\nunder prod-like conditions]
    Synth2 -- fail --> RB2[Ansible rollback:\nremove green container]
    Synth2 -- pass --> Prod[Promote to production]
    Prod --> Mon[Prometheus + Alertmanager\ncontinuous monitoring]
    Mon -- threshold breach --> RB3[Automated rollback\nto last stable version]
    Mon -- healthy --> Prod

Loading

Every promotion gate is enforced by code, not by hand: CI (.github/workflows/ci.yml) blocks on lint/test failures; CD (.github/workflows/cd.yml) drives Ansible playbooks (infra/ansible/) that deploy, health-probe, and — on failure — tear down the bad container automatically; infra/synthetic/dark_synthetic_checks.mjs is the traffic generator that proves the new version actually works before anything sees it.

Tech Stack

Layer	Tools / Technologies
Application	Node.js, Express, better-sqlite3
Frontend	Static HTML/JS (public/)
Testing	Jest, Supertest
Linting	ESLint (flat config, v9)
Containerization	Docker
CI/CD	GitHub Actions
Deployment automation	Ansible (dark-launch deploy + rollback playbooks)
Monitoring	Prometheus, Alertmanager, prom-client
Synthetic monitoring	Custom Node.js health/latency check script

Setup and Run

# 1. Clone
git clone https://github.com/bittubharadwaj/Devops_AutoOps.git
cd Devops_AutoOps

# 2. Install dependencies
npm install

# 3. Configure environment variables
cp .env.example .env
# defaults work out of the box for local runs — see .env.example for what each one does

# 4. Lint and test
npm run lint
npm test

# 5. Run
npm start
# -> TaskTracker running on :8080
curl http://localhost:8080/health

Run in Docker

docker build -t tasktracker .
docker run -p 8080:8080 tasktracker
curl http://localhost:8080/health

Deploy pipeline (CD)

cd.yml and the infra/ansible/ playbooks are the real dark-launch → rollback automation used during development, kept as reference infrastructure-as-code. Activating it against your own host requires a self-hosted GitHub Actions runner and these repo secrets: DOCKERHUB_USERNAME, DOCKERHUB_TOKEN, SSH_PRIVATE_KEY, VM_HOST (see .env.example for what each is for). It isn't wired to run against this repo's main branch out of the box.

Results / Impact

• CI (lint → test → Docker build → container health smoke test) gates every change on main before it's mergeable.
• 5 automated tests cover the health endpoint and the full task CRUD lifecycle (create, list, duplicate-title conflict, delete, delete-missing), run with an isolated in-memory SQLite database.
• The dark/green promotion gate is enforced by measurable thresholds, not eyeballing: synthetic checks fail the deploy if the health endpoint doesn't respond within MAX_HEALTH_LATENCY_MS (500ms) or /tasks reads exceed MAX_API_LATENCY_MS (800ms), and the Ansible rollback playbook removes the bad container automatically on either failure.
• Runtime alerting is threshold-driven via Prometheus (infra/monitoring/alert_rules.yml): an instance down for over a minute, or more than 5 alerts firing for 5+ minutes, pages Alertmanager.

Key Challenges and What I Learned

• Self-hosted runner + VM access issues stalled early CI/CD work. SSH permission errors and missing sudo access on the deployment VM blocked Docker/Ansible setup for a full sprint — the fix ended up being a standardized VM setup script rather than ad hoc SSH debugging each time.
• Migrating to ESLint's flat config (v9) eliminated .eslintrc conflicts but required rethinking per-directory overrides (e.g. a browser-globals override for public/) instead of nested config files.
• Ansible idempotency mattered more than expected: the dark-launch playbook explicitly removes any leftover container from a previous run before deploying, so a failed run never leaves the port in a half-broken state for the next one.
• Splitting CI into independent lint/test/build jobs (instead of one monolithic job) made failures much faster to diagnose — a lint failure surfaces in seconds instead of waiting on the full Docker build.

Future Improvements

• Extend synthetic checks beyond /health to exercise the full /tasks CRUD flow during dark/green validation
• Add Grafana dashboards on top of the existing Prometheus metrics for real-time pipeline/service visualization
• Add pre-commit hooks (Husky + lint-staged) to catch style issues before they reach CI
• Automate metric retention/rotation policy instead of relying on default Prometheus storage settings