CrowdGate 🛡️

A high-performance reverse proxy with intelligent IP blocking powered by CrowdSec, built in Go with comprehensive observability through Prometheus and Grafana.

🎯 What is CrowdGate?

CrowdGate acts as a security gateway for your services, intercepting requests and blocking malicious IPs before they reach your backend applications. It combines:

• CrowdSec Integration: Leverage community-powered threat intelligence
• Custom Blocklists: Add your own IP blocking rules
• Reverse Proxy: Route traffic to multiple backend services
• Real-time Metrics: Prometheus metrics with Grafana dashboards
• High Performance: In-memory caching with <1ms lookup times
• Structured Logging: Nginx-format access logs with async writes

✨ Features

• ✅ Multi-layer IP Blocking

  • CrowdSec community decisions (updated every 15 minutes)
  • Custom static blocklist from config
  • In-memory caching for fast lookups (5-minute TTL)

• ✅ Reverse Proxy

  • Route multiple services from single endpoint
  • Path-based routing with prefix stripping
  • Automatic request header preservation

• ✅ Observability

  • Prometheus metrics (requests, blocks, cache hits, DB queries)
  • Pre-built Grafana dashboard
  • Nginx-format access logs
  • Dedicated blocked request logs

• ✅ Production Ready

  • Singleton database pattern
  • Prepared SQL statements
  • Background decision updater
  • Docker Compose orchestration

🚀 Quick Start

Prerequisites

• Go 1.21+
• Docker & Docker Compose
• A CrowdSec account (free at app.crowdsec.net)

Installation

1. Clone the repository

   git clone https://github.com/yourusername/crowdgate.git
   cd crowdgate

2. Set up environment variables

   cp .env.example .env
   # Edit .env and add your CrowdSec API key

3. Start supporting services

   docker-compose up -d

4. Configure CrowdSec (first time only)

   # Create a bouncer and get API key
   docker exec crowdgate-crowdsec cscli bouncers add crowdgate-bouncer
   # Add the API key to your .env file as CROWDSEC_API=<key>
   
   # Install collections
   docker exec -it crowdgate-crowdsec bash
   cscli collections install crowdsecurity/http-dos crowdsecurity/http-cve crowdsecurity/nginx
   exit
   
   # Enroll in Console for community blocklists (optional but recommended)
   docker exec crowdgate-crowdsec cscli console enroll <your-enrollment-key>

5. Run CrowdGate

   go run cmd/main.go

6. Access the services

   • CrowdGate Proxy: http://localhost:9000
   • Prometheus Metrics: http://localhost:9000/metrics
   • Grafana Dashboard: http://localhost:3000 (admin/admin)
   • Admin UI: http://localhost:9000/admin/

📁 Project Structure

crowd-gate/
├── cmd/
│   └── main.go              # Application entry point
├── config/
│   ├── config.go            # Configuration loading
│   └── service.go           # Service definitions
├── internal/
│   ├── db/                  # Database & CrowdSec LAPI client
│   ├── middleware/          # IP blocking logic
│   ├── proxy/               # Reverse proxy handlers
│   ├── router/              # Route registration
│   ├── logging/             # Access & block logging
│   └── metrics/             # Prometheus metrics
├── services/
│   ├── admin/               # Streamlit admin UI
│   ├── crowdsec/            # CrowdSec configuration
│   ├── grafana/             # Dashboards & datasources
│   ├── landing-page/        # Demo Flask app
│   └── prometheus/          # Prometheus config
├── crowdgate.yml            # Service routes & config
└── docker-compose.yml       # Container orchestration

📖 For detailed explanations, see GUIDE.md

⚙️ Configuration

Environment Variables

Create a .env file in the project root:

# Required: CrowdSec Bouncer API Key
CROWDSEC_API=your_bouncer_api_key_here

# Optional: CrowdSec LAPI URL (defaults to http://localhost:8080)
CROWDSEC_LAPI_URL=http://localhost:8080

Service Configuration

Edit crowdgate.yml to configure your services:

proxy_addr: ":9000"          # CrowdGate listen address
update_freq: "15m"           # How often to fetch CrowdSec decisions

# Custom IP blocklist (optional)
blocklist:
  - "1.2.3.4"
  - "5.6.7.8"

# Backend services
services:
  - name: "my-app"
    path: "/app/"            # URL path to match
    url: "http://localhost:3000"  # Backend URL
    strip_prefix: true       # Remove /app/ before proxying

🔧 Usage

Adding Services

1. Add service to crowdgate.yml:

   services:
     - name: "api"
       path: "/api/"
       url: "http://localhost:8080"
       strip_prefix: true

2. Restart CrowdGate (changes require restart)

Blocking IPs

Custom blocklist:

# In crowdgate.yml
blocklist:
  - "192.168.1.100"
  - "10.0.0.5"

CrowdSec decisions: CrowdSec automatically blocks IPs based on:

• Community blocklists (if enrolled)
• Detected attack patterns from logs
• Manual decisions via cscli

Monitoring

View metrics:

curl http://localhost:9000/metrics

Key metrics:

• crowdgate_requests_total - Total requests by path
• crowdgate_blocked_requests_total - Blocked requests by reason
• crowdgate_cache_hits_total - Cache performance
• crowdgate_db_queries_total - Database query counts

Grafana Dashboard: Visit http://localhost:3000 and import the dashboard from services/grafana/dashboards/

Viewing Logs

# Access logs (all requests)
tail -f logs/access.log

# Blocked requests only
tail -f logs/blocked.log

🔍 CrowdSec Commands

# View current decisions
docker exec crowdgate-crowdsec cscli decisions list

# View metrics
docker exec crowdgate-crowdsec cscli metrics show

# Install collections
docker exec crowdgate-crowdsec cscli collections install crowdsecurity/nginx

# List bouncers
docker exec crowdgate-crowdsec cscli bouncers list

# Add manual ban
docker exec crowdgate-crowdsec cscli decisions add --ip 1.2.3.4 --duration 4h --reason "Manual block"

# Remove ban
docker exec crowdgate-crowdsec cscli decisions delete --ip 1.2.3.4

📊 Performance

• Cache Hit Rate: ~95% (typical)
• Lookup Time: <1ms (cached), ~5ms (uncached)
• Throughput: 10K+ req/s on modern hardware
• Memory Usage: ~50MB base + ~1KB per cached IP

🛠️ Development

Prerequisites

# Install Go dependencies
go mod download

# Install development tools
go install github.com/rakyll/hey@latest  # Load testing

Running Tests

# Unit tests (TODO: add tests)
go test ./...

# Load test
hey -n 10000 -c 100 http://localhost:9000

📝 Roadmap

Planned Enhancements:

• Add SQL indexes for better query performance
• Add comprehensive unit tests
• Dockerize Go application (currently runs outside Docker)
• Promtail + Loki integration for advanced log aggregation
• Additional Grafana dashboards (Docker metrics, CrowdSec analytics)
• Rate limiting middleware
• CloudFlare Tunnel integration for secure remote access
• GitHub Actions CI/CD pipeline

🙏 Acknowledgments

• CrowdSec - Community-powered security
• Prometheus - Metrics & monitoring
• Grafana - Visualization
• Go-Cache - In-memory caching