Semantic Code Index

A production-ready semantic code search system that automatically indexes high-quality Python code snippets from popular open-source repositories. Search through thousands of verified code examples from Django, Flask, FastAPI, Requests, and CPython's standard library - all from your terminal.

Features

• Smart Search: Find Python code snippets using natural language queries
• Quality Verified: Every snippet is validated using heuristic analysis for code quality (7.0+ score)
• Auto-Updated: GitHub Actions runs nightly to keep the database fresh with latest code
• Zero Setup: Pure bash CLI with only curl and jq dependencies
• Curated Sources: 8,000+ snippets from production-grade repositories
• Lightning Fast: Local caching means instant searches after first download
• Context-Rich: See full function code, docstrings, source URLs, and quality scores

Quick Start

1. Install the CLI

# Download the script
curl -o code-snatch https://raw.githubusercontent.com/manusiele/semantic-code-index/main/code-snatch
chmod +x code-snatch

# Move to PATH
sudo mv code-snatch /usr/local/bin/
# Or for local install:
mkdir -p ~/.local/bin
mv code-snatch ~/.local/bin/
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
source ~/.bashrc

2. Install Dependencies

Ubuntu/Debian:

sudo apt update
sudo apt install curl jq

macOS:

brew install curl jq

3. Use It

# First time - download the database
code-snatch --update

# Search for code
code-snatch "async file download"
code-snatch "flask route with error handling"
code-snatch "decorator for timing"

# Check cache info
code-snatch --info

# Get help
code-snatch --help

How It Works

The system operates in two distinct phases: automated indexing in the cloud and local searching on your machine.

Phase 1: Automated Indexing (GitHub Actions)

┌─────────────────────────────────────────────────────────┐
│  GitHub Actions (Runs Daily at 2 AM UTC)               │
├─────────────────────────────────────────────────────────┤
│  1. Clone configured repositories                      │
│  2. Extract Python functions with docstrings           │
│  3. Generate semantic embeddings (sentence-transformers)│
│  4. Apply heuristic quality verification               │
│     • Docstring completeness                           │
│     • Code complexity analysis                         │
│     • Best practices compliance                        │
│  5. Filter snippets (quality score ≥ 7.0)             │
│  6. Commit snippets_db.json (96MB) to repository       │
└─────────────────────────────────────────────────────────┘

Indexing Pipeline Components:

• indexer_git.py: Clones repos and extracts functions using AST parsing
• embedder.py: Generates vector embeddings using all-MiniLM-L6-v2 model
• verifier_simple.py: Validates code quality with heuristic scoring
• Result: ~8,000+ verified, production-ready code snippets

Phase 2: Local Search (Your Machine)

┌─────────────────────────────────────────────────────────┐
│  Bash CLI (code-snatch)                                │
├─────────────────────────────────────────────────────────┤
│  1. Download snippets_db.json on first run             │
│  2. Cache locally (~/.semantic-code-cli)               │
│  3. Search using jq text matching                      │
│  4. Display formatted results with:                    │
│     • Function name and source                         │
│     • Full code with syntax context                    │
│     • Docstring description                            │
│     • GitHub source URL                                │
│     • Quality verification badge                       │
│  5. Auto-refresh cache every 24 hours                  │
└─────────────────────────────────────────────────────────┘

Search Method: Currently uses keyword matching on function names and docstrings. The embeddings are stored for future semantic search capabilities but not yet utilized by the bash CLI (would require Python for vector similarity calculations).

Configuration

The system is highly configurable through config/repos.json:

{
  "repos": [
    {
      "name": "django/django",
      "enabled": true,
      "description": "The Web framework for perfectionists with deadlines"
    },
    {
      "name": "psf/requests",
      "enabled": true,
      "description": "HTTP library for Python"
    }
  ],
  "indexing_settings": {
    "file_extensions": [".py"],
    "max_file_size_kb": 500,
    "min_function_lines": 3,
    "exclude_patterns": [
      "test_", "_test.py", "tests/",
      "locale/", "docs/", "examples/"
    ]
  }
}

Configuration Options:

• repos: List of GitHub repositories to index
  • name: Repository in format owner/repo
  • enabled: Toggle indexing for this repo
  • description: Human-readable description
• indexing_settings:
  • file_extensions: File types to process (currently Python only)
  • max_file_size_kb: Skip files larger than this
  • min_function_lines: Minimum function length to index
  • exclude_patterns: Paths/files to skip (tests, docs, etc.)

Development

Running the Full Pipeline Locally

# 1. Install Python dependencies
cd scripts
pip install -r requirements.txt

# 2. Set GitHub token (for higher API rate limits)
export GITHUB_TOKEN="your_github_token"

# 3. Run indexing pipeline
python indexer_git.py          # Clones repos and extracts functions
python embedder.py             # Generates embeddings
python verifier_simple.py      # Validates quality

# 4. Test the CLI
cd ..
./code-snatch "decorator"

Optional: LLM Verification with Ollama

For more sophisticated quality verification (not used in production due to CI time constraints):

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Start Ollama server
ollama serve &

# Pull the code model
ollama pull qwen2.5-coder:1.5b

# Run LLM verification
python scripts/verifier_ollama.py

Project Structure

semantic-code-index/
├── .github/workflows/
│   └── index-repos.yml         # CI/CD: Daily indexing automation
├── config/
│   └── repos.json              # Repository and indexing configuration
├── scripts/
│   ├── indexer.py              # GitHub API-based indexer (alternative)
│   ├── indexer_git.py          # Git clone-based indexer (production)
│   ├── embedder.py             # Sentence transformer embeddings
│   ├── verifier_simple.py      # Heuristic quality verification (production)
│   ├── verifier_ollama.py      # LLM-based verification (optional)
│   ├── verifier_groq.py        # Groq API verification (alternative)
│   ├── debug_pipeline.py       # Pipeline debugging utility
│   ├── test_indexer.py         # Unit tests
│   └── requirements.txt        # Python dependencies
├── data/
│   ├── .gitkeep
│   └── snippets_db.json        # Generated database (96MB, 8000+ snippets)
└── code-snatch                 # Bash CLI tool

Database Schema

Each snippet in snippets_db.json contains:

{
  "function_name": "user_passes_test",
  "code": "def user_passes_test(...):\n    ...",
  "docstring": "Decorator for views that checks...",
  "filepath": "django/contrib/auth/decorators.py",
  "source_repo": "django/django",
  "source_url": "https://github.com/django/django/blob/main/...",
  "type": "function",
  "embedding": [0.123, -0.456, ...],
  "indexed_date": "2024-02-17T09:40:00",
  "heuristic_verified": true,
  "quality_score": 7.5,
  "verification_method": "heuristic"
}

Adding New Repositories

1. Fork this repository
2. Edit config/repos.json and add your repo:

   {
     "name": "owner/repo-name",
     "enabled": true,
     "description": "Brief description"
   }

3. Commit and push - GitHub Actions will index it automatically
4. Or run locally: python scripts/indexer_git.py

Technical Details

Quality Verification

The heuristic verifier (verifier_simple.py) scores snippets based on:

1. Docstring Quality (40%)

   • Length and completeness
   • Presence of parameter descriptions
   • Return value documentation

2. Code Quality (30%)

   • Function length (not too short/long)
   • Complexity metrics
   • Naming conventions

3. Best Practices (30%)

   • Type hints presence
   • Error handling patterns
   • Documentation standards

Snippets scoring ≥7.0/10 are included in the database.

Why Heuristic Over LLM?

While the project supports LLM verification via Ollama, production uses heuristic verification because:

• Speed: Processes 8,000+ snippets in minutes vs hours
• Cost: Zero API costs, no compute overhead
• Reliability: Deterministic results, no model dependencies
• CI-Friendly: Runs in GitHub Actions without special setup

LLM verification remains available for local use when deeper quality analysis is needed.

Current Indexed Repositories

Repository	Description	Snippets
django/django	Web framework	~3,500
psf/requests	HTTP library	~800
pallets/flask	Micro framework	~600
fastapi/fastapi	Async framework	~900
python/cpython	Standard library	~2,400

Total: ~8,200 verified snippets

Limitations & Future Improvements

Current Limitations:

• Keyword-based search only (embeddings not utilized in CLI)
• Python-only (no multi-language support)
• Text matching can miss semantically similar queries
• 96MB database download on first use

Planned Improvements:

• Python CLI with true semantic search using embeddings
• Web interface for easier browsing
• Support for more languages (JavaScript, Go, Rust)
• Incremental database updates
• User-contributed snippet ratings
• Code snippet categories/tags

Contributing

Contributions are welcome! Here's how you can help:

1. Add Repositories: Submit PRs with new repos in config/repos.json
2. Improve Verification: Enhance heuristic scoring algorithms
3. Build Features: Implement semantic search, web UI, or multi-language support
4. Report Issues: Found a bug or have a suggestion? Open an issue
5. Documentation: Improve guides and examples

Development Workflow

# Fork and clone
git clone https://github.com/manusiele/semantic-code-index.git
cd semantic-code-index

# Create feature branch
git checkout -b feature/your-feature

# Make changes and test
./code-snatch "test query"

# Commit and push
git add .
git commit -m "Add: your feature description"
git push origin feature/your-feature

# Open pull request on GitHub

License

MIT License - see LICENSE file for details

Author

manusiele
manusiele254@gmail.com
GitHub

---

Star ⭐ this repo if you find it useful!