Skip to content

code-assistant

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History
 
 

code-assistant

README.md

English | 简体中文 | 繁體中文 | Русский

Code Assistant

Local LLM with MCP tool access and semantic code search for AI coding assistants (Cline, Claude, Cursor, etc.).

Services: Ollama (LLM) + LiteLLM (gateway) + MCP Gateway + Embeddings

Memory: ~5 GB RAM (with a 3B model)

Platforms: linux/amd64, linux/arm64

Architecture

graph LR
    U["👤 User"] -->|use| C["🤖 AI client<br/>(Cline, Claude, etc.)"]
    C -->|MCP tools| M["MCP Gateway<br/>(MCP endpoint)"]
    C -->|chat| L["LiteLLM<br/>(AI gateway)"]
    L -->|routes to| O["Ollama<br/>(local LLM)"]
    L -->|MCP protocol| M
    C -->|embed| E["Embeddings<br/>(text → vectors)"]
Loading

Services

Service Role Default port
Ollama (LLM) Runs local LLM models (llama3, qwen, mistral, etc.) 11434
LiteLLM AI gateway with Admin UI — routes requests to Ollama and 100+ providers 4000
MCP Gateway Provides MCP tools (filesystem, fetch, GitHub, search, databases) to AI clients 3000
Embeddings Converts text to vectors for semantic search and RAG 8000

Quick start

git clone https://github.com/hwdsl2/docker-ai-stack
cd docker-ai-stack/stacks/code-assistant
docker compose up -d

Pull a model (required before making LLM requests):

docker exec ollama ollama_manage --pull llama3.2:3b

GPU acceleration (NVIDIA CUDA)

For NVIDIA GPU acceleration, use the CUDA compose file:

docker compose -f docker-compose.cuda.yml up -d

Requirements: NVIDIA GPU, NVIDIA driver 535+, and the NVIDIA Container Toolkit installed on the host. CUDA images are linux/amd64 only.

Running without Docker Compose

If you prefer using docker run commands directly, first create a shared network so services can communicate:

docker network create ai-stack

Then start each service on the shared network:

# PostgreSQL (required by LiteLLM)
docker run -d --name litellm-db --restart always \
    --network ai-stack \
    -e POSTGRES_USER=litellm \
    -e POSTGRES_PASSWORD=litellm \
    -e POSTGRES_DB=litellm \
    -v litellm-db:/var/lib/postgresql \
    postgres:18

# Ollama (LLM)
docker run -d --name ollama --restart always \
    --network ai-stack \
    -v ollama-data:/var/lib/ollama \
    -v ollama-shared:/var/lib/ollama-shared \
    hwdsl2/ollama-server

# MCP Gateway
docker run -d --name mcp --restart always \
    --network ai-stack \
    -v mcp-data:/var/lib/mcp \
    -v mcp-shared:/var/lib/mcp-shared \
    hwdsl2/mcp-gateway

# Embeddings
docker run -d --name embeddings --restart always \
    --network ai-stack \
    -p 127.0.0.1:8000:8000 \
    -v embeddings-data:/var/lib/embeddings \
    hwdsl2/embeddings-server

# LiteLLM (AI gateway)
docker run -d --name litellm --restart always \
    --network ai-stack \
    -p 4000:4000 \
    -e LITELLM_OLLAMA_BASE_URL=http://ollama:11434 \
    -e LITELLM_MCP_URL=http://mcp:3000/mcp \
    -e LITELLM_DATABASE_URL=postgresql://litellm:litellm@litellm-db:5432/litellm \
    -v litellm-data:/etc/litellm \
    -v ollama-shared:/var/lib/ollama-shared:ro \
    -v mcp-shared:/var/lib/mcp-shared:ro \
    hwdsl2/litellm-server

Note: The shared network allows services to reach each other by container name (e.g., LiteLLM connects to Ollama via http://ollama:11434).

Pull a model (required before making LLM requests):

docker exec ollama ollama_manage --pull llama3.2:3b

Verify deployment

After starting the stack, you can verify that all services are running correctly:

# Run from the docker-ai-stack root directory
../../stack-check.sh

Access the LiteLLM Admin UI:

Open http://<server-ip>:4000/ui in your browser. Log in with username admin and your LiteLLM master key as the password. The UI provides virtual key management, spend tracking, and model configuration.

Note: For internet-facing deployments, using a reverse proxy to add HTTPS is strongly recommended. In that case, also change "4000:4000/tcp" to "127.0.0.1:4000:4000/tcp" in docker-compose.yml, to prevent direct access to the unencrypted port.

Try it in the Playground:

In the Admin UI, click Playground in the left menu. Select a local model (e.g., ollama/llama3.2:3b) from the dropdown and start chatting — this is a quick way to verify your local LLM is working end-to-end.

Customization

Each service can be configured with an optional env file. Copy the example env file from the respective repository, edit it, and uncomment the volume mount in docker-compose.yml:

Service Env file Repository
Ollama ollama.env docker-ollama
LiteLLM litellm.env docker-litellm
MCP Gateway mcp.env docker-mcp-gateway
Embeddings embed.env docker-embeddings

For detailed configuration options, API reference, and model management, see the documentation in each service's repository.

Internet-facing deployments

By default, all services listen over plain HTTP. For internet-facing deployments, place a reverse proxy (e.g., Caddy, Nginx, or Traefik) in front of the stack to provide HTTPS. Each service repository includes a detailed reverse proxy guide with Caddy and nginx examples.

Backup and restore

For backup/restore instructions, see the Backup and Restore guide.

Update images

To update all services to the latest versions:

docker compose pull
docker compose up -d

Your data is preserved in the Docker volumes. Always back up before upgrading.

Connect MCP Gateway to LiteLLM

LiteLLM and MCP Gateway are automatically wired when using the compose file or the docker run commands above — no manual key setup is needed.

API keys are shared automatically between services via Docker shared volumes:

  • MCP Gateway generates an API key on first start and copies it to the mcp-shared volume
  • LiteLLM reads the MCP key from the shared volume on startup

The LITELLM_MCP_URL=http://mcp:3000/mcp environment variable is pre-configured, so all services are connected automatically.

Usage

# Get API keys
LITELLM_KEY=$(docker exec litellm litellm_manage --getkey)
MCP_KEY=$(docker exec mcp mcp_manage --showkey | grep '^mcp-' | head -1)

# Use with an AI client (e.g., Cline in VS Code):
# LLM endpoint: http://localhost:4000 (with LITELLM_KEY)
# MCP endpoint: http://localhost:3000/mcp (with MCP_KEY)

# Generate embeddings for semantic code search
curl -s http://localhost:8000/v1/embeddings \
    -H "Content-Type: application/json" \
    -d '{"input": "function to handle authentication", "model": "text-embedding-ada-002"}' \
    | jq '.data[0].embedding[:5]'