AIris

An interactive AI companion with a real-time 3D anime character, voice synthesis, and emotion-driven dialogue.

---

Table of Contents

• What is AIris?
• Features
  • Character & Animation
  • Dialogue & Intelligence
  • Voice Synthesis
  • Viewport & Backgrounds
• Mood System
• Tech Stack
• Project Structure
• Prerequisites
• Quick Start
• Configuration
• Available Commands
• License

---

What is AIris?

AIris is a local-first AI companion desktop app that combines a real-time VRM character renderer, LLM-powered dialogue, and neural text-to-speech in a single interface. The character reacts to your messages with matching facial expressions, body animations, and voice — all running on your own hardware.

---

Features

Character & Animation

• VRM character rendering via Three.js + @pixiv/three-vrm — PBR shading, shadow casting, real-time blink and eye tracking
• 16 VRMA body animations — idle, greeting, blush, spin, jump, squat, angry, sad, shoot, look_around, model_pose, relax, sleepy, peace_sign, goodbye, t_pose
• Ambient idle system — automatically plays random animations between dialogue turns, keeping the character lively
• Smooth facial expressions — happy, sad, angry, surprised, relaxed, neutral with animated fade-in/fade-out transitions
• Real-time lipsync — audio-driven mouth movement synchronized to TTS playback
• Multiple characters — Ani and Shiro, each with their own personality, voice, and animation set; automatically switches character when changing chats

Dialogue & Intelligence

• LLM-powered chat — supports local GGUF models via llama-cpp-python and any OpenAI-compatible API (Groq, OpenRouter, etc.)
• Structured output — the LLM returns a single JSON with dialogue text, animation name, facial expression, intonation, and mood field
• Mood system — 7 conversation moods that override LLM personality and expression simultaneously
• Language-adaptive — automatically replies in the user's language regardless of the character's system prompt
• Persistent chat history — sessions stored in a local SQLite database, survives restarts
• Message regeneration — re-roll the last reply with one click
• Message editing — edit any sent message and continue the conversation from that point

Voice Synthesis

• Qwen3-TTS Custom Voice — preset speakers with mood-aware intonation prompts fed directly to the model
• Qwen3-TTS Voice Design — describe any voice in natural language and the model generates it
• Edge-TTS — Microsoft's neural TTS, no GPU required
• Silero — fast, lightweight fully offline TTS

Viewport & Backgrounds

• 2D anime scene backgrounds — 5 visual novel-style interior illustrations (Bedroom, Kitchen, Lounge, Bathroom, Entrance)
• Solid color background — custom color picker
• Scene mode restrictions — when a 2D scene is active: ground ring hides, zoom locks at a fixed camera distance, camera rotation is limited to the horizontal axis only
• Camera auto-rotate — optional slow orbit around the character
• Resizable chat panel — drag the panel edge to adjust width, or collapse it entirely with a toggle button

---

Mood System

The mood is set per conversation via the chat header and affects both the LLM's personality and the character's facial expression:

Mood	LLM Behaviour	Expression
✨ Natural	LLM decides freely	Dynamic
😊 Happy	Warm, joyful, enthusiastic	happy
😜 Playful	Teasing, cheeky, lighthearted	happy
😠 Angry	Irritated, edgy, short-tempered	angry
😢 Sad	Melancholic, wistful, quiet	sad
🥰 Romantic	Tender, affectionate, intimate	relaxed
😐 Serious	Direct, focused, restrained	neutral

---

Tech Stack

Layer	Technology
Frontend	Vue 3, Vite, TypeScript, Pinia
3D Rendering	Three.js, @pixiv/three-vrm, @pixiv/three-vrm-animation
Backend	Python, FastAPI, LangChain
LLM	llama-cpp-python (local) · OpenAI-compatible API (external)
TTS	Custom airis-tts package — Qwen3-TTS · Edge-TTS · Silero
Database	SQLite via SQLAlchemy
Assets	VRM/VRMA models tracked via Git LFS

---

Project Structure

AIris/
├── AIris-backend/              # FastAPI backend
│   └── src/
│       ├── api/routers/        # characters · chat · tts
│       ├── core/               # LLM factory · config
│       ├── services/           # chat_service · session_service · prompt templates
│       └── models/             # Pydantic schemas
│
├── AIris-desktop/              # Vue 3 frontend
│   ├── public/
│   │   ├── models/             # VRM characters + VRMA animations (Git LFS)
│   │   └── scenes/             # 2D background illustrations + thumbnails
│   └── src/
│       ├── components/
│       │   ├── chat/           # ChatWindow · ChatHeader · ChatMessage · ChatInput
│       │   ├── layout/         # AppLayout — panel resize, character auto-switching
│       │   ├── settings/       # SettingsModal
│       │   ├── sidebar/        # Sidebar · ChatHistory
│       │   └── viewer/         # ModelViewer · AnimationPlayer
│       ├── composables/        # useVRM · useVRMA · useAudio · useLipsync · useHDRI
│       ├── data/               # scenePresets
│       ├── stores/             # chat · model · settings  (Pinia)
│       └── types/              # shared TypeScript types
│
└── AIris-tts/                  # Custom TTS package (submodule)
    └── tts_service/
        └── backends/           # qwen3_backend · edge_backend · silero_backend

---

Prerequisites

Tool	Version	Notes
Python	≥ 3.13	via uv
Node.js	≥ 20 LTS	auto-installed by setup.sh if missing
pnpm	any	auto-installed by setup.sh if missing
uv	any	auto-installed by setup.sh if missing
CUDA GPU	≥ 4 GB VRAM	for Qwen3-TTS; use TTS_BACKEND=edge for CPU

---

Quick Start

git clone --recurse-submodules https://github.com/2EAI/AIris.git
cd AIris
./setup.sh

setup.sh will:

• Install Node.js, pnpm, and uv if not present
• Create AIris-backend/.env from .env.example
• Install all backend and frontend dependencies
• Run database migrations

Then edit AIris-backend/.env to configure your LLM and TTS, then:

make run

This starts the FastAPI backend on http://localhost:8000 and the UI at http://localhost:5173.

---

Configuration

Copy .env.example to AIris-backend/.env and edit:

Option A — Groq (free, no GPU needed for LLM)

Get a free API key at console.groq.com.

LLM_PROVIDER=openai
BASE_URL=https://api.groq.com/openai/v1
EXTERNAL_API_KEY=gsk_...
EXTERNAL_MODEL_NAME=llama-3.1-8b-instant
TTS_BACKEND=qwen3

Option B — Local GGUF model

LLM_PROVIDER=local
LOCAL_MODEL_PATH=/path/to/model.gguf
N_GPU_LAYERS=-1
LLM_BACKEND=cuda   # cuda / vulkan / metal / cpu
TTS_BACKEND=qwen3

TTS Backends

Value	Model	Requirements
qwen3	Qwen3-TTS 1.7B CustomVoice	GPU, ~4 GB VRAM
qwen3-design	Qwen3-TTS 1.7B VoiceDesign	GPU, ~4 GB VRAM
edge	Microsoft Edge TTS	internet, no GPU
silero	Silero TTS	CPU, fully offline

Download Qwen3-TTS models once before first run:

huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice
huggingface-cli download Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign

For faster inference install flash-attn (CUDA only):

make install-backend-flash

---

Available Commands

make install                # install all dependencies
make migrate                # run DB migrations
make run                    # start backend + open UI in browser
make run-backend            # start only the FastAPI backend
make run-desktop            # start only the Vite dev server
make install-backend-flash  # install flash-attn (CUDA, optional)

---

License

Proprietary. Background illustrations sourced from Spiral Atlas VN House Backgrounds (CC-BY 3.0).