aicompanion

KO | EN

gitlite — search

#video #python #hacktoberfest #react #web #generative-ai #ui #database #command-line #postgresql

(★ 9)

Local-first, voice-driven AI companion — 3D avatar with lip-sync, streaming voice, and persistent memory. Runs on your Mac; only the LLM call is remote.

File Explorer

ZIP Download

- - __init__.py
  - asr_whisper.py
  - interrupt.py
  - llm.py
  - memory.py
  - mock_services.py
  - noise.py
  - timing.py
  - transport.py
  - tts_kokoro.py
  - vad.py
- __init__.py
- config.py
- kokoro_synth.py
- main.py
- protocol.py
- server.py
- banner.svg
- screenshot.png
- app.js
- avatar.js
- index.html
- lipsync.js
- package-lock.json
- package.json
- ui.js
- ai_companion_build_plan.md
- run.sh
- setup.sh
- stop.sh
- lessons.md
- todo.md
- lipsync_check.mjs
- test_contract.py
.env.example
.gitignore
.python-version
LICENSE
README.md
requirements.txt

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Talk to a 3D avatar that listens, replies in a real voice, lip-syncs, and remembers you — all running on your Mac. 
 Only the LLM call ever leaves your machine.

---

## What is this?

**AI Companion** is a locally-running, voice-driven companion. You speak; a friendly female
upper-torso avatar in your browser listens, thinks, and talks back in a natural voice with
real-time lip-sync. Over time it gets to know you — it asks questions, remembers your answers and
the name you give it, and draws connections across conversations.

Everything runs on the host Mac — microphone noise suppression, voice-activity detection,
speech-to-text, text-to-speech, the 3D avatar, and the memory store. The **only** thing that leaves
your machine is the LLM call (and the small fact-extraction call for memory).

> **Pipecat is the brain; TalkingHead is the mouth.** Pipecat owns mic → noise → VAD → STT → memory
> → LLM → sentence-chunking → Kokoro synthesis and barge-in detection. It does **not** play the TTS
> audio — each synthesized sentence (audio + word/phoneme timestamps) is shipped to the browser,
> which plays it through `TalkingHead.speakAudio()` so visemes ride the audio clock. You can
> interrupt it at any time (barge-in).

## ✨ Features

- 🎙️ **Real voice conversation** — local Whisper STT, streaming LLM, local Kokoro TTS.
- 🧑‍🦰 **3D talking avatar** with phoneme-accurate lip-sync (TalkingHead + three.js).
- 🧠 **Persistent memory** — remembers facts about you and the name you give it, across sessions (Hindsight).
- ⚡ **Low latency** — ~1.5–2.5 s from "you stop" to "it speaks"; sentences pipeline as they stream.
- ✋ **Barge-in** — talk over it and it stops and listens.
- 🔒 **Local-first & private** — audio, models, and memory stay on your machine.
- 🧪 **Mock mode** — run the whole loop with no models and no API key.

## 🎬 How it works

```mermaid
flowchart LR
 MIC["🎙️ Browser mic"]:::edge
 subgraph PC["Pipecat — local voice pipeline (your Mac)"]
 direction LR
 RNN["RNNoise noise suppression"] --> VAD["Silero VAD turns + barge-in"] --> ASR["Whisper speech to text"] --> MEM["Hindsight memory · optional"] --> LLM["LLM OpenAI · remote"]:::remote --> TTS["Kokoro TTS audio + timestamps"]
 end
 TH["TalkingHead 3D avatar + lip-sync"]:::edge
 MIC --> RNN
 TTS --> TH
 TH -. "barge in anytime" .-> MIC
 classDef edge fill:#1b2a3a,stroke:#9ecbff,color:#e7eaee
 classDef remote fill:#3a2330,stroke:#d06a86,color:#ffffff
```

The avatar plays each sentence as soon as it's synthesized, while the next one is still being
generated. Full design, rationale, and the hard-won gotchas are in
**[`ai_companion_build_plan.md`](plan/ai_companion_build_plan.md)**.

## 📋 Requirements

- macOS on **Apple Silicon** (for MLX/Metal Whisper; a CPU fallback works anywhere)
- **Python 3.10–3.12** (Kokoro/torch need `<3.13`); this repo pins **3.12.8** via pyenv
- **`espeak-ng`** (Kokoro G2P): `brew install espeak-ng`
- **Node/npm** (bundles the Pipecat JS client — `setup.sh` runs the build)
- An **`OPENAI_API_KEY`** — the only required secret (powers the LLM and memory's fact extraction)

## 🚀 Quick start

```bash
scripts/setup.sh          # pyenv 3.12.8 + venv + deps + espeak-ng + frontend bundle
cp .env.example .env      # add your OPENAI_API_KEY (set MEMORY_ENABLED=1 to enable memory)
scripts/run.sh            # runs in this terminal (logs here); opens the browser
```

Open <http://localhost:8080>, click **Start**, and talk. The first run downloads the Whisper,
Kokoro, and Silero weights into `./models` (they persist). 🎧 **Headphones recommended** — they stop
the avatar's voice from re-entering your mic.

### Try it with no models or API key

Every stage has a mock, so you can verify the full transport + lip-sync loop with zero downloads:

```bash
. .venv/bin/activate
KOKORO_MOCK=1 ASR_ENGINE=mock LLM_MOCK=1 uvicorn backend.server:app --host 0.0.0.0 --port 8080
```

The avatar boots, connects, and lip-syncs a canned reply over a sine tone.

## ⚙️ Configuration

All via `.env` (see [`.env.example`](.env.example)). Key knobs:

| Var | Default | Meaning |
|---|---|---|
| `OPENAI_API_KEY` | — | required (LLM + memory extraction) |
| `OPENAI_MODEL` | `gpt-4o-mini` | LLM model — keep it **non-reasoning** for low latency |
| `ASR_ENGINE` | `whisper-mlx` | `whisper-mlx` (Apple GPU) · `whisper-cpu` · `mock` |
| `KOKORO_VOICE` | `af_heart` | Kokoro voice id |
| `WHISPER_NO_SPEECH_PROB` | `0.6` | drop non-speech segments (anti-hallucination) |
| `MEMORY_ENABLED` | `0` | `1` = persistent Hindsight memory (recall + retain) |
| `MEMORY_USER_ID` | `default` | per-user memory bank |
| `KOKORO_MOCK` / `LLM_MOCK` | `0` | mock TTS / mock LLM for model-free runs |

## 🧠 How memory works

Set `MEMORY_ENABLED=1` and `run.sh` also starts a local **Hindsight** server (embedded
Postgres + pgvector, local embeddings). Before each reply it recalls relevant memories; after each
exchange it extracts and stores facts. Nothing about you leaves your machine except the small
OpenAI extraction call. Tell it your name and a few things about yourself, restart, and it'll greet
you knowing them.

## 🗂️ Project layout

```
backend/
  server.py            FastAPI: /ws, /health, /config, serves the frontend
  main.py              builds + runs one Pipecat pipeline per WS connection
  config.py            env parsing + the companion persona
  protocol.py          server→client message schemas (single source of truth)
  kokoro_synth.py      Kokoro synthesis + WAV helpers
  pipeline/            transport · noise · vad · asr_whisper · llm · memory
                       tts_kokoro · interrupt · mock_services · timing
frontend/
  index.html app.js avatar.js lipsync.js ui.js   # source
  package.json                                    # esbuild → app.bundle.js
scripts/  setup.sh · run.sh · stop.sh
tests/    test_contract.py · lipsync_check.mjs
docs/     banner.svg · screenshot.png
```

## ✅ Verify

```bash
. .venv/bin/activate
python -m py_compile backend/*.py backend/pipeline/*.py    # everything compiles
python tests/test_contract.py                              # mock synth + packet schema
node tests/lipsync_check.mjs                                # phoneme→viseme reduction
```

## 🛠️ Built with

[Pipecat](https://github.com/pipecat-ai/pipecat) · [TalkingHead](https://github.com/met4citizen/TalkingHead)
· [Kokoro TTS](https://github.com/hexgrad/kokoro) · [Whisper](https://github.com/openai/whisper)
(MLX / faster-whisper) · [Silero VAD](https://github.com/snakers4/silero-vad)
· [three.js](https://threejs.org) · [Hindsight](https://pypi.org/project/hindsight-all/) · OpenAI

## 📖 Design doc

The full architecture, every decision, the measured latency budget, and the debugging lessons live
in **[`ai_companion_build_plan.md`](plan/ai_companion_build_plan.md)** — written so someone could rebuild
this from scratch.

## 📄 License

[MIT](LICENSE) © Xavier Anguera. Built on excellent open-source projects (see *Built with*);
their respective licenses apply to those components.

aicompanion

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

aicompanion

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README