realtime-multilingual-asr-router

KO | EN

gitlite — search

#javascript #react #ai #llm #android #machine-learning #rust #linux #docker #java #pytorch #ios #mcp #open-source

realtime-multilingual-asr-router

(★ 20)

No description available.

realtime-multilingual-asr-router Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Realtime Multilingual ASR Router

Real-time multilingual ASR with code-switching orchestration. A lightweight, CPU-friendly coordinator routes between small monolingual Zipformer models (VAD + LID + rollback) instead of one large multilingual model.

## Installation

### Prerequisites

- Python 3.11+
- [uv](https://github.com/astral-sh/uv)

### Setup

```bash
git clone https://github.com/gladiaio/realtime_multilingual_asr_router.git
cd realtime_multilingual_asr_router

uv sync

# Download community Kroko ASR models (en, fr, es by default) plus VAD and LID
uv run scripts/download_models.py
```

**Notes:**

- Default Kroko download is ~450 MB (`en`, `fr`, `es`). Set `HF_TOKEN` for higher Hugging Face rate limits.
- First server start preloads one ASR model per downloaded language (~2s per language).
- REST: `GET /health`, `GET /languages` (component status and discovered ASR languages).

More languages and chunk size:

The list of available KrokoAI models is [here](https://huggingface.co/Banafo/Kroko-ASR/tree/main).
This solution is compatible with any [online transducer model](https://k2-fsa.github.io/sherpa/onnx/pretrained_models/online-transducer/index.html).

ASR languages: Kroko `de`, `en`, `es`, `fr`, `it`, `iw` (Hebrew), `nl`, `pt`, `sv`, `tr` (`sv` only supports chunk size 64), and `ru` (from AlphaCephei).

```bash
# All Kroko languages + Russian (~1.6 GB)
uv run scripts/download_models.py --languages de en es fr it iw nl pt sv tr ru

# Subset example
uv run scripts/download_models.py --languages en fr es de --chunk-size 64
```

### Configuration (optional)

Settings live in `src/realtime_multilingual_asr_router/config.py` and are loaded from:

1. Defaults in code
2. Environment variables with the `LR_` prefix (e.g. `LR_ASR_CHUNK_SIZE=64`)
3. A `.env` file in the project root (copy from `.env.example`)

```bash
cp .env.example .env
```

ASR languages are discovered from `models/streaming_transducers/<chunk_size>/<lang>/` (each folder needs `encoder.onnx`, `decoder.onnx`, `joiner.onnx`, and `tokens.txt`).

### Run

```bash
uv run uvicorn realtime_multilingual_asr_router.main:app
```

Open `http://127.0.0.1:8000/` for the frontend. WebSocket: `/ws/transcribe`.

### Development

```bash
uv run pytest
uv run ruff check .
uv run ruff format .
```

Model-backed tests are marked `slow` and may be skipped without downloaded weights.

### Project structure

```
realtime_multilingual_asr_router/
├── src/realtime_multilingual_asr_router/   # Application package
├── frontend/                   # Web UI
├── assets/                     # Architecture diagram and benchmark charts
├── tests/
├── scripts/download_models.py
├── models/                     # Downloaded weights (gitignored)
└── .env.example
```

### Models

| Model | Role | Source |
| --- | --- | --- |
| Silero VAD | Voice activity | [snakers4/silero-vad](https://github.com/snakers4/silero-vad) |
| Kroko ASR (community) | Streaming ASR (`de`, `en`, `es`, `fr`, `it`, `iw`, `nl`, `pt`, `sv`, `tr`) | [Banafo/Kroko-ASR](https://huggingface.co/Banafo/Kroko-ASR) |
| Russian Zipformer | Streaming ASR (`ru`, pass `--languages ru`) | [sherpa-onnx-streaming-zipformer-small-ru-vosk](https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-small-ru-vosk-2025-08-16) |
| VoxLingua107-ECAPA-TDNN | Language ID | [speechbrain/lang-id-voxlingua107-ecapa](https://huggingface.co/speechbrain/lang-id-voxlingua107-ecapa) |

`scripts/download_models.py` downloads only the languages you list. Kroko codes use `.data` extraction; `ru` pulls ONNX files from the Hub (no `.data`). Default: `en`, `fr`, `es`.

**Stack:** FastAPI, uvicorn, sherpa-onnx, silero-vad, speechbrain, torch, torchcodec, pytest, ruff.

## Architecture

| Component | Location |
| --- | --- |
| WebSocket server | `src/realtime_multilingual_asr_router/api/websocket.py` |
| Audio resampling (16 kHz) | `src/realtime_multilingual_asr_router/audio/resampler.py` |
| Sample-count master clock | `src/realtime_multilingual_asr_router/coordinator/sample_clock.py` |
| VAD (Silero v6) | `src/realtime_multilingual_asr_router/vad/silero.py` |
| ASR (Zipformer / sherpa-onnx) | `src/realtime_multilingual_asr_router/asr/kroko.py` |
| LID scheduler | `src/realtime_multilingual_asr_router/lid/scheduler.py` |
| Rollback engine | `src/realtime_multilingual_asr_router/rollback/engine.py` |
| Audio ring buffer | `src/realtime_multilingual_asr_router/audio/buffer.py` |
| REST API | `src/realtime_multilingual_asr_router/api/rest.py` |
| Frontend | `frontend/` |

### Asynchronous rollback pipeline

1. **Immediate transcription** — Active monolingual ASR transcribes in real time.
2. **Asynchronous monitoring** — After each VAD boundary, LID runs on expanding windows in the background.
3. **Rollback trigger** — On a confident language switch, swap ASR model, roll the transcript back to the segment boundary, and re-infer buffered audio.

Wrong-language text may appear briefly in partials; finals have clean language boundaries.

![Asynchronous rollback pipeline](https://raw.githubusercontent.com/gladiaio/realtime-multilingual-asr-router/main/assets/01_baseline_black_frame.svg)

## Evaluation

Benchmarks on three datasets: monolingual baseline ([FLEURS](https://huggingface.co/datasets/google/fleurs)), inter-utterance code-switch ([synthetic FLEURS blend](https://huggingface.co/datasets/BrunoHays/fleurs_code_switching_test)), and intra-utterance code-switch ([Bangor–Miami](https://huggingface.co/datasets/BrunoHays/Bangor-Miami-Spanish-English-Corpus)).

**This work:** 640 ms Zipformer chunks — [Kroko](https://huggingface.co/Banafo/Kroko-ASR) for `en/fr/es/de/it/pt/nl`, [Alphacephei `ru`](https://huggingface.co/csukuangfj/sherpa-onnx-streaming-zipformer-small-ru-vosk-2025-08-16).

**Baselines:** Deepgram Nova-3, ElevenLabs Scribe v2, Mistral Voxtral-Mini-4B-Realtime (480 ms), AssemblyAI u3-rt-pro (no `nl`/`ru` — excluded where those languages appear).

### Results

| Scenario | This work | Deepgram | ElevenLabs | Voxtral Mini |
| --- | --- | --- | --- | --- |
| Inter-utterance (synthetic FLEURS) | **~13%** | ~14% | ~14% | ~21% |
| Intra-utterance (Miami) | ~41% | ~29% | ~26% | ~76% |

![Monolingual FLEURS WER by language](https://raw.githubusercontent.com/gladiaio/realtime-multilingual-asr-router/main/assets/fleurs.png)

![Code-switching WER — synthetic FLEURS vs Miami corpus](https://raw.githubusercontent.com/gladiaio/realtime-multilingual-asr-router/main/assets/code-switching.png)

### Detailed WER (%)

| Dataset | Voxtral Mini | Deepgram Nova-3 | ElevenLabs Scribe v2 | AAI u3-rt-pro | This work |
| --- | --- | --- | --- | --- | --- |
| Italian (FLEURS) | 3.30 | 7.10 | 2.00 | 2.50 | 4.30 |
| Russian (FLEURS) | 5.20 | 10.20 | 7.40 | — | 14.30 |
| Portuguese (FLEURS) | 4.90 | 10.30 | 3.90 | 3.90 | 9.20 |
| English (FLEURS) | 12.40 | 10.20 | 4.30 | 3.40 | 13.20 |
| French (FLEURS) | 8.80 | 10.60 | 5.50 | 4.10 | 9.90 |
| Dutch (FLEURS) | 8.20 | 13.30 | 5.30 | — | 11.90 |
| German (FLEURS) | 6.00 | 10.00 | 3.90 | 3.40 | 11.00 |
| Spanish (FLEURS) | 3.10 | 5.90 | 2.70 | 2.40 | 4.70 |
| Code-switch (synthetic FLEURS) | 21.30 | 14.30 | 13.60 | — | **13.20** |
| Miami (intra-utterance) | 76.30 | 28.60 | 26.50 | 34.00 | 41.00 |
| **Avg (FLEURS only)** | 6.50 | 9.70 | 4.40 | **3.30** | 9.80 |

### Limitations

- **Intra-utterance gap** — Mid-sentence mixing (Spanglish, etc.) outpaces VAD boundaries; rollback rarely fires.
- **Per-language ceilings** — Accuracy depends on open-source Zipformer quality per language.

### References

**Bangor–Miami** — [Mozilla Data Collective](https://datacollective.mozillafoundation.org/). Deuchar et al. (2014), *Advances in the Study of Bilingualism*, Multilingual Matters.

**FLEURS:**

```bibtex
@article{fleurs2022arxiv,
  title = {FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech},
  author = {Conneau, Alexis and Ma, Min and Khanuja, Simran and Zhang, Yu and Axelrod, Vera and Dalmia, Siddharth and Riesa, Jason and Rivera, Clara and Bapna, Ankur},
  journal = {arXiv preprint arXiv:2205.12446},
  url = {https://arxiv.org/abs/2205.12446},
  year = {2022}
}
```

## License

MIT — see [LICENSE](LICENSE).

realtime-multilingual-asr-router

realtime-multilingual-asr-router Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

realtime-multilingual-asr-router

realtime-multilingual-asr-router Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README