mlx-AuK

KO | EN

gitlite — search

#nlp #php #go #ui #github #linux #awesome-list #framework #javascript #computer-vision

(★ 9)

AuK: Native Apple Silicon MLX Port for 1.5B Speech Foundation Model (TTS, Vocal/Content Edit, Enhancement & Separation)

File Explorer

ZIP Download

- - accent-dongbei-input.wav
  - accent-fujian-input.wav
  - accent-hubei-input.wav
  - accent-hunan-input.wav
  - accent-india-input.wav
  - accent-japan-input.wav
  - accent-sichuan-input.wav
  - accent-tibetan-input.wav
  - ce-en-1-input.wav
  - ce-en-2-input.wav
  - ce-zh-1-input.wav
  - ce-zh-2-input.wav
  - emotion-edit_en-1-input.wav
  - emotion-edit_en-2-input.wav
  - emotion-edit_zh-1-input.wav
  - emotion-edit_zh-2-input.wav
  - en-1-input.wav
  - en-2-input.wav
  - en-c-input.wav
  - en-d-input.wav
  - energy-edit-1-input.wav
  - energy-edit-2-input.wav
  - pitch-1-input.wav
  - pitch-2-input.wav
  - se-en-1-input.wav
  - se-en-2-input.wav
  - se-zh-1-input.wav
  - se-zh-2-input.wav
  - speed-edit-1-input.wav
  - speed-edit-input.wav
  - sr-en-1-input.wav
  - sr-en-2-input.wav
  - sr-zh-1-input.wav
  - sr-zh-2-input.wav
  - vc-1-input.wav
  - vc-2-input.wav
  - vc-3-input.wav
  - vc-4-input.wav
  - vocal-1-input.wav
  - vocal-2-input.wav
  - vocal-3-input.wav
  - vocal-4-input.wav
  - vocaledit-en-1-input.wav
  - vocaledit-en-2-input.wav
  - vocaledit-zh-1-input.wav
  - vocaledit-zh-2-input.wav
  - wh-n2w-en-input.wav
  - wh-n2w-zh-input.wav
  - wh-w2n-en-input.wav
  - wh-w2n-zh-input.wav
  - zh-1-input.wav
  - zh-2-input.wav
  - zh-a-input.wav
  - zh-b-input.wav
  - zs-tts-1-input.wav
  - zs-tts-2-input.wav
  - zs-tts-3-input.wav
  - zs-tts-4-input.wav
- - accent-dongbei-output.wav
  - accent-fujian-output.wav
  - accent-hubei-output.wav
  - accent-hunan-output.wav
  - accent-india-output.wav
  - accent-japan-output.wav
  - accent-sichuan-output.wav
  - accent-tibetan-output.wav
  - ce-en-1-add.wav
  - ce-en-1-change.wav
  - ce-en-1-delete.wav
  - ce-en-1-input.wav
  - ce-en-2-add.wav
  - ce-en-2-change.wav
  - ce-en-2-delete.wav
  - ce-en-2-input.wav
  - ce-zh-1-add.wav
  - ce-zh-1-change.wav
  - ce-zh-1-delete.wav
  - ce-zh-1-input.wav
  - ce-zh-2-add.wav
  - ce-zh-2-change.wav
  - ce-zh-2-delete.wav
  - ce-zh-2-input.wav
  - emotion-edit_en-1-input.wav
  - emotion-edit_en-2-input.wav
  - emotion-edit_zh-1-input.wav
  - emotion-edit_zh-2-input.wav
  - en-1-afraid.wav
  - en-1-angry.wav
  - en-1-content.wav
  - en-1-happy.wav
  - en-1-input.wav
  - en-1-number.wav
  - en-1-sad.wav
  - en-2-afraid.wav
  - en-2-angry.wav
  - en-2-content.wav
  - en-2-happy.wav
  - en-2-input.wav
  - en-2-number.wav
  - en-2-sad.wav
  - en-c-breath.wav
  - en-c-input.wav
  - en-c-pause.wav
  - en-c-sneeze.wav
  - en-d-cry.wav
  - en-d-hiss.wav
  - en-d-hmm.wav
  - en-d-input.wav
  - energy-edit-1-+10.wav
  - energy-edit-1-+15.wav
  - energy-edit-1-+5.wav
  - energy-edit-1--10.wav
  - energy-edit-1--15.wav
  - energy-edit-1--5.wav
  - energy-edit-1-input.wav
  - energy-edit-2-+10.wav
  - energy-edit-2-+15.wav
  - energy-edit-2-+5.wav
  - energy-edit-2--10.wav
  - energy-edit-2--15.wav
  - energy-edit-2--5.wav
  - energy-edit-2-input.wav
  - instruct-tts-1-output.wav
  - instruct-tts-2-output.wav
  - instruct-tts-3-output.wav
  - instruct-tts-4-output.wav
  - pitch-1-+1.wav
  - pitch-1-+2.wav
  - pitch-1-+3.wav
  - pitch-1--1.wav
  - pitch-1--2.wav
  - pitch-1--3.wav
  - pitch-1-input.wav
  - pitch-2-+1.wav
  - pitch-2-+2.wav
  - pitch-2-+3.wav
  - pitch-2--1.wav
  - pitch-2--2.wav
  - pitch-2--3.wav
  - pitch-2-input.wav
  - se-en-1-output.wav
  - se-en-2-output.wav
  - se-zh-1-output.wav
  - se-zh-2-output.wav
  - speed-edit-0.5.wav
  - speed-edit-0.75.wav
  - speed-edit-1-0.5.wav
  - speed-edit-1-0.75.wav
  - speed-edit-1-1.25.wav
  - speed-edit-1-1.5.wav
  - speed-edit-1-2.wav
  - speed-edit-1-input.wav
  - speed-edit-1.25.wav
  - speed-edit-1.5.wav
  - speed-edit-2.wav
  - speed-edit-input.wav
  - sr-en-1-output.wav
  - sr-en-2-output.wav
  - sr-zh-1-output.wav
  - sr-zh-2-output.wav
  - vc-1-output.wav
  - vc-2-output.wav
  - vc-3-output.wav
  - vc-4-output.wav
  - vocal-1-output.wav
  - vocal-2-output.wav
  - vocal-3-output.wav
  - vocal-4-output.wav
  - vocaledit-en-1-output.wav
  - vocaledit-en-2-output.wav
  - vocaledit-zh-1-output.wav
  - vocaledit-zh-2-output.wav
  - wh-n2w-en-output.wav
  - wh-n2w-zh-output.wav
  - wh-w2n-en-output.wav
  - wh-w2n-zh-output.wav
  - zh-1-afraid.wav
  - zh-1-calm.wav
  - zh-1-content.wav
  - zh-1-happy.wav
  - zh-1-input.wav
  - zh-1-number.wav
  - zh-1-sad.wav
  - zh-2-afraid.wav
  - zh-2-angry.wav
  - zh-2-content.wav
  - zh-2-happy.wav
  - zh-2-input.wav
  - zh-2-number.wav
  - zh-2-sad.wav
  - zh-a-cough.wav
  - zh-a-filler.wav
  - zh-a-input.wav
  - zh-a-sigh.wav
  - zh-b-inhale.wav
  - zh-b-input.wav
  - zh-b-laugh.wav
  - zh-b-surprise.wav
  - zs-tts-1-output.wav
  - zs-tts-2-output.wav
  - zs-tts-3-output.wav
  - zs-tts-4-output.wav
- .DS_Store
- - accent-edit_accent-dongbei.wav
  - accent-edit_accent-fujian.wav
  - accent-edit_accent-hubei.wav
  - accent-edit_accent-hunan.wav
  - accent-edit_accent-india.wav
  - accent-edit_accent-japan.wav
  - accent-edit_accent-sichuan.wav
  - accent-edit_accent-tibetan.wav
  - ce-en-1_add.wav
  - ce-en-1_delete.wav
  - ce-en-1_replace.wav
  - ce-en-2_add.wav
  - ce-en-2_delete.wav
  - ce-en-2_replace.wav
  - ce-zh-1_add.wav
  - ce-zh-1_delete.wav
  - ce-zh-1_replace.wav
  - ce-zh-2_add.wav
  - ce-zh-2_delete.wav
  - ce-zh-2_replace.wav
  - content-edit-speech_ce-en-1.wav
  - content-edit-speech_ce-en-2.wav
  - content-edit-speech_ce-zh-1.wav
  - content-edit-speech_ce-zh-2.wav
  - emotion-edit_emo-en-1.wav
  - emotion-edit_emo-en-2.wav
  - emotion-edit_emo-zh-1.wav
  - emotion-edit_emo-zh-2.wav
  - en-1_content.wav
  - en-1_number.wav
  - en-2_content.wav
  - en-2_number.wav
  - enhance-speech_se-en-1.wav
  - enhance-speech_se-en-2.wav
  - enhance-speech_se-zh-1.wav
  - enhance-speech_se-zh-2.wav
  - extract-vocals_ev-1.wav
  - extract-vocals_ev-2.wav
  - extract-vocals_ev-3.wav
  - extract-vocals_ev-4.wav
  - instruct-tts_instruct-tts-1.wav
  - instruct-tts_instruct-tts-2.wav
  - instruct-tts_instruct-tts-3.wav
  - instruct-tts_instruct-tts-4.wav
  - nonverbal-edit_en-c.wav
  - nonverbal-edit_en-d.wav
  - nonverbal-edit_zh-a.wav
  - nonverbal-edit_zh-b.wav
  - pitch-edit_pitch-1.wav
  - pitch-edit_pitch-2.wav
  - separate-speech_en-1.wav
  - separate-speech_en-2.wav
  - separate-speech_zh-1.wav
  - separate-speech_zh-2.wav
  - speed-edit_speed-1.wav
  - speed-edit_speed-2.wav
  - super-resolution_sr-en-1.wav
  - super-resolution_sr-en-2.wav
  - super-resolution_sr-zh-1.wav
  - super-resolution_sr-zh-2.wav
  - vocal-edit_vocaledit-en-1.wav
  - vocal-edit_vocaledit-en-2.wav
  - vocal-edit_vocaledit-zh-1.wav
  - vocal-edit_vocaledit-zh-2.wav
  - voice-edit_vc-1.wav
  - voice-edit_vc-2.wav
  - voice-edit_vc-3.wav
  - voice-edit_vc-4.wav
  - volume-edit_volume-1.wav
  - volume-edit_volume-2.wav
  - whisper-edit_wh-n2w-en.wav
  - whisper-edit_wh-n2w-zh.wav
  - whisper-edit_wh-w2n-en.wav
  - whisper-edit_wh-w2n-zh.wav
  - zero-shot-tts_zs-tts-1.wav
  - zero-shot-tts_zs-tts-2.wav
  - zero-shot-tts_zs-tts-3.wav
  - zero-shot-tts_zs-tts-4.wav
  - zh-1_content.wav
  - zh-1_number.wav
  - zh-2_content.wav
  - zh-2_number.wav
- - accent-edit_accent-dongbei.wav
  - accent-edit_accent-fujian.wav
  - accent-edit_accent-hubei.wav
  - accent-edit_accent-hunan.wav
  - accent-edit_accent-india.wav
  - accent-edit_accent-japan.wav
  - accent-edit_accent-sichuan.wav
  - accent-edit_accent-tibetan.wav
  - ce-en-1_add.wav
  - ce-en-1_delete.wav
  - ce-en-1_replace.wav
  - ce-en-2_add.wav
  - ce-en-2_delete.wav
  - ce-en-2_replace.wav
  - ce-zh-1_add.wav
  - ce-zh-1_delete.wav
  - ce-zh-1_replace.wav
  - ce-zh-2_add.wav
  - ce-zh-2_delete.wav
  - ce-zh-2_replace.wav
  - content-edit-speech_ce-en-1.wav
  - content-edit-speech_ce-en-2.wav
  - content-edit-speech_ce-zh-1.wav
  - content-edit-speech_ce-zh-2.wav
  - emotion-edit_emo-en-1.wav
  - emotion-edit_emo-en-2.wav
  - emotion-edit_emo-zh-1.wav
  - emotion-edit_emo-zh-2.wav
  - en-1_content.wav
  - en-1_number.wav
  - en-2_content.wav
  - en-2_number.wav
  - enhance-speech_se-en-1.wav
  - enhance-speech_se-en-2.wav
  - enhance-speech_se-zh-1.wav
  - enhance-speech_se-zh-2.wav
  - extract-vocals_ev-1.wav
  - extract-vocals_ev-2.wav
  - extract-vocals_ev-3.wav
  - extract-vocals_ev-4.wav
  - instruct-tts_instruct-tts-1.wav
  - instruct-tts_instruct-tts-2.wav
  - instruct-tts_instruct-tts-3.wav
  - instruct-tts_instruct-tts-4.wav
  - nonverbal-edit_en-c.wav
  - nonverbal-edit_en-d.wav
  - nonverbal-edit_zh-a.wav
  - nonverbal-edit_zh-b.wav
  - pitch-edit_pitch-1.wav
  - pitch-edit_pitch-2.wav
  - separate-speech_en-1.wav
  - separate-speech_en-2.wav
  - separate-speech_zh-1.wav
  - separate-speech_zh-2.wav
  - speed-edit_speed-1.wav
  - speed-edit_speed-2.wav
  - super-resolution_sr-en-1.wav
  - super-resolution_sr-en-2.wav
  - super-resolution_sr-zh-1.wav
  - super-resolution_sr-zh-2.wav
  - vocal-edit_vocaledit-en-1.wav
  - vocal-edit_vocaledit-en-2.wav
  - vocal-edit_vocaledit-zh-1.wav
  - vocal-edit_vocaledit-zh-2.wav
  - voice-edit_vc-1.wav
  - voice-edit_vc-2.wav
  - voice-edit_vc-3.wav
  - voice-edit_vc-4.wav
  - volume-edit_volume-1.wav
  - volume-edit_volume-2.wav
  - whisper-edit_wh-n2w-en.wav
  - whisper-edit_wh-n2w-zh.wav
  - whisper-edit_wh-w2n-en.wav
  - whisper-edit_wh-w2n-zh.wav
  - zero-shot-tts_zs-tts-1.wav
  - zero-shot-tts_zs-tts-2.wav
  - zero-shot-tts_zs-tts-3.wav
  - zero-shot-tts_zs-tts-4.wav
  - zh-1_content.wav
  - zh-1_number.wav
  - zh-2_content.wav
  - zh-2_number.wav
- ALL_62_DEMOS_RTF_REPORT.md
- OPTIMIZATION_AND_QUANTIZATION_REPORT.md
- RTF_BENCHMARK_REPORT.md
- convert_to_mlx.py
- download_demo_assets.py
- download_models.py
- quantize_mlx.py
- rebuild_all_natural_speed.py
- run_62_base_benchmarks.py
- run_all_62_benchmarks.py
- run_auk_web_benchmarks.py
- serve_demo.py
- - __init__.py
  - anchoring.py
  - config.py
  - infer.py
- test_anchoring.py
- test_base_model.py
- test_cfm.py
- test_dit.py
- test_isolation.py
- test_quantization.py
- test_vae.py
- data.json
- index.html
.ai_memory.md
.ai_project.md
.gitignore
index.html
pyproject.toml
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**MLX-AuK** is the native Apple Silicon port of [Tencent-Hunyuan/AuK](https://github.com/Tencent-Hunyuan/AuK) (arXiv:2609.08936), an all-in-one 1.5B speech foundation model capable of zero-shot text-to-speech, speech editing, paralinguistic modification, enhancement, and voice separation through unified natural language instructions.

This repository provides:
1. **Native MLX Model Architectures**: MMDiT Transformer (`Flux2Edit`), Flow-Matching (`CFMEdit`), Causal `BigVGANFlowVAE`, and Thinker feature extraction.
2. **Apple Silicon 8-Bit Quantization**: Group-wise affine quantization reducing weights by **90.1%** (5.7GB → 0.56GB) with zero memory swapping on 16GB/18GB Mac devices.
3. **Context-Adaptive Temporal Anchoring**: Proprietary anchoring engine that eliminates attention degradation in long-audio lyric/content editing tasks.
4. **Official Demo A/B Comparison Board**: Standalone web UI comparing all **62 official benchmark samples** across 5 task families side-by-side with real-time RTF metrics.

---

## ⚡ Performance Benchmark on Apple Silicon

Benchmarked on Apple Silicon M-series unified memory architecture (10.0s target audio, 4-step DMD distillation):

| Model Variant / Runtime | 4-Step Sampling Latency | Real-Time Factor (RTF) | Generation Speed | Backbone Memory |
| :--- | :--- | :--- | :--- | :--- |
| **PyTorch MPS Baseline** | 3.820s | 0.3820 | 2.61x real-time | 5.70 GB |
| **Native MLX (FP32/BF16)** | **0.992s** | **0.0992** | **10.08x real-time** | 5.70 GB |
| **Native MLX (8-Bit Quantized)** | **1.022s** | **0.1022** | **9.79x real-time** | **0.56 GB (-90.1%)** |
| **Native MLX (4-Bit Quantized)** | **1.003s** | **0.1003** | **9.97x real-time** | **0.32 GB (-94.3%)** |

> **Key Takeaway**: Native MLX eliminates PyTorch Metal barrier synchronizations and CPU-GPU dispatch latency, pushing end-to-end 10-second audio generation under **1.0 second**.

---

## 🎯 Supported Task Families (100% Parity)

| Task Family | Capabilities |
| :--- | :--- |
| **1. Text-to-Speech (TTS)** | Instruct TTS (detailed timbre & style prompts), Zero-Shot Voice Cloning (3s reference) |
| **2. Content Editing** | Speech Content Insertion (+Add), Deletion (-Delete), Replacement, and Lyric Editing (Vocal Edit) |
| **3. Enhancement & Separation** | Denoising (Enhance Speech), Audio Quality Restoration (Super-Resolution), Vocal Extraction, Multi-speaker Separation |
| **4. Paralinguistic Editing** | Nonverbal Sound Insertion (laughter, sigh, cough, throat-clearing), Emotion Conversion, Accent Normalization, Whisper-to-Normal |
| **5. Acoustic Editing** | Semitone Pitch Modification, Volume/Gain Adjustment, Speech Rate & Tempo Scaling |

---

## 🧠 Context-Adaptive Temporal Anchoring

### Problem Solved
AuK-Flash uses a 4-step DMD distilled flow-matching recipe with CFG strictly locked to 0.0. When editing long singing or speech clips (>10s), abstract global prompts such as `Replace "X" with "Y" in the lyrics` often fail because the model relies heavily on the acoustic latent prior of the original recording, re-generating the unedited lyrics.

### Solution
`mlx_auk.anchoring` automatically inspects spoken transcripts and dynamically synthesizes temporal context windows (e.g. `把歌词中的“当恩怨搁一半”改成“当笑容搁一半”。`), giving the MMDiT attention layers localized grounding:
- **Sample `vocaledit-zh-2`**: Whisper ASR confirms successful vocal rewrite from `當恩怨過一半` to **`當笑容`**!
- **Sample `vocaledit-zh-1`**: Whisper ASR confirms successful replacement of `寂寞` with **`瘋狂`** singing vibrato!

---

## 📦 Hugging Face Weights

Pre-converted MLX safetensors for both **AuK-Flash** and **AuK-Base** are hosted directly on Hugging Face:

---

## 🚀 Quickstart

### 1. Installation

```bash
# Clone the repository
git clone https://github.com/vanch007/mlx-AuK.git
cd mlx-AuK

# Install dependencies
pip install mlx soundfile numpy torch torchaudio transformers huggingface_hub
```

### 2. Python Inference API

```python
from mlx_auk.infer import AukInfer, save_audio

# Automatically loads local Native MLX weights or downloads from Hugging Face
# Automatically loads AuK-Flash (4-step distilled) by default
infer = AukInfer(variant="flash")

# Or load the full AuK-Base foundation model (32-step high-fidelity generation)
# infer = AukInfer(variant="base")

# 1. Zero-shot Voice Cloning
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Say the following with the same voice: \"Hello from Apple Silicon MLX!\""},
            {"type": "audio", "audio": "assets/demo_assets/zs-tts-1-input.wav"}
        ]
    }
]

wav, sr, metrics = infer.generate(messages, audio="assets/demo_assets/zs-tts-1-input.wav", gen_seconds=4.0)
save_audio(wav, sr, "output_clone.wav")
print(f"Generated {metrics['audio_duration']:.2f}s audio in {metrics['latency']:.2f}s (RTF: {metrics['rtf']:.3f})")
```

### 3. Running 8-Bit Quantization

```bash
python scripts/quantize_mlx.py
```

---

## 🎧 Interactive A/B Comparison Web Board

### 🌐 Live Online Demo (GitHub Pages)
You can directly experience and test all 62 samples in your browser without cloning or running any commands:
👉 **[https://vanch007.github.io/mlx-AuK/](https://vanch007.github.io/mlx-AuK/)**

### 💻 Running Locally
Launch the local web comparison board to listen to all 62 official demo samples side-by-side with MLX outputs:

```bash
python scripts/serve_demo.py 8765
```

Open your browser at: **[http://localhost:8765/web/index.html](http://localhost:8765/web/index.html)**

Features:
- **3-Track Synchronous Audio Player**: Reference Input, Official AuK Output, and Local MLX-AuK Output.
- **Diff Markup Highlighting**: Exact visual diff tagging for `+ Add`, `- Delete`, and `Replace` tasks.
- **RTF & Latency Badges**: Verified benchmark metrics on each sample card.

---

## 🧪 Running Automated Unit Tests

```bash
python tests/test_base_model.py
python tests/test_anchoring.py
python tests/test_quantization.py
python tests/test_dit.py
python tests/test_cfm.py
python tests/test_vae.py
python tests/test_isolation.py
```

---

## 📄 License & Acknowledgements

- MLX implementation licensed under the **MIT License**.
- AuK model architecture and pre-trained weights by **Tencent Hunyuan** and **Shanghai Jiao Tong University**.

mlx-AuK

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README