ComfyUI_RH_FireRedAudio

KO | EN

gitlite — search

#privacy #rust #pytorch #agents #ai #react #cross-platform #typescript #claude-code #reactjs

ComfyUI_RH_FireRedAudio

(★ 9)

ComfyUI customnodes for FireRedAudio, supporting ASR, audio understanding, zero-shot TTS, speech editing, and voice design with optimized model caching and low-VRAM execution

File Explorer

ZIP Download

- NODE_USAGE.md
- - asr_zh_fleurs.wav
  - assets_mmau_test.wav
  - ATTRIBUTION.md
  - edit_acoustic_zh_ref.wav
  - tts_zh_prompt.wav
- fireredaudio_all_nodes_api.json
- README.md
- - nodeDefs.json
- __init__.py
- test_node_utils.py
- test_nodes_contract.py
- test_transformers_compat_runtime.py
- test_vendor_static.py
- test_workflow_contract.py
- validate_api_workflow.py
- - __init__.py
  - configuration_fireredaudio.py
  - inference.py
  - loading.py
  - modeling_fireredaudio.py
- - __init__.py
  - configuration_qwen3_5.py
  - initialization.py
  - modeling_qwen3_5.py
- __init__.py
.gitignore
__init__.py
LICENSE
nodes.py
NOTICE
README.md
README_CN.md
requirements.txt
utils.py

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![RunningHub China](https://img.shields.io/badge/RunningHub-China%20Online%20Platform-2f80ed?labelColor=333333)](https://www.runninghub.cn/?inviteCode=rh-v1367)
[![RunningHub International](https://img.shields.io/badge/RunningHub-International%20Online%20Platform-2f80ed?labelColor=333333)](https://www.runninghub.ai/?inviteCode=rh-v1367)
[![License](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://raw.githubusercontent.com/RH-RunningHub/ComfyUI_RH_FireRedAudio/main/LICENSE)

ComfyUI V3 nodes for [FireRedAudio](https://github.com/FireRedTeam/FireRedAudio),
a unified 9B audio-language model supporting speech recognition, audio
understanding, zero-shot TTS, speech editing, and voice design.

This plugin vendors the FireRedAudio inference source and a private Qwen3.5
compatibility layer for `transformers 4.57.x` environments. It does not upgrade or
monkeypatch the environment-wide Transformers installation.

## Nodes

| Node | Purpose | Output |
|---|---|---|
| `RH_FireRedAudioLoader` | Load and cache the main model; optionally load RedAE | `FIREREDAUDIO_MODEL` |
| `RH_FireRedAudioASR` | Transcribe one AUDIO with optional fixed-segment timestamps | text + timestamp text |
| `RH_FireRedAudioUnderstand` | Answer questions about one or two AUDIO inputs | answer + reasoning |
| `RH_FireRedAudioTTS` | Clone a reference voice and synthesize text | 24kHz `AUDIO` |
| `RH_FireRedAudioEdit` | Semantic or pitch/speed/volume speech editing | `AUDIO` + rewritten text |
| `RH_FireRedAudioVoiceDesign` | Synthesize speech from a timbre description | 24kHz `AUDIO` |

All media inputs and outputs use ComfyUI's standard `AUDIO` value. The nodes mix
channels to mono and resample internally; no model or audio file path is exposed as
a node input. There is no fixed input-duration limit; long recordings can require
substantial processing time, CPU RAM, and KV-cache VRAM.

## Model layout

FireRedAudio is routed as a TTS model because four of its five user-facing tasks
produce speech and the Hugging Face repository cannot be split into independent
understanding and generation weights.

Place the complete repository under the following fixed path:

```text
ComfyUI/models/TTS/FireRedAudio/
├── FireRedAudio/
│   ├── config.json
│   ├── model.safetensors.index.json
│   ├── model-00001-of-00005.safetensors
│   └── ... tokenizer and processor files
└── RedAE_decoder/
    └── model.pt
```

### Download methods

Hugging Face (recommended), run from the ComfyUI models directory:

```bash
hf download FireRedTeam/FireRedAudio --local-dir TTS/FireRedAudio
```

ModelScope mirror for users in China:

```bash
pip install modelscope
modelscope download --model FireRedTeam/FireRedAudio --local_dir TTS/FireRedAudio
```

For manual installation, download the complete package from
[Hugging Face](https://huggingface.co/FireRedTeam/FireRedAudio) or
[ModelScope](https://www.modelscope.cn/models/FireRedTeam/FireRedAudio) and preserve
the directory structure shown above.

The plugin never downloads weights automatically. ASR and understanding can use a
loader with `load_decoder=false`; TTS, edit, and voice design require
`load_decoder=true`.

## Installation

Copy this directory into `ComfyUI/custom_nodes/`, then install only the lightweight
dependency if it is not already present:

```bash
pip install -r ComfyUI_RH_FireRedAudio/requirements.txt
```

Do not replace the environment's torch, torchaudio, Transformers, CUDA, or NVIDIA
packages. The compatibility layer targets Transformers `4.57.x`.

## All-nodes workflow

The API-format workflow
[`examples/fireredaudio_all_nodes_api.json`](examples/fireredaudio_all_nodes_api.json)
contains all six plugin nodes, four core `LoadAudio` nodes, text previews, and audio
save nodes. It uses one shared decoder-enabled loader, so the 30GB model stack is
not cold-loaded once per task.

Before running, copy the bundled official samples into the ComfyUI input directory:

```bash
cp ComfyUI_RH_FireRedAudio/examples/assets/*.wav ComfyUI/input/
```

The workflow saves generated files under `ComfyUI/output/fireredaudio/`. It uses
the optional AlekPet `PreviewTextNode` for text display; replace those preview nodes
with any STRING display/save node if your installation does not provide it. More
details are in [`examples/README.md`](examples/README.md).

The API submission envelope is already included. Standard ComfyUI ignores the
additional empty `workflow_config`, while RunningHub accepts it directly:

```json
{
  "prompt": { "...": "18 workflow nodes" },
  "workflow_config": {}
}
```

## Memory behavior

- `auto`: keeps the main model resident when possible. On a 24GB card, generation
  moves the main model to CPU before moving RedAE to GPU, then restores it.
- `low_vram`: offloads both the main model and decoder after every task. CPU RAM must
  be large enough to hold the approximately 21GB main model.
- On large-memory GPUs, `auto` may keep both components resident when measured free
  memory is sufficient.

The cache key contains only model path, dtype, attention implementation, and decoder
presence. ASR, understanding, TTS, edit, and voice design share the same cached
engine when they use one loader.

## Important generation parameters

- `max_new_audio_steps`: hard-capped at 750 in both validation and execution;
  one step is approximately 0.16 seconds.
- `n_timesteps`: flow-matching denoising iterations per audio chunk.
- `inference_cfg`: classifier-free guidance strength.
- `seed`: clamped to the NumPy/PyTorch-compatible range.
- Acoustic edit builds the exact training phrases for pitch, speed, and volume;
  unsupported free-form acoustic commands are not passed through.

## Limitations and safety

- Audio understanding, TTS, edit, and voice design support Chinese and English.
  ASR supports additional languages.
- Generated audio is stochastic; keep the seed fixed for repeatability.
- The 300-second node input limit is intentionally stricter than the upstream
  one-hour large-memory claim.
- `RedAE_decoder/model.pt` is loaded with `torch.load(..., weights_only=True)`.
- Empty audio generations and missing files raise explicit errors; the nodes do not
  fabricate successful results.

### Upstream usage disclaimer

> The project incorporates zero-shot voice cloning functionality; Please note that this capability is intended **solely for academic research purposes**.

Do not use the model for illegal, abusive, fraudulent, or unauthorized voice
impersonation. The voice-cloning restriction is a material commercial/publishing
risk even though the repository and weights are distributed under Apache-2.0.

## License

The plugin contains and adapts Apache-2.0 FireRedAudio and Hugging Face Transformers
source. See [`LICENSE`](LICENSE), [`NOTICE`](NOTICE), and the provenance headers in
`vendor/`. Bundled sample audio attribution is documented in
[`examples/assets/ATTRIBUTION.md`](examples/assets/ATTRIBUTION.md).

## Sources and related links

- Upstream code: <https://github.com/FireRedTeam/FireRedAudio>
- Model: <https://huggingface.co/FireRedTeam/FireRedAudio>
- ModelScope mirror: <https://www.modelscope.cn/models/FireRedTeam/FireRedAudio>
- Paper: <https://arxiv.org/abs/2608.24168>
- Project demos: <https://fireredteam.github.io/demos/fireredaudio/>
- Apache-2.0: <https://www.apache.org/licenses/LICENSE-2.0>
- Qwen3-Next architecture reference: <https://huggingface.co/docs/transformers/model_doc/qwen3_next>

## Acknowledgements

Thanks to the [FireRed Team](https://github.com/FireRedTeam) for FireRedAudio, the
[Hugging Face Transformers](https://github.com/huggingface/transformers) maintainers
for the Qwen implementations, and the ComfyUI community.

ComfyUI_RH_FireRedAudio

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

ComfyUI_RH_FireRedAudio

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README