Awesome-Speech-Generation

KO | EN

gitlite — search

#ai #typescript #llm #go #machine-learning #rust #docker #nodejs #macos #kubernetes #vue #security #pytorch #ios

Awesome-Speech-Generation

(★ 21)

Survey on speech generation work.

Awesome-Speech-Generation 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Awesome-Speech-Generation

A Survey on speech generation research.

## Text-Guided Voice Conversion

Style Voice Conversion by Natural Language Prompts.

* (2023) Towards General-Purpose Text-Instruction-Guided Voice Conversion
    [[paper](https://arxiv.org/abs/2309.14324)] [[demo](https://text-guided-vc.github.io/asru2023_demo/)]

* (2023) PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts
    
    [[paper](https://arxiv.org/abs/2309.09262)] [[demo](https://yaoxunji.github.io/prompt_vc/)]

## Audio-Guided Voice Conversion

Style Voice Conversion by Reference Speech.

* (2023) HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation by Hierarchical Variational Inference for Zero-shot Speech Synthesis

[[paper](https://arxiv.org/abs/2309.11977)] [[demo](https://sh-lee-prml.github.io/HierSpeechpp-demo/)] [[code](https://github.com/sh-lee-prml/HierSpeechpp)]

* (2023) PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts
    
    [[paper](https://arxiv.org/abs/2309.09262)] [[demo](https://yaoxunji.github.io/prompt_vc/)]

## Text-Guided Text-to-Speech

Style Specified by a Text Prompt.

* (2023) PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptions

[[paper](https://arxiv.org/abs/2309.08140)] [[demo](https://reppy4620.github.io/demo.promptttspp/)]

* (2023) PromptTTS 2: Describing and Generating Voices with Text Prompt
    
    [[paper](https://arxiv.org/abs/2309.02285)] [[demo](https://speechresearch.github.io/prompttts2/)]

* (2023) PromptStyle: Controllable Style Transfer for Text-to-Speech with Natural Language Descriptions
    
    [[paper](https://arxiv.org/abs/2305.19522)] [[demo](https://promptstyle.github.io/PromptStyle)]

* (2023) TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Models
    
    [[paper](https://arxiv.org/abs/2308.14430)] [[demo](https://sall-e.github.io/)]

* (2023) InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt
    
    [[paper](https://arxiv.org/abs/2301.13662)] [[demo](http://dongchaoyang.top/InstructTTS/)]

* (2022) PromptTTS: controllable text-to-speech with text descriptions
    
    [[paper](https://arxiv.org/abs/2211.12171)] [[demo](https://speechresearch.github.io/prompttts/)]

## Audio-Guided Text-to-Speech

Style Specified by an Audio (Speech) Prompt

* (2023) HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation by Hierarchical Variational Inference for Zero-shot Speech Synthesis

[[paper](https://arxiv.org/abs/2309.11977)] [[demo](https://sh-lee-prml.github.io/HierSpeechpp-demo/)] [[code](https://github.com/sh-lee-prml/HierSpeechpp)]

* (2023) Improving Language Model-Based Zero-Shot Text-to-Speech Synthesis with Multi-Scale Acoustic Prompts
    
    [[paper](https://arxiv.org/abs/2309.11977)] [[demo](https://thuhcsi.github.io/icassp2024-msvalle/)]

* (2023) Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale

[[paper](https://arxiv.org/abs/2306.15687)] [[demo](https://voicebox.metademolab.com/)]

* (2023) SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
    
    [[paper](https://arxiv.org/abs/2308.06873)] [[demo](https://www.microsoft.com/en-us/research/project/speechx/)]

* (2023) VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation
    
    [[paper](https://arxiv.org/abs/2305.16107)]

* (2023) SoundStorm: Efficient Parallel Audio Generation
    
    [[paper](https://arxiv.org/abs/2305.09636)] [[demo](https://google-research.github.io/seanet/soundstorm/examples/)]

* (2023) Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (VALL-E)
    
    [[paper](https://arxiv.org/abs/2301.02111)] [[demo](https://www.microsoft.com/en-us/research/project/vall-e-x/vall-e/)]

* (2023) Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (VALL-E X)
    
    [[paper](https://arxiv.org/abs/2303.03926)] [[demo](https://www.microsoft.com/en-us/research/project/vall-e-x/)]

* (2023) Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision
    
    [[paper](https://arxiv.org/abs/2302.03540)] [[demo](https://google-research.github.io/seanet/speartts/examples/)]

## Text-Guided Text-to-Audio

Style Specified by a Text Prompt

* (2023) Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
    
    [[paper](https://arxiv.org/abs/2305.18474)] [[demo](https://make-an-audio-2.github.io/)]

* (2023) Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model
    
    [[paper](https://arxiv.org/abs/2304.13731)] [[demo](https://tango-web.github.io/)] [[github](https://github.com/declare-lab/tango)]

* (2023) AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
    
    [[paper](https://arxiv.org/abs/2301.12503)] [[demo](https://audioldm.github.io/)] [[github](https://github.com/haoheliu/AudioLDM)]

* (2023) Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
    
    [[paper](https://arxiv.org/abs/2301.12661)] [[demo](https://text-to-audio.github.io/)] [[github](https://github.com/Text-to-Audio/Make-An-Audio)]

* (2022) AudioGen: Textually Guided Audio Generation
    
    [[paper](https://arxiv.org/abs/2209.15352)] [[demo](https://felixkreuk.github.io/audiogen/)]

## Text-Guided Text-to-Music

Style Specified by a Text Prompt

* (2023) Simple and Controllable Music Generation
    
    [[paper](https://arxiv.org/abs/2306.05284)] [[demo](https://ai.honu.io/papers/musicgen/)] [[github](https://github.com/facebookresearch/audiocraft)]

## Audio Codec

* (2023) DAC: High-Fidelity Audio Compression with Improved RVQGAN

[[paper](https://arxiv.org/abs/2306.06546)] [[github](https://github.com/descriptinc/descript-audio-codec)]

* (2023) AudioDec: An Open-source Streaming High-fidelity Neural Audio Codec

[[paper](https://arxiv.org/abs/2305.16608)] [[github](https://github.com/facebookresearch/AudioDec)]

* (2023) From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion

[[paper](https://arxiv.org/abs/2308.02560)] [[github](https://github.com/facebookresearch/audiocraft/blob/main/docs/MBD.md)]

* (2022) High Fidelity Neural Audio Compression

[[paper](https://arxiv.org/abs/2210.13438)] [[github](https://github.com/facebookresearch/encodec)]

* (2021) SoundStream: An End-to-End Neural Audio Codec

[[paper](https://arxiv.org/abs/2107.03312)]

## Multi-modality Large Language Model

* (2023) Prompting Large Language Models with Speech Recognition Abilities
    
    [[paper](https://arxiv.org/abs/2307.11795)] [[demo]()]

* (2023) On decoder-only architecture for speech-to-text and large language model integration
    
    [[paper](https://arxiv.org/abs/2307.03917)] [[demo]()]

* (2023) LMs with a Voice: Spoken Language Modeling beyond Speech Tokens
    
    [[paper](https://arxiv.org/abs/2305.15255)] [[demo]()]

* (2023) AudioPaLM: A Large Language Model That Can Speak and Listen
    
    [[paper](https://arxiv.org/abs/2306.12925)] [[demo]()]

* (2023) Listen, Think, and Understand
    
    [[paper](https://arxiv.org/abs/2305.10790)] [[demo]()]

* (2023) SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities
   
   [[paper](https://arxiv.org/abs/2305.11000)] [[demo]()]

## Speech Continuation

* (2023) Textually Pretrained Speech Language Models
    
    [[paper](https://arxiv.org/abs/2305.13009)] [[demo](https://pages.cs.huji.ac.il/adiyoss-lab/twist/)] 
    [[github](https://github.com/facebookresearch/textlesslib/tree/main/examples/twist)]

## Speech-to-Speech Translation

* (2023) SeamlessM4T—Massively Multilingual & Multimodal Machine Translation
    
    [[paper](https://ai.meta.com/research/publications/seamlessm4t-massively-multilingual-multimodal-machine-translation/)] [[demo](https://seamless.metademolab.com/)] 
    [[github](https://github.com/facebookresearch/seamless_communication)]
## Speech-to-Speech 
* (2023) SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities

## Code-to-Speech

## Audio-to-Text

* SALMONN: Speech Audio Language Music Open Neural Network

## Speech-to-Text

* SALMONN: Speech Audio Language Music Open Neural Network

## Evaluation

* (2023) EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis
* (2022) SpeechLMScore: Evaluating speech generation using speech language model

## Voice Conversion

* (2023) HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation by Hierarchical Variational Inference for Zero-shot Speech Synthesis

[[paper](https://arxiv.org/abs/2309.11977)] [[demo](https://sh-lee-prml.github.io/HierSpeechpp-demo/)] [[code](https://github.com/sh-lee-prml/HierSpeechpp)]

* (2023) Towards General-Purpose Text-Instruction-Guided Voice Conversion
    [[paper](https://arxiv.org/abs/2309.14324)] [[demo](https://text-guided-vc.github.io/asru2023_demo/)]

* (2023) PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Prompts
    
    [[paper](https://arxiv.org/abs/2309.09262)] [[demo](https://yaoxunji.github.io/prompt_vc/)]

## Text-to-Audio

* (2023) Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
    
    [[paper](https://arxiv.org/abs/2305.18474)] [[demo](https://make-an-audio-2.github.io/)]

* (2023) Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model
    
    [[paper](https://arxiv.org/abs/2304.13731)] [[demo](https://tango-web.github.io/)] [[github](https://github.com/declare-lab/tango)]

* (2023) AudioLDM: Text-to-Audio Generation with Latent Diffusion Models
    
    [[paper](https://arxiv.org/abs/2301.12503)] [[demo](https://audioldm.github.io/)] [[github](https://github.com/haoheliu/AudioLDM)]

* (2023) Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models
    
    [[paper](https://arxiv.org/abs/2301.12661)] [[demo](https://text-to-audio.github.io/)] [[github](https://github.com/Text-to-Audio/Make-An-Audio)]

* (2022) AudioGen: Textually Guided Audio Generation
    
    [[paper](https://arxiv.org/abs/2209.15352)] [[demo](https://felixkreuk.github.io/audiogen/)]

## Text-to-Music

* (2023) Simple and Controllable Music Generation
    
    [[paper](https://arxiv.org/abs/2306.05284)] [[demo](https://ai.honu.io/papers/musicgen/)] [[github](https://github.com/facebookresearch/audiocraft)]

## Text-to-Speech