mosel

KO | EN

gitlite — search

#golang #go #android #machine-learning #deep-learning #docker #macos #nodejs #windows #vue #security #ios #mcp #open-source

(★ 166)

Collection of Open Source Speech Data

File Explorer

Download Latest Version (.zip)

- issue_template.md
- pull_request_template.md
LICENSE
mosel-logo-white.png
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages
<p align="center" width="100%">
<a href="http://arxiv.org/abs/2410.01036"><img src="https://img.shields.io/badge/arXiv-2410.01036-b31b1b.svg"></a>
</p>

## :grapes: Open-Source Compliant Speech Dataset List

<table>
  <thead>
    <tr>
      <th><strong>Name</strong></th>
      <th><strong>License</strong></th>
      <th><strong>Hours</strong></th>
      <th><strong>Languages</strong></th>
      <th><strong>Label</strong></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><a href="https://commonvoice.mozilla.org/en/datasets">CommonVoice</a></td>
      <td>CC 0</td>
      <td>6,732</td>
      <td>bg, cs, da, nl, en, et, fi, fr, de, el, hu, ga, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://github.com/facebookresearch/covost">CoVoST2</a></td>
      <td>CC 0</td>
      <td>687</td>
      <td>en, fr, it, es, pt, et, nl, sv, lv, sl</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://github.com/Kyubyong/css10">CSS10</a></td>
      <td>Public Domain</td>
      <td>99</td>
      <td>nl, fi, fr, de, el, hu, es</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://ips-lmu.github.io/EMU.html">EMU</a></td>
      <td>CC BY 3.0</td>
      <td>56</td>
      <td>pl</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://clarin-pl.eu/dspace/handle/11321/821">EU Parliament</a></td>
      <td>CC BY 4.0</td>
      <td>32</td>
      <td>pl</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://huggingface.co/datasets/google/fleurs">FLEURS</a></td>
      <td>CC BY 4.0</td>
      <td>215</td>
      <td>bg, cs, da, nl, en, et, fi, fr, de, el, hu, ga, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://lindat.cz/repository/xmlui/handle/11234/1-3126">Large Corpus of Czech Parliament Plenary Hearings</a></td>
      <td>CC BY 4.0</td>
      <td>444</td>
      <td>cs</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://github.com/facebookresearch/libri-light">LibriLight</a></td>
      <td>Public Domain</td>
      <td>57,706</td>
      <td>en</td>
      <td>&#x274C;</td>
    </tr>
    <tr>
      <td><a href="https://www.openslr.org/60/">LibriTTS</a></td>
      <td>CC BY 4.0</td>
      <td>585</td>
      <td>en</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://www.openslr.org/12">LibriSpeech</a></td>
      <td>CC BY 4.0</td>
      <td>360</td>
      <td>en</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://www.cl.uni-heidelberg.de/statnlpgroup/librivoxdeen/">LibriVoxDeEn</a></td>
      <td>Public Domain</td>
      <td>547</td>
      <td>de</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://github.com/czyzi0/the-mc-speech-dataset">MC Speech</a></td>
      <td>CC 0</td>
      <td>22</td>
      <td>pl</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://www.openslr.org/94/">Multilingual LibriSpeech</a></td>
      <td>CC BY 4.0</td>
      <td>50,687</td>
      <td>nl, en, fr, de, it, pl, pt, es</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://datashare.ed.ac.uk/handle/10283/2353">SIWIS</a></td>
      <td>CC BY 4.0</td>
      <td>11</td>
      <td>fr</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="http://download.tensorflow.org/data/speech_commands_v0.02.tar.gz">Speech Commands</a></td>
      <td>CC BY 4.0</td>
      <td>18</td>
      <td>en</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://datashare.ed.ac.uk/handle/10283/3443">VCTK</a></td>
      <td>CC BY 4.0</td>
      <td>44</td>
      <td>en</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td rowspan="2"><a href="https://github.com/facebookresearch/voxpopuli">VoxPopuli</a></td>
      <td rowspan="2">CC 0</td>
      <td>383,500</td>
      <td>bg, hr, cs, da, nl, en, et, fi, fr, de, el, hu, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv</td>
      <td>&#x274C;</td>
    </tr>
    <tr>
      <td>1,791</td>
      <td>hr, cs, nl, en, et, fu, fr, de, hu, it, lt, pl, ro, sk, sl, es</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td rowspan="2"><a href="https://huggingface.co/datasets/PleIAs/YouTube-Commons">YouTube-Commons</a></td>
      <td rowspan="2">CC BY 4.0</td>
      <td>3,261</td>
      <td>bg, cs, nl, en, et, fr, de, el, hu, it, pl, pt, ro, es</td>
      <td>&#x274C;</td>
    </tr>
    <tr>
      <td>443,396</td>
      <td>bg, cs, nl, en, et, fi, fr, de, el, hu, it, lv, lt, pl, pt, ro, es, sv</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://huggingface.co/datasets/FBK-MT/mosel">MOSEL :grapes: <a/></td>
      <td>CC BY 4.0</td>
      <td>441,206</td>
      <td>bg, hr, cs, da, nl, en, et, fi, fr, de, el, hu, it, lv, lt, mt, pl, pt, ro, sk, sl, es, sv</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td colspan="5" style="text-align:center; font-weight:bold;">
        Datasets added after MOSEL release
      </td>
  </tr>
    <tr>
      <td><a href="https://huggingface.co/datasets/espnet/yodas">Yodas<a/></td>
      <td>CC BY 3.0</td>
      <td>369,510</td>
      <td>149</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://huggingface.co/datasets/speechbrain/LoquaciousSet">LoquaciousSet<a/></td>
      <td>CC BY 3.0/4.0</td>
      <td>25,000</td>
      <td>en</td>
      <td>&#x2705;</td>
    </tr>
    <tr>
      <td><a href="https://huggingface.co/datasets/nvidia/Granary">Granary :corn: <a/></td>
      <td>CC BY 3.0/4.0</td>
      <td>~1M</td>
      <td>bg, cs, da, de, el, en, es, et, fi, fr, hr, hu, it, lt, lv, mt, nl, pl, pt, ro, sk, sl, sv, uk, ru</td>
      <td>&#x2705;</td>
    </tr>
  </tbody>
</table>

For the languages, [two-letter ISO 639 codes are used](https://en.wikipedia.org/wiki/List_of_ISO_639_language_codes).

## :warning: Contribute and Report Issues
If you want to add an open-source compliant dataset to the list, please fill a [Pull Request](compare). If you want to report an issue about existing content, please use the issues section.

## :checkered_flag: Citation
If you use MOSEL dataset, please cite:
```bibtex
@inproceedings{mosel,
  title = {{MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languages}},
  author = {Marco Gaido and Sara Papi and Luisa Bentivogli and Alessio Brutti and Mauro Cettolo and Roberto Gretter and Marco Matassoni and Mohamed Nabihand Matteo Negri},
  booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
  month = nov,
  year = "2024",
  address = "Miami, United States",
  publisher = "Association for Computational Linguistics",
}
```

mosel

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

mosel

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README