Sound-of-Pixels

KO | EN

gitlite — search

#python #hacktoberfest #react #ai #llm #android #rust #deep-learning #java #windows #cli #security #pytorch #mcp

Sound-of-Pixels

(★ 393)

Codebase for ECCV18 "The Sound of Pixels"

File Explorer

Download Latest Version (.zip)

- train_dummy.csv
- val_dummy.csv
- __init__.py
- base.py
- music.py
- video_transforms.py
- __init__.py
- audio_net.py
- criterion.py
- synthesizer_net.py
- vision_net.py
- create_index_files.py
- download_trained_model.sh
- eval_MUSIC.sh
- train_MUSIC.sh
.gitignore
arguments.py
LICENSE
main.py
README.md
teaser.png
utils.py
viz.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Sound-of-Pixels
Codebase for ECCV18 "The Sound of Pixels".

*This repository is under construction, but the core parts are already there.

## Environment
The code is developed under the following configurations.
- Hardware: 1-4 GPUs (change ```[--num_gpus NUM_GPUS]``` accordingly)
- Software: Ubuntu 16.04.3 LTS, ***CUDA>=8.0, Python>=3.5, PyTorch>=0.4.0***

## Training
1. Prepare video dataset.

a. Download MUSIC dataset from: https://github.com/roudimit/MUSIC_dataset
    
    b. Download videos.

2. Preprocess videos. You can do it in your own way as long as the index files are similar.

b. Make training/validation index files by running:
    ```
    python scripts/create_index_files.py
    ```
    It will create index files ```train.csv```/```val.csv``` with the following format:
    ```
    ./data/audio/acoustic_guitar/M3dekVSwNjY.mp3,./data/frames/acoustic_guitar/M3dekVSwNjY.mp4,1580
    ./data/audio/trumpet/STKXyBGSGyE.mp3,./data/frames/trumpet/STKXyBGSGyE.mp4,493
    ```
    For each row, it stores the information: ```AUDIO_PATH,FRAMES_PATH,NUMBER_FRAMES```

3. Train the default model.
```bash
./scripts/train_MUSIC.sh
```

5. During training, visualizations are saved in HTML format under ```ckpt/MODEL_ID/visualization/```.

## Evaluation
0. (Optional) Download our trained model weights for evaluation.
```bash
./scripts/download_trained_model.sh
```

1. Evaluate the trained model performance.
```bash
./scripts/eval_MUSIC.sh
```

## Reference
If you use the code or dataset from the project, please cite:
```bibtex
    @InProceedings{Zhao_2018_ECCV,
        author = {Zhao, Hang and Gan, Chuang and Rouditchenko, Andrew and Vondrick, Carl and McDermott, Josh and Torralba, Antonio},
        title = {The Sound of Pixels},
        booktitle = {The European Conference on Computer Vision (ECCV)},
        month = {September},
        year = {2018}
    }
```

Sound-of-Pixels

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Sound-of-Pixels

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README