ics-av-deepfake - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #react #ai #golang #android #machine-learning #rust #deep-learning #docker #nodejs #kubernetes #ios #openai

ics-av-deepfake

(★ 9)

[ICCV'25] Intra-modal and Cross-modal Synchronization for Audio-visual Deepfake Detection and Temporal Localization

파일 탐색기

최종 버전 다운로드 (.zip)

- - meta_data.json
  - test_data.json
  - train_data.json
  - val_data.json
  - voxceleb_data.json
- __init__.py
- collater.py
- dfdc_classification.py
- dfdc_inference.py
- fakeavceleb_classification.py
- fakeavceleb_inference.py
- fakeavceleb_preprocess.py
- kodf_classification.py
- kodf_inference.py
- lavdf_inference.py
- lavdf_regression_bmn.py
- lrs2.py
- voxceleb.py
- audio_encoder.py
- boundary_module.py
- boundary_module_plus.py
- class_model.py
- pretrain_model.py
- reg_model_bmn_plus.py
- resnet_avhubert.py
- sync_model.py
- transformer.py
- video_encoder.py
- vit_encoder.py
- - nms_cpu.cpp
- nms.py
- setup.py
.gitignore
create_env.txt
evaluate_classification_pt_lightning.py
evaluate_regression_bmn_pt_lightning.py
metrics.py
pretrain_inference_pt_lightining.py
pretrain_pytorch_lightning.py
README.md
regression_loss.py
train_classification_pt_lightning.py
train_regression_pt_lightning.py
utils.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/AshutoshAnshul/ics-av-deepfake

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd ics-av-deepfake

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install .

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ics-av-deepfake

**[ICCV 2025] Intra-modal and Cross-modal Synchronization for Audio-visual Deepfake Detection and Temporal Localization**

Official implementation of the ICCV 2025 paper:

> **Intra-modal and Cross-modal Synchronization for Audio-visual Deepfake Detection and Temporal Localization**

---

## Environment Setup

We recommend using **Conda - miniconda3**.

### Using Conda (Recommended)

Simply follow the instructions in:

```text
create_env.txt
```

Execute the commands **line by line**, and the complete environment will be created automatically.

### Without Conda

If you prefer another environment manager (e.g., `venv`), `create_env.txt` also serves as a complete list of the required libraries and package versions. Install the listed dependencies manually before running the code.

---

## Dataset Preparation

### 1. VoxCeleb2

Download the official **VoxCeleb2** dataset.

The extracted dataset should contain the following folders:

```text
VoxCeleb2/
├── dev/
└── test/
```

For our pretraining protocol, add the following metadata files:

```text
VoxCeleb2/
├── dev/
│   └── filtered_file_ffmpeg_next.pkl
└── test/
    └── filtered_file_ffmpeg_next.pkl
```

These files are provided in this repository under:

```text
data_files/
└── VoxCeleb/
    ├── dev/
    │   └── filtered_file_ffmpeg_next.pkl
    └── test/
        └── filtered_file_ffmpeg_next.pkl
```

Copy each file into its corresponding `dev` and `test` directory of the downloaded VoxCeleb2 dataset.

These metadata files contain the filtered subset used for the VoxCeleb2 pretraining experiments.

---

### 2. FakeAVCeleb

Download the official **FakeAVCeleb** dataset.

Copy **all JSON files** provided in this repository into the **root directory** of the FakeAVCeleb dataset.

These JSON files include:

* Dataset splits used in our experiments.
* Additional VoxCeleb samples augmented into FakeAVCeleb for training.

After copying, the root directory of FakeAVCeleb should contain both the original dataset and the provided JSON annotation files.

---

## Pretrained Checkpoints

Pretrained checkpoints are available on **[OneDrive](https://entuedu-my.sharepoint.com/:f:/g/personal/ashutosh007_e_ntu_edu_sg/IgA2y9zgbicyQKMSfu67CL2vAedLrsCq1wIdMtYg91yaEFo?e=NHtbie)**.
The checkpoint directory follows the structure below:

```text
checkpoints/
└── pretrained-on-{dataset_name}/
    └── {variant_name}_{dataset_name}/
        ├── ckpts/
        └── class_ckpts_{leave_section}/
```

where:

* `pretrained-on-{dataset_name}` denotes the dataset used for self-supervised pretraining.
* `{variant_name}` refers to the model variant (e.g., `decoder` or `sparse_encoder`).
* `ckpts/` contains the self-supervised pretraining checkpoints.
* `class_ckpts_{leave_section}/` contains the downstream deepfake classification checkpoints.

The classification checkpoints are organized according to the corresponding `leave_section` setting used during training.

If `leave_section = none`, the checkpoint corresponds to the standard intra-dataset training setup without leaving out any manipulation category.

---

## Self-Supervised Pretraining

The self-supervised pretraining stage can be launched using:

```bash
python pretrain_pytorch_lightning.py \
    --dataset_path {dataset_root} \
    --dataset_padding_num_frames 500 \
    --dataset_train_split dev \
    --dataset_eval_split test \
    --model_v_encoder_variant resnet \
    --model_a_encoder_variant vit_t \
    --model_sync_loss gaussian \
    --modal_consistency Sync \
    --model_depth 3 \
    --model_feature_in 256 \
    --model_temporal_dim 500 \
    --model_select_tgt_layer combiner \
    --train_batch_size {batch_size} \
    --gradient_accumulation_steps {gradient_accumulation_step} \
    --dataset_name {voxceleb OR lrs2} \
    --sync_model_type {decoder OR sparse_encoder} \
    --resume_checkpoint_path {path_to_checkpoint} \
    --n_gpu {num_gpus}
```

### Argument Description

* **`dataset_path`**: Root directory of the downloaded dataset.

* For example, if VoxCeleb2 is located at:

```text
    /abc/def/VoxCeleb2
    ```

then

```text
    dataset_path=/abc/def/VoxCeleb2
    ```

The same convention applies to the LRS2 dataset.

* **`dataset_name`**: Dataset used for pretraining (`voxceleb` or `lrs2`).

* **`train_batch_size`**: Batch size per GPU.

* **`gradient_accumulation_steps`**: Number of gradient accumulation steps. The effective batch size is equal to:

```text
  train_batch_size × gradient_accumulation_steps × n_gpu
  ```

* **`sync_model_type`**: Pretraining model variant. Supported options are:

* `decoder`
  * `sparse_encoder`

* **`resume_checkpoint_path`**: Path to a previously saved pretraining checkpoint. Leave this empty to train from scratch.

* **`n_gpu`**: Number of GPUs used for distributed training.

---

## Feature Extraction

After self-supervised pretraining, the pretrained model is used to extract audio-visual features for downstream deepfake classification. Features can be extracted for either the **FakeAVCeleb** or **Lav-DF** dataset using:

```bash
python pretrain_inference_pt_lightining.py \
    --dataset_path {dataset_root} \
    --dataset_padding_num_frames 500 \
    --eval_batch_size {batch_size} \
    --dataset_name {dataset_name} \
    --sync_model_type decoder \
    --model_checkpoint_path {pretrained_model_checkpoint} \
    --n_gpu {num_gpus}
```

### Argument Description

* **`dataset_path`**:

* **Lav-DF:** Specify the root directory of the downloaded dataset, following the same convention as VoxCeleb2 during pretraining.
  * **FakeAVCeleb:** Specify the **parent directory** containing the `FakeAVCeleb` folder.

For example, if the dataset is located at

```text
    /abc/def/FakeAVCeleb
    ```

then

```text
    dataset_path=/abc/def
    ```

This is because the FakeAVCeleb metadata internally resolves the dataset location. Apologies for the inconvenience!

* **`dataset_name`**: Dataset on which to extract features (e.g., `fakeavceleb` or `lavdf`).

* **`eval_batch_size`**: Batch size used during feature extraction.

* **`model_checkpoint_path`**: Path to the pretrained self-supervised checkpoint (`ckpts`) obtained from the pretraining stage.

* **`n_gpu`**: Number of GPUs used for inference.

### Output Features

The extracted features are automatically written to

```text
<dataset_root_directory>/decoder_voxceleb_feat
```

The output directory is currently hard-coded in the dataset loaders:

* `./dataset/fakeavceleb_inference.py` (around line 78)
* `./dataset/lavdf_inference.py` (around line 64)

If you would like the extracted features to be stored in a different location, simply modify the corresponding output path in these files before running feature extraction.

---

## Deepfake Classification Training

After extracting the pretrained audio-visual features, the downstream deepfake classifier can be trained using:

```bash
python train_classification_pt_lightning.py \
    --dataset_path {dataset_root} \
    --dataset_padding_num_frames 500 \
    --dataset_train_split train \
    --dataset_val_split val \
    --dataset_test_split test \
    --model_depth 3 \
    --model_feature_in 256 \
    --train_batch_size {batch_size} \
    --leave_section {leave_section} \
    --dataset_name {pretraining_dataset_name} \
    --sync_model_type {decoder OR sparse_encoder} \
    --train_dataset_name {training_dataset_name} \
    --n_gpu {num_gpus}
```

### Argument Description

* **`dataset_path`**:

* For **FakeAVCeleb**, this follows the same convention as feature extraction. If the dataset is located at

```text
    /abc/def/FakeAVCeleb
    ```

then

```text
    dataset_path=/abc/def
    ```

* **`dataset_name`**: The dataset on which the self-supervised model was pretrained (e.g., `voxceleb` or `lrs2`). This is used to locate the corresponding pretrained features.

* **`sync_model_type`**: The pretrained model variant used for feature extraction (`decoder` or `sparse_encoder`).

* **`train_dataset_name`**: Dataset used for downstream classification training (e.g., `fakeavceleb`).

* **`train_batch_size`**: Batch size per GPU.

* **`leave_section`**: Specifies the manipulation category to exclude during training for leave-one-category-out evaluation.

In the paper, we report results for the following leave-out settings:

* `RVFA`
  * `FVFA-FS`
  * `FVFA-WL`
  * `FVFA-GAN`

To train using the complete dataset (i.e., standard intra-dataset training), set `leave_section` to the corresponding **None** option defined in the dataset loader.

The complete list of supported values is defined in `./dataset/fakeavceleb_classification.py` (see the `leave_section` definitions).

* **`n_gpu`**: Number of GPUs used for distributed training.

---

## Deepfake Classification Evaluation

To evaluate a trained deepfake classification model, run:

```bash
python evaluate_classification_pt_lightning.py \
    --dataset_path {dataset_root} \
    --dataset_name fakeavceleb \
    --dataset_padding_num_frames 500 \
    --dataset_train_split train \
    --dataset_val_split val \
    --dataset_test_split test \
    --eval_batch_size {batch_size} \
    --model_checkpoint_path {classification_checkpoint} \
    --n_gpu {num_gpus} \
    --leave_section {leave_section} \
    --pretrain_dataset_name {voxceleb OR lrs2} \
    --sync_model_type {decoder OR sparse_encoder}
```

### Argument Description

* **`dataset_path`**:

* For **FakeAVCeleb**, this follows the same convention as feature extraction and classification training. If the dataset is located at

```text
    /abc/def/FakeAVCeleb
    ```

then

```text
    dataset_path=/abc/def
    ```

* **`dataset_name`**: Dataset used for evaluation (currently `fakeavceleb`).

* **`eval_batch_size`**: Batch size used during evaluation.

* **`model_checkpoint_path`**: Path to the trained deepfake classification checkpoint located in the `class_ckpts` directory.

* **`leave_section`**: The leave-one-category-out setting used during training. This should match the value used to train the classification checkpoint.

* **`pretrain_dataset_name`**: Dataset on which the self-supervised model was pretrained (`voxceleb` or `lrs2`).

* **`sync_model_type`**: Pretrained model variant used for feature extraction and classification (`decoder` or `sparse_encoder`).

* **`n_gpu`**: Number of GPUs used for evaluation.

---

## Temporal Localization Training

Before training or evaluating the temporal localization model, the BMN Non-Maximum Suppression (NMS) utilities must be installed. This only needs to be done **once**.

```bash id="m5iq7w"
cd nms_utils
python setup.py install --user
cd ..
```

After the installation is complete, train the temporal localization model using:

```bash id="8nqjlwm"
python train_regression_pt_lightning.py \
    --dataset_path {dataset_root} \
    --dataset_padding_num_frames 500 \
    --dataset_train_split train \
    --dataset_val_split val \
    --dataset_test_split test \
    --model_depth 6 \
    --model_feature_in 256 \
    --train_batch_size {batch_size} \
    --gradient_accumulation_steps 1 \
    --leave_section None \
    --dataset_name {pretrain_dataset_name} \
    --sync_model_type decoder \
    --train_dataset_name lavdf \
    --include_encoders True \
    --regression_head bmn \
    --n_gpu {num_gpus}
```

### Argument Description

* **`dataset_path`**: Root directory of the Lav-DF dataset, following the same convention used during feature extraction.

* **`dataset_name`**: Dataset on which the self-supervised model was pretrained (`voxceleb` or `lrs2`).

* **`train_dataset_name`**: Downstream temporal localization dataset (`lavdf`).

* **`include_encoders`**: Whether to jointly fine-tune the pretrained encoders during temporal localization training.

* **`regression_head`**: Regression head used for temporal localization. The current implementation supports `bmn`.

* **`train_batch_size`**: Batch size per GPU.

* **`n_gpu`**: Number of GPUs used for distributed training.

---

## Temporal Localization Evaluation

If the BMN NMS utilities have not already been installed, first run:

```bash id="ib9u5w"
cd nms_utils
python setup.py install --user
cd ..
```

Then evaluate a trained temporal localization model using:

```bash id="d3fdb4"
python evaluate_regression_bmn_pt_lightning.py \
    --dataset_path {dataset_root} \
    --dataset_name lavdf \
    --dataset_padding_num_frames 500 \
    --dataset_train_split train \
    --dataset_val_split val \
    --dataset_test_split test \
    --eval_batch_size {batch_size} \
    --leave_section None \
    --regression_head bmn \
    --exp_name_prefix lavdf_inference \
    --n_gpu 1 \
    --model_checkpoint_path {reg_ckpt_path}
```

### Argument Description

* **`dataset_path`**: Root directory of the Lav-DF dataset.

* **`dataset_name`**: Evaluation dataset (`lavdf`).

* **`eval_batch_size`**: Batch size used during evaluation.

* **`regression_head`**: Temporal localization head to evaluate (`bmn`).

* **`exp_name_prefix`**: Prefix used when saving the generated temporal localization predictions.

* **`model_checkpoint_path`**: Path to the trained temporal localization checkpoint.

* **`n_gpu`**: Number of GPUs used for evaluation.

---

## Citation

If you find this repository useful in your research, please consider citing our paper:

```bibtex
@inproceedings{anshul2025intra,
  title={Intra-modal and cross-modal synchronization for audio-visual deepfake detection and temporal localization},
  author={Anshul, Ashutosh and Gopal, Shreyas and Rajan, Deepu and Chng, Eng Siong},
  booktitle={2025 IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={13826--13836},
  year={2025},
  organization={IEEE}
}
```