SVHighlights

KO | EN

gitlite — search

#video #ai-agents #mcp #c #api #shell #gemini #security #gpt #generative-ai

(★ 9)

[KDD 2026 Datasets & Benchmarks Track] SVHighlights: a benchmark for highlight detection in extremely long sports videos

File Explorer

ZIP Download

- alignment_pipeline.png
- full_distribution.png
- highlight_distribution.png
- method.png
- teaser.png
- align.py
- eval_clip_similarity.py
- eval_matching_quality.py
- filter_frames.py
- labeling.py
- README.md
- trim_video.py
- .nojekyll
- index.html
- - llm.py
  - vlm.py
- command.txt
- dataset.py
- main.py
- parse.py
- README.md
- segment.py
- segment_captioning.py
- shot_boundary.py
- transcribe.py
- util.py
- volume.py
- volume_minmax.py
.gitignore
eval.py
LICENSE
README.md
requirements.txt

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

# Project Badges

arXiv Visit
Project Page Visit
🤗 Dataset Visit
License: Apache 2.0 Visit
Conference Visit
Python Visit

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Donggyu Lee**\* &nbsp;·&nbsp; **Youngbin Ki**\* &nbsp;·&nbsp; **Jeonghun Kang** &nbsp;·&nbsp; **Taehwan Kim**

Ulsan National Institute of Science and Technology (UNIST)

📍 **KDD 2026** &nbsp;·&nbsp; Datasets & Benchmarks Track &nbsp;·&nbsp; <sub>*equal contribution</sub>

<em>SVHighlights videos average <b>2 hours</b> — roughly 30–60× longer than existing highlight detection benchmarks.</em>

</div>

---

## 📝 Abstract

While highlight detection for long-form videos is of great practical
importance, most existing methods remain limited to short-form content, largely
due to the absence of a suitable benchmark. To bridge this gap, we introduce
SVHighlights, to the best of our knowledge, the first benchmark for highlight
detection in extremely long sports videos, each exceeding one hour in duration,
across multiple sports categories. SVHighlights is constructed from pairs of
full-length sports videos and their corresponding official highlight videos
using a dataset generation pipeline, enabling scalable and cost-effective
label generation without conventional per-clip saliency annotation. The
benchmark comprises 320 videos spanning a wide range of sports, with an average
duration of 2.00 hours and a total of 640.18 hours, substantially exceeding
previous highlight detection datasets. Beyond the lack of benchmarks, existing
methods also face fundamental challenges on long videos: models trained on
short clips of only a few minutes fail to generalize to hour-long content, and
their clip-level scoring lacks the broader context needed to identify
highlights in long-form videos. To address these challenges and provide a
strong baseline for SVHighlights, we present TF-SELECTOR, a training-free
segment-based approach that divides each video into context-aware segments by
merging adjacent shots sharing the same semantic content, and predicts
segment-level saliency scores using a large language model (LLM) with
multimodal inputs including visual captions, transcripts, and audio volume.
Extensive experiments demonstrate that TF-SELECTOR achieves superior
performance across most evaluation metrics compared to Video Temporal Grounding
(VTG)-tuned baselines, with improvements of +3.12 in HIT@1, +4.06 in HIT@K, and
+2.95 in IoU. These results establish SVHighlights as a challenging testbed for
long-form highlight detection and demonstrate that a simple segment-based
strategy can effectively scale to hour-long videos.

## 🔥 Update

- **[2026.06.05]** 📄 The paper is now available on [arXiv](https://arxiv.org/abs/2606.06926).
- **[2026.05.25]** 🤖 Released the TF-SELECTOR model code.
- **[2026.05.23]** 🚀 Released the SVHighlights dataset (annotations & features) on Hugging Face and the preprocessing code.
- **[2026.05.17]** 🎉 SVHighlights was accepted to **KDD 2026** (Datasets & Benchmarks Track).

## 📺 SVHighlights

SVHighlights pairs **320** full-length sports broadcasts with their official
highlight videos, spanning **8 sports** — american football, baseball,
basketball, ice hockey, racing, rugby, soccer, and volleyball (40 videos each).

### Highlight alignment

Rather than relying on manual per-clip annotation, we align each official
highlight video to its full-length broadcast. Every 1-second highlight clip is
matched to the most similar full-video frame via a pixel-level **PSNR** score, a
post-processing step enforces temporal consistency, and a filtering step removes
mismatched frames.

### Qualitative comparison

Each video places the **official highlight video** (left) next to the
**full-video frames aligned by our pipeline** (right) — for every highlight
second, a 1-second clip cut from the full broadcast at the frame our highlight
alignment matched.

<table width="100%">
<tr>
<td width="50%" align="center"><b>🏈 American Football</b><br><video src="https://github.com/user-attachments/assets/00203f1c-07cc-4aae-86ca-b27d0112410f" controls muted width="100%"></video></td>
<td width="50%" align="center"><b>⚾ Baseball</b><br><video src="https://github.com/user-attachments/assets/79a085b2-d753-4605-a615-c23a1277b4cd" controls muted width="100%"></video></td>
</tr>
<tr>
<td width="50%" align="center"><b>🏀 Basketball</b><br><video src="https://github.com/user-attachments/assets/20e98ad3-b1b7-4831-b9b4-ab29082ed48e" controls muted width="100%"></video></td>
<td width="50%" align="center"><b>🏒 Ice Hockey</b><br><video src="https://github.com/user-attachments/assets/112c64e8-f92a-4211-9e83-65a9a3bb7ec8" controls muted width="100%"></video></td>
</tr>
<tr>
<td width="50%" align="center"><b>🏁 Racing</b><br><video src="https://github.com/user-attachments/assets/def80680-9ff7-4811-8802-57e52a7b6d86" controls muted width="100%"></video></td>
<td width="50%" align="center"><b>🏉 Rugby</b><br><video src="https://github.com/user-attachments/assets/d43b7302-f21b-4502-b0f6-fc8c14fd93fa" controls muted width="100%"></video></td>
</tr>
<tr>
<td width="50%" align="center"><b>⚽ Soccer</b><br><video src="https://github.com/user-attachments/assets/963d97c5-db41-4a6f-b86e-f8d1e1336416" controls muted width="100%"></video></td>
<td width="50%" align="center"><b>🏐 Volleyball</b><br><video src="https://github.com/user-attachments/assets/dd024b45-91b0-4d98-a552-fab94d3716f6" controls muted width="100%"></video></td>
</tr>
</table>

## 🤖 TF-SELECTOR

**TF-SELECTOR** (**T**raining-**F**ree **S**egment-based **E**xtremely **L**ong
video highlight det**ECTOR**) is a training-free baseline with three stages:

1. **Context-aware segmentation** — shots are detected with a shot-boundary
   detector and adjacent shots that share the same spoken sentence (from an ASR
   model) are merged into coherent segments.
2. **Segment captioning** — a vision–language model (VLM) generates a textual
   description for each segment.
3. **Segment-level scoring** — a large language model (LLM) predicts a saliency
   score per segment from the caption, transcript, and audio volume; the score
   is then assigned to every clip in the segment.

## 🗂️ Repository Structure

The code is split into two parts:

- **[`benchmark/`](benchmark/)** — **SVHighlights dataset construction** (paper
  Section 3): video trimming, highlight alignment, filtering, and label
  generation, plus the alignment-quality evaluation tools.
- **[`tf_selector/`](tf_selector/)** — **TF-SELECTOR baseline** (paper Section 4):
  preprocessing (shot-boundary detection, speech recognition, context-aware
  segmentation, audio-volume extraction) and inference (segment captioning
  with a VLM, LLM-based saliency scoring, and prediction parsing into the
  per-clip saliency-score format consumed by `eval.py`).

See each directory's `README.md` for the full pipeline and usage instructions.
Highlight-detection predictions are scored with **[`eval.py`](eval.py)** — see
[Evaluation](#-evaluation).

## ⬇️ Download

We do **not** redistribute the original videos. In line with the dataset's
terms of use, we release only the video URLs (`video_list.csv`), the extracted
features, and the annotation labels, as a Hugging Face dataset:

> 🤗 **<https://huggingface.co/datasets/idong1004/SVHighlights>**

Both the directory-style annotations and the features are released as tarballs
so they download as a few large Xet objects rather than thousands of small
files — this avoids Hugging Face's anonymous API rate limit on snapshot
downloads. The four `annotations/*.tar` cover all 8 sports together; features
are split per sport (`features/<sport>.tar`).

```bash
# 1) Download the dataset (use --max-workers 1 to stay under the HF rate limit)
huggingface-cli download idong1004/SVHighlights \
  --repo-type dataset --local-dir ./data --max-workers 1

# 2) Extract the annotation tarballs in place
for a in alignment segments whisper shots; do
  tar -xf data/annotations/${a}.tar -C data/annotations/ && rm data/annotations/${a}.tar
done

# 3) Extract the per-sport feature tarballs in place
for s in american_football baseball basketball ice_hockey race rugby soccer volleyball; do
  tar -xf data/features/${s}.tar -C data/features/ && rm data/features/${s}.tar
done
```

**Data layout** (after extraction)

```
data/
├── metadata/
│   └── video_list.csv       # video URLs + per-video trim boundaries
├── annotations/
│   ├── alignment/            # benchmark/align.py            (from alignment.tar)
│   ├── all_filtered_frame_idx.json   # benchmark/filter_frames.py + manual filtering
│   ├── label.json            # benchmark/labeling.py
│   ├── shots/                # tf_selector/shot_boundary.py  (from shots.tar)
│   ├── whisper/              # tf_selector/transcribe.py     (from whisper.tar)
│   ├── segments/             # tf_selector/segment.py        (from segments.tar)
│   ├── segment_caption.json  # tf_selector/segment_captioning.py (VLM)
│   ├── volume.json           # tf_selector/volume.py
│   └── minmax_volume.json    # tf_selector/volume_minmax.py
└── features/
    └── <sport>/                 # extracted from features/<sport>.tar
        ├── vid_clip/            # HERO video CLIP features
        ├── vid_slowfast/        # HERO video SlowFast features
        ├── txt_clip/            # HERO query CLIP features
        └── aud_pann/            # PANN audio features
```

Videos are named `<sport>_<idx>.mp4`, where `<sport>` is one of:
`american_football`, `baseball`, `basketball`, `ice_hockey`, `race`, `rugby`,
`soccer`, `volleyball`.

**Annotations**

| Artifact | Produced by | Description |
|---|---|---|
| `annotations/alignment/` | `benchmark/align.py` | Per-video raw PSNR alignment between highlight clips and full-video frames. |
| `annotations/all_filtered_frame_idx.json` | `benchmark/filter_frames.py` + manual filtering | Final aligned frame index per highlight clip. |
| `annotations/label.json` | `benchmark/labeling.py` | Per-clip `{0, 1}` highlight labels (one score per 2-second clip) — the evaluation ground truth. |
| `annotations/shots/` | `tf_selector/shot_boundary.py` | Per-video shot-boundary detection (start/end frame index per shot). |
| `annotations/whisper/` | `tf_selector/transcribe.py` | Word-level WhisperX transcripts. |
| `annotations/segments/` | `tf_selector/segment.py` | Context-aware segments (2-minute maximum length). |
| `annotations/segment_caption.json` | `tf_selector/segment_captioning.py` | Per-segment captions from the InternVL2_5-8B VLM (input to the LLM scorer). |
| `annotations/volume.json` | `tf_selector/volume.py` | Per-clip audio loudness (dBFS) for every video. |
| `annotations/minmax_volume.json` | `tf_selector/volume_minmax.py` | Min-max normalized per-clip audio volume in `[0, 1]`. |

**Features** (QVHighlights-style)

| Feature | Directory | Extractor |
|---|---|---|
| Video — CLIP | `features/<sport>/vid_clip/` | HERO Video Feature Extractor |
| Video — SlowFast | `features/<sport>/vid_slowfast/` | HERO Video Feature Extractor |
| Query — CLIP | `features/<sport>/txt_clip/` | HERO Video Feature Extractor |
| Audio — PANN | `features/<sport>/aud_pann/` | audioset_tagging_cnn |

The query feature is the CLIP text embedding of the fixed query
*"Highlight of this {sport} video"*. Video and query features were extracted
with the [HERO Video Feature Extractor](https://github.com/linjieli222/HERO_Video_Feature_Extractor);
audio features with [PANN / audioset_tagging_cnn](https://github.com/qiuqiangkong/audioset_tagging_cnn).

## 🎮 Environment Setup

`benchmark/` and `tf_selector/` run in a **single conda environment** (named
`svhighlights` here, Python 3.9).

```bash
conda create -n svhighlights python=3.9 -y
conda activate svhighlights

# ffmpeg / ffprobe — used by almost every script
conda install -c conda-forge ffmpeg -y

# Python dependencies (covers benchmark/ and tf_selector/ — preprocessing + model inference)
pip install -r requirements.txt

# PyTorch with CUDA 12.1 — required by tf_selector/ model inference
pip install torch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 --index-url https://download.pytorch.org/whl/cu121

# External packages installed from source:
pip install git+https://github.com/openai/CLIP.git        # benchmark/eval_clip_similarity.py
git clone https://github.com/soCzech/TransNetV2           # tf_selector/shot_boundary.py
```

For GPU use, install the PyTorch build that matches your CUDA version (the
pipeline was run with PyTorch 2.3.1 / CUDA 12.1). The two external packages
above are each needed by only one script — skip either you do not plan to run.

Tested with Python 3.9, PyTorch 2.3.1 (CUDA 12.1), and ffmpeg 4.2.9.

> **Note** — `tf_selector/main.py` and `tf_selector/segment_captioning.py` log
> in to the Hugging Face Hub at startup via
> `login(token=os.environ.get('HF_TOKEN'))`; export your access token before
> running, e.g. `export HF_TOKEN=hf_...`. In `tf_selector/parse.py`, replace
> the `path/to/frame/{vid}` placeholder with the directory that contains your
> per-video frames.

## 🚀 Usage

The code is split into two pipelines — see each directory for the full
step-by-step commands and CLI arguments.

- **Dataset construction** — [`benchmark/`](benchmark/):
  video trimming, highlight alignment, PSNR filtering, and per-clip label
  generation (paper Section 3).
- **TF-SELECTOR baseline** — [`tf_selector/`](tf_selector/):
  preprocessing (shot boundaries, ASR, context-aware segmentation, audio
  volume, VLM segment captioning) and training-free inference (LLM saliency
  scoring + prediction parsing) (paper Section 4). All preprocessing outputs
  are released under `annotations/`, so these steps are optional.

Every script accepts `--sports` to restrict processing to a subset of sports
and `-h` for the full argument list.

## 📊 Evaluation

To evaluate a highlight-detection model on SVHighlights, produce a **prediction
file** with a per-clip saliency score for every video, then run `eval.py`.

**Prediction file** — a JSON list with one entry per video:

```json
[
  {
    "vid": "american_football_1",
    "pred_saliency_scores": [0.895, 0.880, 0.863, ...]
  },
  ...
]
```

- `vid` — video id (`<sport>_<idx>`)
- `pred_saliency_scores` — one predicted score per **2-second clip**, in clip
  order (each video is divided into non-overlapping 2-second clips)

**Ground-truth file** — `annotations/label.json` from the dataset: a JSON list
of `{"vid", "saliency_scores"}`, where `saliency_scores` are the per-clip
`{0, 1}` highlight labels.

**Run:**

```bash
python eval.py \
    --submission_path path/to/predictions.json \
    --gt_path        annotations/label.json \
    --save_path      path/to/results.json
```

`eval.py` reports four metrics, computed per sport and over `all`:

| Metric | Description |
|---|---|
| `HL-mAP` | Clip-level mean Average Precision |
| `HL-Hit1` | Whether the top-scored clip is a ground-truth highlight |
| `HL-Hitk` | Hit rate among the top-K clips (K = number of GT highlight clips) |
| `HL-IoU` | Temporal overlap between the top-K predictions and GT highlights |

## 📩 Request Frame Data

We do **not** redistribute the original sports broadcasts. For research that
needs the raw visual content, we share per-video frames sampled at **1 frame
per second (1 fps)** on request, subject to a research-only, no-redistribution
agreement.

> 📩 **[Request access to the frame data](https://docs.google.com/forms/d/e/1FAIpQLSctZ6XMCND3HJqR8huV6k7N4phbSRTIu7vQ0cT6hPk2QHjkOQ/viewform)**

Approved requests receive a Google Drive link with a 7-day access window. For
any issue with the form, please email `leedongkyu2019@unist.ac.kr` or open an
issue on this repository.

## 📚 Citation

If you find SVHighlights useful in your research, please cite:

```bibtex
@inproceedings{Lee_2026,
  title={SVHighlights: Towards Extremely Long Sport Video Highlight Detection},
  url={http://dx.doi.org/10.1145/3770855.3817564},
  DOI={10.1145/3770855.3817564},
  booktitle={Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2},
  publisher={ACM},
  author={Lee, Donggyu and Ki, Youngbin and Kang, Jeonghun and Kim, Taehwan},
  year={2026},
  month=Aug,
  pages={9254–9265}
}
```

SVHighlights

File Explorer

# Use via CDN

# Project Badges

Was this content helpful?

Recent Feedback

Download README