PlanTalk

KO | EN

gitlite — search

#docker #gpt #svg #github #privacy #nodejs #openai #gemini #video #electron

(★ 9)

No description available.

File Explorer

ZIP Download

- future_plan_tokenizer.yaml
- intenttalk.yaml
- - __init__.py
  - data.py
  - features.py
  - parsers.py
  - preprocessing.py
  - Quaternions.py
  - rotation_tools.py
  - viz_tools.py
  - writers.py
- - __init__.py
  - audio_features.py
  - other_tools.py
  - other_tools_hf.py
  - rotation_conversions.py
  - ted_utlis.py
- __init__.py
- beat2_add_hubert.py
- build_vocab.py
- data_tools.py
- intenttalk_dataloader.py
- intenttalk_preprocess_common.py
- save_test_dataset.py
- save_train_dataset.py
- - __init__.py
  - layer.py
  - skeleton.py
- __init__.py
- backbone.py
- encdec.py
- future_plan_tokenizer.py
- intenttalk.py
- motion_encoder.py
- motion_representation.py
- quantizer.py
- residual_vq.py
- resnet.py
- rvq.py
- - __init__.py
  - adabelief.py
  - adafactor.py
  - adahessian.py
  - adamp.py
  - adamw.py
  - cosine_lr.py
  - lookahead.py
  - nadam.py
  - novograd.py
  - nvnovograd.py
  - plateau_lr.py
  - radam.py
  - rmsprop_tf.py
  - scheduler.py
  - sgdp.py
  - step_lr.py
  - tanh_lr.py
- __init__.py
- loss_factory.py
- optim_factory.py
- scheduler_factory.py
- - bulma-carousel.min.css
  - bulma-slider.min.css
  - bulma.css.map.txt
  - bulma.min.css
  - fontawesome.all.min.css
  - index.css
- - ablation.png
  - favicon.ico
  - results.png
  - teaser.png
- - bulma-carousel.js
  - bulma-carousel.min.js
  - bulma-slider.js
  - bulma-slider.min.js
  - fontawesome.all.min.js
  - index.js
- - TMM2026_suplementary_material.pdf
  - TMM_20260903.pdf
- - cat_2_scott_0_1_1.mp4
  - cat_2_scott_0_2_2.mp4
  - cat_2_scott_0_3_3.mp4
  - cat_2_scott_0_4_4.mp4
  - res_10_kieks_0_4_4.mp4
  - res_12_zhao_0_81_81.mp4
  - res_1_wayne_0_1_1.mp4
  - res_20_li_0_81_81.mp4
  - res_28_tiffnay_0_87_87.mp4
  - res_2_scott_0_73_73.mp4
  - res_3_solomon_0_8_8.mp4
  - res_4_lawrence_0_95_95.mp4
  - res_7_sophie_0_81_81.mp4
- teaser.png
- __init__.py
- audio_to_frame_tokens.py
- config.py
- data_transfer.py
- fast_render.py
- intenttalk_components.py
- joint_layout.py
- logger_tools.py
- media.py
- metric.py
- other_tools.py
- project_paths.py
- rotation_conversions.py
- rvq_distributed.py
.gitignore
.nojekyll
future_plan_tokenizer_trainer.py
generation_trainer.py
index.html
intenttalk_trainer.py
README.md
requirements.txt
train.py

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h2>IntentTalk: Future-Grounded Motion Planning for Co-Speech Gesture Generation</h2>

<p>
<strong>Official implementation of IntentTalk, a future-grounded speech-to-plan-to-motion framework for holistic co-speech gesture generation.</strong>
</p>

</div>

* **[2026-09]** Release training and inference code for IntentTalk.
* **[Coming Soon]** Release pretrained future motion planner and IntentTalk checkpoints.
* **[Coming Soon]** Release evaluation scripts and qualitative examples on BEAT2.

# 💡 Overview

Natural co-speech gestures are not merely reactive responses to the current speech signal.
Humans often organize upcoming gestures before their salient realization, involving
preparation, semantic activation, and coordinated evolution across multiple body regions.

Most existing speech-driven gesture generation methods directly map speech features to
low-level motion sequences, leaving high-level gesture organization and detailed motion
realization entangled.

**IntentTalk** introduces an explicit intermediate **future motion plan** to model upcoming
gesture organization before detailed motion generation.

The framework follows a

**Speech → Future Motion Plan → Holistic Motion**

generation paradigm.

Given speech context, IntentTalk first predicts a discrete future-oriented motion plan that
captures the organization of an upcoming motion window. The predicted plan is subsequently
used to regulate **when** semantic motion should be activated and **how** the planned motion
should be realized across different body regions.

The framework consists of three key components:

1. **Future Motion Planning**
   A discrete motion-plan space is learned from future motion windows. At inference time,
   the corresponding future plan is predicted using speech alone.

2. **Event-Aware Semantic Gating**
   The predicted plan cooperates with speech semantics to determine when semantically
   meaningful motion should be emphasized, reducing delayed or spurious gesture activation.

3. **Hierarchical Plan-Conditioned Modulation**
   The predicted future plan modulates motion generation across different anatomical
   regions, enabling coordinated realization of upper-body, hand, and lower-body motion.

# 🧠 Future-Grounded Motion Planning

Instead of representing only the current motion state, IntentTalk explicitly summarizes
an upcoming motion window

$$
\mathcal{W}_{t}^{+}
$$

into a discrete motion-plan token.

During training, future motion provides supervision for learning the motion-plan space.
At inference time, future motion is unavailable; therefore, a speech-to-plan predictor
estimates the upcoming motion plan directly from speech context.

This design encourages the intermediate representation to capture higher-level information
such as:

* upcoming gesture activation,
* motion intensity and evolution,
* body-part participation,
* preparation-to-realization structure,
* cross-part motion organization.

The predicted plan subsequently provides structured guidance for detailed motion synthesis.

# 📊 Results

## BEAT2 — Speaker 2 Protocol

| Method         |     FGD ↓ |      BC ↑ |     DIV ↑ |    SRGR ↑ | MSE ↓ |     LVD ↓ |
| -------------- | --------: | --------: | --------: | --------: | ----: | --------: |
| EMAGE          |     5.512 |     0.772 |     13.06 |     0.323 | 7.680 |     7.556 |
| ProbTalk       |     5.127 |     0.771 |     13.27 |     0.300 | 8.617 |     7.634 |
| MambaTalk      |     5.366 |     0.781 |     13.05 |     0.290 | 6.289 |     6.897 |
| RAG-Gesture    |     8.082 |     0.734 |     11.97 |     0.390 | 7.248 |     6.947 |
| SemTalk        |     4.278 |     0.777 |     12.91 |     0.430 | 6.153 |     6.938 |
| **IntentTalk** | **3.812** | **0.817** | **13.95** | **0.442** |     — | **6.462** |

IntentTalk achieves improved motion quality and semantic relevance while maintaining strong
gesture diversity and speech-motion synchronization. In particular, the improvements in
FGD and SRGR indicate that explicitly reasoning about upcoming gesture organization benefits
both holistic motion generation and semantically meaningful gesture realization.

> **Note:** The table will be updated with the complete results and all evaluated baselines
> after the official release.

# 🔬 Planning Analysis

To investigate whether the proposed representation behaves as a meaningful future motion
plan rather than merely an additional latent variable, we evaluate several alternative
planning conditions.

| Planning Condition        |     FGD ↓ |    SRGR ↑ | Event F1 ↑ |   Acc@1 ↑ |   Acc@5 ↑ |
| ------------------------- | --------: | --------: | ---------: | --------: | --------: |
| No Plan                   |     4.347 |     0.424 |      0.668 |         – |         – |
| Random Plan               |     4.412 |     0.418 |      0.641 |         – |         – |
| Current Motion Token      |     4.286 |     0.429 |      0.706 |     0.462 |     0.731 |
| **Predicted Future Plan** | **3.812** | **0.442** |  **0.784** | **0.536** | **0.798** |
| Oracle Future Plan        |     3.615 |     0.457 |      0.836 |         – |         – |

Compared with current-state motion tokens, future-grounded plans provide more informative
guidance for semantic event prediction and motion generation. The oracle-plan result further
reveals the potential performance gain obtainable from improved speech-to-plan prediction.

# ⚡ Quick Start

## 1. Create Environment

We recommend using a dedicated Conda environment.

```bash
conda create -n intenttalk python=3.8 -y
conda activate intenttalk
```

Install PyTorch and the required dependencies:

```bash
pip install -r requirements.txt
```

Please ensure that the CUDA and PyTorch versions are compatible with your local environment.

# 📦 Dataset

## BEAT2

IntentTalk is primarily evaluated on the **BEAT2** dataset using SMPL-X motion
representations.

Please download BEAT2 following the official dataset instructions and organize it as:

```text
IntentTalk/
├── BEAT2/
│   └── beat_english_v2.0.0/
├── configs/
├── dataloaders/
├── models/
├── weights/
├── utils/
└── train.py
```

The original dataset is not redistributed in this repository.

# 📥 Pretrained Models

IntentTalk requires several pretrained components, including:

* motion representation models,
* speech feature encoder,
* future motion-plan representation model,
* speech-to-plan predictor,
* holistic motion generator.

After downloading the released checkpoints, organize the weights as:

```text
weights/
├── pretrained_vq/
├── future_plan/
│   └── intenttalk_future_plan_vq.bin
├── speech_encoder/
├── base_motion/
└── intenttalk/
    └── intenttalk_best.bin
```

Pretrained checkpoints will be released after publication.

# 🚀 Training

IntentTalk follows a staged training strategy.

## Stage 1: Learn the Future Motion-Plan Space

The first stage learns a discrete representation from future motion windows.

Conceptually,

```text
Future Motion Window
        ↓
Motion Plan Encoder
        ↓
Vector Quantization
        ↓
Discrete Future Plan
```

The learned codebook provides a compact representation of upcoming motion organization.

Example:

```bash
python train.py \
    --config configs/intenttalk_future_plan.yaml \
    --train_plan
```

After training, save the best future-plan representation model under:

```text
weights/future_plan/
```

## Stage 2: Train Speech-to-Plan Prediction

Given speech context, the planner predicts the discrete motion plan corresponding to the
upcoming motion window.

```text
Speech Context
      ↓
Speech Encoder
      ↓
Planner
      ↓
Predicted Future Plan
```

Example:

```bash
python train.py \
    --config configs/intenttalk.yaml \
    --train_planner \
    --plan_vq_ckpt weights/future_plan/intenttalk_future_plan_vq.bin
```

## Stage 3: Train IntentTalk Motion Generation

The predicted future plan is used by both the event-aware semantic gate and hierarchical
plan-conditioned modulation modules.

```text
                    ┌──────────────────────┐
Speech ────────────→│ Speech Representation│
                    └──────────┬───────────┘
                               │
                               ▼
                    ┌──────────────────────┐
                    │ Future Plan Predictor│
                    └──────────┬───────────┘
                               │
                     Predicted Future Plan
                         ┌─────┴─────┐
                         ▼           ▼
                 Event-Aware     Plan-Conditioned
                 Semantic Gate     Modulation
                         \           /
                          \         /
                           ▼       ▼
                         Holistic Motion
```

Example training command:

```bash
python train.py \
    --config configs/intenttalk.yaml \
    --plan_vq_ckpt weights/future_plan/intenttalk_future_plan_vq.bin
```

# 🧪 Testing

To evaluate a trained IntentTalk model:

```bash
python train.py \
    --config configs/intenttalk.yaml \
    --test_state \
    --load_ckpt weights/intenttalk/intenttalk_best.bin \
    --plan_vq_ckpt weights/future_plan/intenttalk_future_plan_vq.bin
```

The evaluation includes metrics for:

* motion distribution quality,
* gesture diversity,
* speech-motion synchronization,
* semantic relevance,
* motion reconstruction,
* semantic event prediction,
* future-plan prediction.

Representative metrics include:

```text
FGD
BC
DIV
SRGR
MSE
LVD
Event F1
Plan Acc@1
Plan Acc@5
```

# 🎙️ Inference

At inference time, IntentTalk requires **speech only**.

No future motion or oracle motion-plan information is used.

The inference pipeline is:

```text
Speech
  ↓
Speech Representation
  ↓
Future Plan Prediction
  ↓
Event-Aware Semantic Gating
  +
Plan-Conditioned Motion Generation
  ↓
SMPL-X Motion
```

Example:

```bash
python train.py \
    --config configs/intenttalk.yaml \
    --inference \
    --audio_infer_path demo/example.wav \
    --load_ckpt weights/intenttalk/intenttalk_best.bin \
    --plan_vq_ckpt weights/future_plan/intenttalk_future_plan_vq.bin
```

# 🧩 Ablation Studies

The repository will include configurations for reproducing the major ablations reported
in the paper.

## Without Future Plan

```text
Speech → Motion Generator → Motion
```

This variant removes explicit future-oriented motion organization.

## Without Event-Aware Gate

```text
Future Plan
    ↓
Motion Generator
```

Semantic motion activation is no longer explicitly regulated, which can introduce
spurious or temporally misplaced semantic gestures.

## Without Plan-Conditioned Modulation

The future plan is not hierarchically injected into different body regions, weakening
cross-part coordination.

## Full IntentTalk

```text
Future Motion Planning
        +
Event-Aware Semantic Gating
        +
Hierarchical Plan-Conditioned Modulation
        ↓
Coherent Holistic Co-Speech Motion
```

# 🔍 Plan Representation Analysis

Beyond generation metrics, IntentTalk analyzes what information is encoded in the learned
motion-plan representation.

The released analysis code will include:

### Linear Probing

We probe the learned representations for information about:

* current body-part activity,
* future body-part activity,
* future motion energy,
* future motion evolution.

### Plan Prototype Visualization

Motion samples assigned to the same plan code are visualized to examine whether individual
codes correspond to consistent motion organizations.

### Representation Visualization

t-SNE or UMAP is used to visualize the structure of the learned motion-plan space.

### Current vs. Future Representation

We compare:

```text
Current Motion Token
Continuous Future Feature
Discrete Future Plan
```

to study whether discrete future-oriented representations better capture upcoming gesture
organization.

# 🎬 Qualitative Results

IntentTalk is designed to generate gestures with coherent temporal progression rather than
isolated locally plausible poses.

Typical examples demonstrate:

```text
Preparation
    ↓
Semantic Activation
    ↓
Gesture Realization
    ↓
Cross-Part Coordination
```

Compared with previous methods, IntentTalk can more clearly reproduce preparation-to-
realization transitions and coordinate semantic gestures across the upper body, hands,
torso, and lower-body stance.

Qualitative videos and side-by-side comparisons will be provided in the project page.

# 🗂️ Repository Layout

The main repository structure is expected to be:

```text
IntentTalk/
├── BEAT2/
├── configs/
│   ├── intenttalk.yaml
│   └── intenttalk_future_plan.yaml
├── dataloaders/
├── datasets/
├── models/
│   ├── motion representations
│   ├── future motion planner
│   ├── event-aware semantic gate
│   └── IntentTalk motion generator
├── utils/
├── weights/
│   ├── pretrained_vq/
│   ├── future_plan/
│   └── intenttalk/
├── demo/
├── train.py
└── README.md
```

# 🎨 Visualization

Generated motions are represented using SMPL-X parameters.

We recommend using the official SMPL-X model together with Blender or the BEAT2
visualization tools to render generated motions.

The repository will also provide scripts for generating qualitative comparison videos.

# 📐 Method at a Glance

The key difference between conventional speech-to-motion generation and IntentTalk is:

### Previous Paradigm

```text
Speech
  ↓
Low-Level Motion Generation
  ↓
Gesture
```

High-level gesture organization and detailed motion realization are learned jointly.

### IntentTalk

```text
Speech
  ↓
Predict Upcoming Motion Organization
  ↓
Discrete Future Motion Plan
  ↓
Determine WHEN semantic motion occurs
  ↓
Determine HOW the plan is realized
  ↓
Holistic Motion
```

This explicit future-oriented intermediate representation allows the generator to reason
about upcoming gesture structure before synthesizing detailed motion.

# 📄 Citation

If you find this work useful for your research, please consider citing:

```bibtex
@article{zhang2026intenttalk,
  title   = {IntentTalk: Future-Grounded Motion Planning for Co-Speech Gesture Generation},
  author  = {Jiye Zhang, Guibiao Liao, Dingwei Liu, Gaolin Yang, Xiuhua Jiang, and Jiangbo Xu},
  year    = {2026}
}
```

> The BibTeX entry will be updated after publication.

# 🙏 Acknowledgments

Our implementation builds upon several excellent open-source projects and prior work in
holistic co-speech gesture generation.

We sincerely thank the authors of **BEAT2**, **EMAGE**, **SemTalk**, and other related
projects for releasing their datasets, models, and code.

# 📜 License

Please follow the licenses of the corresponding datasets, SMPL-X models, pretrained
speech models, and third-party repositories.

The license for the IntentTalk source code will be provided upon release.

PlanTalk

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

PlanTalk

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README