EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling

Haiyang Liu^1*, Zihao Zhu^2*, Giorgio Becherini³, Yichen Peng⁴, Mingyang Su⁵,
You Zhou, Xuefei Zhe, Naoya Iwamoto, Bo Zheng, Michael J. Black³

(^*Equal Contribution)

¹The University of Tokyo, ²Keio University, ⁴Japan Advanced Institute of Science and Technology,
³Max Planck Institute for Intelligent Systems, ⁵Tsinghua University

♾️ Colab 🤗 Gradio arXiv Video Dataset Code Blender Add-On

Holistic Dataset

1-Wayne

2-Scott

3-Solomon

4-Lawrence

5-Stewart

11-Nidal

12-Zhao

13-Lu

15-Carlos

16-Jorge

17-Itoi

18-Daiki

20-Li

6-Carla

7-Sophie

9-Miranda

10-Kieks

21-Ayana

22-Luqi

23-Hailing

24-Kexin

25-Goto

27-Yingqing

28-Tiffnay

30-Katya

Face Zoom In

1-Wayne

2-Scott

3-Solomon

4-Lawrence

5-Stewart

11-Nidal

12-Zhao

13-Lu

15-Carlos

16-Jorge

17-Itoi

18-Daiki

20-Li

6-Carla

7-Sophie

9-Miranda

10-Kieks

21-Ayana

22-Luqi

23-Hailing

24-Kexin

25-Goto

27-Yingqing

28-Tiffnay

30-Katya

Generated Results

Sequence-1

Sequence-2

Sequence-3

Sequence-4

Sequence-5

Sequence-6

Sequence-6

Sequence-8

Demo Video