Multimodal generation
Generative models for high-quality, controllable image and video synthesis, including diffusion and unified generative models.
Zhuofan Zong 宗卓凡 · Ph.D. Candidate at CUHK
I build generative models that understand, reason, and create across modalities. I am a final-year Ph.D. candidate at MMLab, CUHK, advised by Prof. Hongsheng Li and Prof. Xiaogang Wang, and currently a Research Intern at ByteDance Seed. My current research focuses on agentic visual generation and autoregressive video generation.
Seeking 2027 Research Scientist opportunities.
I expect to graduate in July 2027 and am looking for roles focused on
visual generation and world models.
Get in touch →
Research
Multimodal understanding
Multimodal large language models that perceive diverse visual inputs and reason over complex multimodal content.
Large vision foundation models
Scalable visual representation learning, model architectures, and efficient training for general-purpose vision systems.
Experience
Beijing, China
ByteDance Seed
Research Intern · Seed Vision · Mentor: Liyang Liu
Working on enhancing Seedream's agentic generation capabilities.
Beijing, China
SenseTime Research
Research Intern · Interaction Model R&D · Mentor: Mingjie Zhan
Designed visual output capabilities for interactive foundation models, with an emphasis on real-time, synchronized, and high-fidelity audio-video generation.
Beijing, China
SenseTime Research
Research Intern · Base Model R&D · Mentors: Guanglu Song & Yu Liu
Researched diffusion models, multimodal LLMs, and large vision foundation models. A core founding member of frontier R&D projects including a large vision foundation model, a multimodal interactive model, and SenseMirage.
Publications
Multimodal Models
2023 — 2026
VoCa: Unified Autoregressive Modeling for Talking Audio-Video Generation
A unified autoregressive framework for synchronized talking audio-video generation.
ECCV
2026
2026
VividFace: A Diffusion-Based Hybrid Framework for High-Fidelity Video Face Swapping
Hybrid diffusion training and 3D reconstruction for identity-preserving, temporally consistent video face swapping.
NeurIPS
2025
2025
EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM
A generalist multimodal LLM that models consistent visual elements from diverse groups of reference images.
ICML
2025
2025
MoVA: Adapting Mixture of Vision Experts to Multimodal Context
A coarse-to-fine routing mechanism that adaptively fuses task-specific vision experts for multimodal language models.
NeurIPS
2024
2024
Visual CoT: Advancing Multi-Modal Language Models with Chain-of-Thought Reasoning
A dataset, benchmark, and pipeline for interpretable visual chain-of-thought reasoning over complex inputs.
NeurIPS
2024Spotlight
2024Spotlight
LiDiT: Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
Unleashing LLM prompt encoding for text-to-image diffusion models, powering core technology in SenseMirage.
NeurIPS
2024
2024
CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching
A text-only fine-tuning strategy that improves concept-level alignment in text-to-image diffusion models.
NeurIPS
2024
2024
RAPHAEL: Text-to-Image Generation via Large Mixture of Diffusion Paths
Space- and time-MoE layers for highly scalable text-to-image generation, also powering SenseMirage.
NeurIPS
2023
2023
Vision Foundation Models
2020 — 2023
ICCV
2023
2023
ICCV
2023
2023
NeurIPS
2022
2022
ECCV
2022
2022
ACM MM
2021
2021
AAAI
2020
2020
* Equal contribution. Author lists are abbreviated only where shown.
Education & Service
The Chinese University of Hong Kong
Ph.D. in Electronic Engineering · MMLab
Aug 2023 — Jul 2027 (expected) · Advisors: Prof. Hongsheng Li and Prof. Xiaogang Wang
Beihang University
MPhil in Computer Technology
Aug 2020 — Jan 2023 · Advisor: Prof. Biao Leng
Beihang University
BEng in Computer Science and Engineering
Aug 2016 — Jun 2020 · Advisors: Prof. Biao Leng & Prof. Xianglong Liu