Zhuofan Zong

I am a second-year Ph.D. student from MMLab, The Chinese University of Hong Kong. I'm supervised by Prof. Hongsheng Li. I received both my Bachelor's and Master's degrees from Beihang University, supervised by Prof. Biao Leng.

I worked as a research intern in the Base Model Department at SenseTime Research, working closely with Guanglu Song and Yu Liu. During my internship at SenseTime, I was a core member of the founding team for frontline R&D projects, including the large vision fundation model, the multimodal interactive model, and the AIGC product SenseMirage.

Feel free to reach out for research discussions, collaborations or just to have a chat!

Email  /  Google Scholar  /  Github  /  WeChat

My research interests lie in the area of Diffusion Models and Multimodal Large Language Models. My previous works also included 2D and 3D Visual Perception. (*: Equal Contribution †: Corresponding Author)

We present EasyRef, the first work that capable of modeling the consistent visual elements of various group image references with a single generalist multimodal LLM for diffusion models.

We propose a diffusion-based framework for video face swapping, featuring hybrid training, an AIDT dataset, and 3D reconstruction for superior identity preservation and temporal consistency.

MoVA is a novel MLLM that can adaptively route and fuse multiple task-specific vision experts in a coarse-to-fine mechanism, alleviating the bias of CLIP vision encoder. Without any bells and whistles, MoVA can achieve significant performance gains over current state-of-the-art methods.

We propose to unleash the prompt encoding capability of large language models (LLMs) for diffusion models. LiDiT-10B surpasses state-of-the-art models including Stable Diffusion 3, DALL-E 3, and Midjourney V6. The proposed method is also one of the core technologies powering SenseMirage.

We propose Visual CoT, including a new pipeline/dataset/benchmark that enhances the interpretability of MLLMs by incorporating visual Chain-of-Thought reasoning, optimizing for complex visual inputs.

We propose a fine-tuning strategy to address the text-to-image misalignment issue with image-to-text concept matching. The training data only includes text prompts.

RAPHAEL proposes space-MoE and time-MoE layers and achieves a state-of-the-art zero-shot FID score of 6.61 on the COCO dataset. The proposed method is also one of the core technologies powering SenseMirage.

We propose a new paradigm, named Historical Object Prediction (HoP) for multi-view 3D detection to leverage temporal information more effectively. HoP achieves 68.5% NDS and 62.4% mAP with ViT-L, outperforming all the counterparts on the nuScenes detection (camera-only) leaderboard.

We present a novel collaborative hybrid assignments training scheme and achieve state-of-the-art performances on object detection and instance segmentation tasks. Specifically, Co-DETR is the first model to achieve 66.0 box AP and 57.0 mask AP on the COCO test-dev leaderboard.

We propose Adaptive Gradient Variance Modulator (AGVM) to train dense visual predictors with very large batch size. It enables training an object detector with one billion parameters in just 3.5 hours, reducing the training time by 20.9×, whilst achieving 62.2 mAP on the COCO val leaderboard.

We propose a generic self-slimmed learning approach for ViT token pruning. Our method can speed up ViTs by 1.7x with negligible accuracy drop, and even speed up ViTs by 3.6x while maintaining 97% of their performance on the ImageNet-1K dataset.

We introduce RCNet, a novel architecture for multiscale feature fusion in object detection, addressing the inefficiencies and limitations of traditional Feature Pyramid Networks (FPN). Experiments on the MS COCO dataset show RCNet can bring significant performance gains.

We propose graph attention-based 3D CNNs (AGCN) for video action detection, addressing the limitations of existing models that overlook intra and inter-proposal relationships. AGCN achieves the state-of-the-art performance and improves the average mAP by 3.7% on the THUMOS 2014 dataset.