|
β¨Bonjour, I am a Ph.D. student in the Department of Computer Science, University of Rochester (UR), advised by Prof. Jiebo Luo. I aim at building multimodal interactive AI systems that can not only ground, reason and generate over the external world signals, to understand human language, but also assist humans in decision-making and efficiently solving social concerns, e.g., robot, medical. As steps towards this goal, my research interests include but are not limited to multimodal understanding, multimodal generation and multimodal foundation model post-training. Prior to that, I got my master's degree from Peking University (PKU) in 2023, advised by Prof. Li Yuan and Prof. Jie Chen. And I obtained the honored bachelor's degree from University of Electronic Science and Technology of China (UESTC) in 2020.Email / Google Scholar / Github / Twitter / Zhihu / LinkedIn |
Winter 2024, Puerto Ricoβ¨
|
|
 Awesome Concealed Object Segmentation Awesome Autoregressive Models in Vision   A Practical Guide for Medical Large Language Models. |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
My current research mainly focuses on the vision+language, generative model. *Equal Contribution. |
|
MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators Shenghai Yuan*, Jinfa Huang*, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, Jiebo Luo IEEE Transactions on Pattern Analysis and Machine Intelligence, TPAMI 2025 (Github Repo 1300+ Starsπ) [Paperlink], [Code], [Page], Area: Text-to-Video Generation, Diffusion Model, Time-lapse Videos Existing text-to-video generation models have not adequately encoded physical knowledge of the real world, thus generated videos tend to have limited motion and poor variations. In this paper, we propose MagicTime, a metamorphic time-lapse video generation model, which learns real-world physics knowledge from time-lapse videos and implements metamorphic video generation. |
|
Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, Jie Chen IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2023 (Highlight, Top 2.5%) [Paperlink], [Code], [Page],  Area: Video-and-Language Representation, Machine Learning, Video-Text Retrieval, Video Captioning To solve the problem of the modality gap in video-text feature space, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. We use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. |
|
Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations Peng Jin*, Jinfa Huang*, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A. Clifton, Jie Chen Conference on Neural Information Processing Systems, NeurIPS 2022 (Spotlight Presentation, Top 5%) [Paperlink], [Code],  Area: Video-and-Language Representation, Machine Learning, Video-Text Retrieval, Video Captioning To solve the problem of the modality gap in video-text feature space, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. We use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. |
|
|
|
|
|
|
|
|
|
|
|
My hometown is Guangdong, you can call me Cantonese name: Gamfaat Wong.
This awesome template is inspired from this good man.
|