|
Currently, I am a first-year Ph.D. student in the Department of Computer Science, University of Rochester (UR), advised by Prof. Jiebo Luo. I aim at building multimodal interactive AI systems that can not only ground and reason over the external world signals, to understand human language, but also assist humans in decision-making and efficiently solving social concerns, e.g., robot. As steps towards this goal, my research interests include but are not limited to Multimodal Large Language Models, Video Generation and Multimodal Agents. Prior to that, I got my master's degree from Peking University (PKU) in 2023, advised by Prof. Li Yuan and Prof. Jie Chen. And I obtained the honored bachelor's degree from University of Electronic Science and Technology of China (UESTC) in 2020. |
|
|
 MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators. A Practical Guide for Medical Large Language Models. |
|
|
|
|
|
|
|
|
|
My current research mainly focuses on vision+language, generative model. *Equal Contribution. |
|
MagicTime: Time-lapse Video Generation Models as Metamorphic Simulators Shenghai Yuan*, Jinfa Huang*, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, Jiebo Luo Arxiv [Paperlink], [Code], [Page] Area: Text-to-Video Generation, Diffusion Model, Time-lapse Videos Existing text-to-video models have not adequately encoded physical knowledge of the real world, thus generated videos tend to have limited motion and poor variations. In this paper, we propose MagicTime, a metamorphic time-lapse video generation model, which learns real-world physics knowledge from time-lapse videos and implements metamorphic video generation. |
|
Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach Shaofeng Zhang, Jinfa Huang, Qiang Zhou, Zhibin Wang, Fan Wang, Jiebo Luo, Junchi Yan Conference on International Conference on Learning Representations, ICLR 2024 (Review Rating: 866) [Paperlink], [Code] Area: Image Outpainting, Diffusion Model, GenAI We have proposed PQDiff, which learns the positional relationships and pixel information at the same time. Methodically, PQDiff can outpaint at any multiple in only one step, greatly increasing the applicability of image outpainting. |
|
Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learning Peng Jin, Jinfa Huang, Pengfei Xiong, Shangxuan Tian, Chang Liu, Xiangyang Ji, Li Yuan, Jie Chen IEEE International Conference on Computer Vision and Pattern Recognition, CVPR 2023 (Highlight, Top 2.5%) [Paperlink], [Code] Area: Video-and-Language Representation, Machine Learning, Video-Text Retrieval, Video Captioning To solve the problem of modality gap in video-text feature space, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. We use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. |
|
Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations Peng Jin*, Jinfa Huang*, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A. Clifton, Jie Chen Conference on Neural Information Processing Systems, NeurIPS 2022 (Spotlight Presentation, Top 5%) [Paperlink], [Code] Area: Video-and-Language Representation, Machine Learning, Video-Text Retrieval, Video Captioning To solve the problem of the modality gap in video-text feature space, we propose Expectation-Maximization Contrastive Learning (EMCL) to learn compact video-and-language representations. We use the Expectation-Maximization algorithm to find a compact set of bases for the latent space, where the features could be concisely represented as the linear combinations of these bases. |
|
|
|
|
|
|
|
Last updated on April, 2024.
This awesome template borrowed from this good man💗💗💗.
|
