Ruofan Liu

Ruofan Liu

Ph.D. @ Computer Science, Institute of Science Tokyo

RUOFAN LIU - About Me

Ph.D. student (3rd year) at the Deparment of Computer Science, Institute of Science Tokyo, supervised by Prof. Hideki Koike, currently a visiting researcher at Stanford Computational Imaging Lab, Stanford University, supervised by Prof. Gordon Wetzstein, and a research fellow at Sony Computer Science Laboratories, supervised by Prof. Shinichi Furuya.

I obtained my M.E. in Computer Science at the Institute of Science Tokyo with Prof. Hideki Koike, and obtained my B.E. in Computer Science and Technology at Shanghai Jiao Tong University with Prof. Baoliang Lu.

My research focuses on computational sensing, multimodal generative methods, and human-AI interaction. My long-term goal is to create a next-generation deep learning framework and a generative interaction model that enables scalable, generalizable, and explainable human-centered AI systems. I believe the critical path toward achieving this involves advancing cross-modal synthesis and large-scale learning, particularly when direct sensing is expensive, intrusive, or impractical. To this end, I have been focusing on the following topics:

  • Generative AI, Foundation Model, and Content Creation: Video generation model post-training for content creation (in progress), Audio-to-motion generation based on DiT (under review for CVPR '26), VLM-based garment generation from a single image (under review for CVPR '26)
  • Cross-modal Synthesis and Computational Sensing: pose-to-EMG estimation via multimodal learning for hand muscles (NeurIPS '25) and for large muscles (TVCG, IEEE VR '26), VQ-VAE-based cross-modal synthesis (CHI '25), Hand muscle electromyography inference (SA '24)
  • Interative System and XR Prototype: Embodied and detached golf muscle training in AR (TVCG, IEEE VR '26), Aligment-based piano AR prototype (ISMAR '23), Hand pose discrepancy visualization for motor skill learning (CHI EA '23)

Beyond research, I enjoy go-karting, piano playing, snowboarding, indoor bouldering, traveling, and walking friends’ dogs.

Research Interests

Generative AI, Multimodal Learning, Large Foundation Model, Augmented Reality, Human-Computer Interaction

For more info

More detailed info can be found in CV and Publications. Feel free to DM me using LinkedIn! Happy to have any chats!