3D Human Digitization from a Single Image!

Jia-Bin Huang · Beginner ·📄 Research Papers Explained ·2y ago

About this lesson

Single-Image 3D Human Digitization with Shape-Guided Diffusion Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang ACM SIGGRAPH Asia 2023 📝 Paper: https://human-sgd.github.io/ 🌐 Website: https://human-sgd.github.io/ 💻 Code: https://human-sgd.github.io/ Abstract: We present an approach to generate a 360-degree view of a person with a consistent, high-resolution appearance from a single input image. NeRF and its variants typically require videos or images from different viewpoints. Most existing approaches taking monocular input either rely on ground-truth 3D scans for supervision or lack 3D consistency. While recent 3D generative models show promise of 3D consistent human digitization, these approaches do not generalize well to diverse clothing appearances, and the results lack photorealism. Unlike existing work, we utilize high capacity 2D diffusion models pretrained for general image synthesis tasks as an appearance prior of clothed humans. To achieve better 3D consistency while retaining the input identity, we progressively synthesize multiple views of the human in the input image by inpainting missing regions with shape guided diffusion conditioned on silhouette and surface normal. We then fuse these synthesized multi-view images via inverse rendering to obtain a fully textured high-resolution 3D mesh of the given person. Experiments show that our approach outperforms prior methods and achieves photorealistic 360-degree synthesis of a wide range of clothed humans with complex textures from a single image. Compared baseline methods: - PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization, ICCV 2019 https://shunsukesaito.github.io/PIFu/ - Text-Guided Texturing of 3D Shapes, SIGGRAPH 2023 https://texturepaper.github.io/TEXTurePaper/ - Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors, arXiv 2023 https://guochengqian.github.io/project/magic123/

Original Description

Single-Image 3D Human Digitization with Shape-Guided Diffusion Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang ACM SIGGRAPH Asia 2023 📝 Paper: https://human-sgd.github.io/ 🌐 Website: https://human-sgd.github.io/ 💻 Code: https://human-sgd.github.io/ Abstract: We present an approach to generate a 360-degree view of a person with a consistent, high-resolution appearance from a single input image. NeRF and its variants typically require videos or images from different viewpoints. Most existing approaches taking monocular input either rely on ground-truth 3D scans for supervision or lack 3D consistency. While recent 3D generative models show promise of 3D consistent human digitization, these approaches do not generalize well to diverse clothing appearances, and the results lack photorealism. Unlike existing work, we utilize high capacity 2D diffusion models pretrained for general image synthesis tasks as an appearance prior of clothed humans. To achieve better 3D consistency while retaining the input identity, we progressively synthesize multiple views of the human in the input image by inpainting missing regions with shape guided diffusion conditioned on silhouette and surface normal. We then fuse these synthesized multi-view images via inverse rendering to obtain a fully textured high-resolution 3D mesh of the given person. Experiments show that our approach outperforms prior methods and achieves photorealistic 360-degree synthesis of a wide range of clothed humans with complex textures from a single image. Compared baseline methods: - PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization, ICCV 2019 https://shunsukesaito.github.io/PIFu/ - Text-Guided Texturing of 3D Shapes, SIGGRAPH 2023 https://texturepaper.github.io/TEXTurePaper/ - Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors, arXiv 2023 https://guochengqian.github.io/project/magic123/
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Related Reads

📰
On July 1, 2026, arXiv will spin out from Cornell University, its home for the past 25 years, to become an independent nonprofit organization. Major funding support from Simons Foundation and Schmidt Sciences. Ditching the red for their website. [N]
arXiv is becoming an independent nonprofit organization after 25 years at Cornell University, backed by major funding, which will impact the future of research and academia
Reddit r/MachineLearning
 📰
CS-NRRM™ Official Publications: Paper 1 and Paper 2 Are Now Available
Learn about the CS-NRRM's official publications on a 12-year longitudinal human observation archive and its significance in research and development
Medium · Data Science
 📰
Building a Research Pipeline: From Google Scholar Search to Citation Network Analysis
Learn to build a research pipeline to efficiently manage and analyze academic papers and citations, staying current in a fast-moving research field
Dev.to · NexGenData
 📰
Challenges of Developing a New Conceptual Framework
Learn to overcome the challenges of developing a new conceptual framework to improve your research and problem-solving skills
Dev.to · 根本卓哉　Takuya Nemoto
Up next
Indians Under House Arrest in America? 😱 Immigration Crisis Explained | SumanTV Classroom
SumanTV Classroom
Watch →