ASU researchers, private-sector collaborators publish research on AI-generated 3D models
A team of researchers including ASU faculty, graduate students and alumni, along with industry partners, published breakthrough research last week in the journal Transactions on Machine Learning Research, better known as TMLR.
The paper titled “DecompDreamer: A Composition-Aware Curriculum for Structured 3D Asset Generation,” observes that creating realistic 3D scenes from text prompts is still a major challenge for AI. While today's tools are good at generating a single object like a chair or a tree, they often struggle when asked to create scenes with multiple objects interacting, such as “a knight in shining armor gallops on a brown horse, wearing a blue hat, holding a sword in his right hand.” The researchers argue that this inadequacy is primarily a failure of the tools’ optimization schedules rather than the capabilities of the tools themselves.
To prove this concept, the researchers created DecompDreamer, a new AI approach that tackles this problem by breaking the task into stages. Instead of trying to create the entire scene at once, the system first lays out where each object belongs and how they relate to one another, then goes back to add detail to each object individually. This step-by-step process produces 3D scenes that are more accurate, realistic and visually coherent than existing methods.
As a result, it has numerous potential applications in fields like game development, virtual reality, digital design, animation and more.
The journal also awarded the paper a J2C Certification. This certification is awarded to publications based on responses by the action editor and reviewers to the question of whether the work would be appropriate for the joint NeurIPS/ICLR/ICML Journal-to-Conference Track. Receiving this selective certification gives the researchers the opportunity to present their paper at one of these prestigious participating academic conferences.
Three faculty from The GAME School — Pavan Turaga, Mark Ollila, Tejaswi Gowda — along with computer science PhD student Rajeev Goel (advised by Gowda) and two ASU alumni joined two collaborators from the private sector to conduct the research.
The J2C Certification is exciting news for the research team. According to Turaga, “In the AI world, prestigious conferences carry more weight than journals.” The team has yet to decide at which conference they will present, but they look forward to presenting DecompDreamer and their findings on this new AI method to their industry peers.
-
Figure 1 from the research paper: Illustration of high-quality compositional 3D assets generated by DecompDreamer for complex text prompts. Existing methods often miss objects or distort spatial relationships, while DecompDreamer accurately captures geometry and preserves inter-object layout.
Courtesy image
-
Figure 3 from the research paper: Overview of the DecompDreamer pipeline. Given a text prompt, a VLM generates a scene graph to guide a coarse initialization. The core of our method is a composition-aware optimization curriculum that first models joint relationships to build a coherent structure, then refines individual objects to produce high-fidelity, disentangled 3D assets.
Courtesy image
-
Figure 4 from the research paper: Qualitative comparison between the proposed DecompDreamer and state-of-the-art text-to-3D generators. Additional results can be found in the appendix of the published paper.
Courtesy image
More about the authors
- Utkarsh Nath is an ASU alum who was wrapping up his doctoral degree program in computer science at the time of the project. Turaga was his advisor in the Geometric Media Lab. He is now a senior machine learning engineer at LinkedIn.
- Rajeev Goel is a computer science PhD student in the Ira A. Fulton Schools of Engineering. Tejaswi Gowda is his advisor.
- Rahul Khurana is an ASU alum who received his master’s degree in computer science. He was advised by Pavan Turaga in the Geometric Media Lab at the time the research was conducted. He is now a software development engineer at Amazon Web Services.
- Kyle Min is presently a principal applied science at Oracle, though he was working as a staff research scientist at Intel Corp. during his contributions to the project.
- Mark Ollila is a professor of practice in The GAME School and is the program director of ASU’s Endless Games and Learning Lab.
- Pavan Turaga is a professor and director of The GAME School at ASU.
- Varun Jampani is currently chief AI officer at Arcade AI. His contributions to the project were made while he was still vice president of research at Stability AI.
- Tejaswi Gowda is an assistant professor in The GAME School with a focus on extended reality technologies.