CinemaWorld: Generative Augmented Reality with LLMs and 3D Scene Generation for Movie Augmentation

Problem — augmented viewing has to be predesigned, and stays 2D

Systems like IllumiRoom and ExtVision extend a film beyond the screen, but the effects are prepared by hand for a specific title, so they don’t scale, and the augmentation stays a 2D projection around the display rather than something embedded in the room.

Solution — generate the augmentation from the film itself

CinemaWorld is a generative augmented reality (GenAR) system: the augmentation is produced automatically from the movie instead of being authored in advance, and it is blended into the space as 3D content on floors, walls, furniture, and even the viewer’s own body. A formative elicitation study with eight film students gave us the design space — seven augmentation methods: particle effects, surrounding objects, room textures, character presence, body transformation, window augmentation, and lighting effects.

The pipeline runs in three stages. A vision–language model analyzes the scene to extract timestamp, context, effect types, objects, background textures, and lighting. The room is captured through Quest 3 surface detection or higher-fidelity LiDAR scanning. Generative AI then produces the textures, lighting, particles, and 3D objects, rendered on the Quest 3 aligned to the room geometry and synchronized with the movie timeline.

Evaluation

Across 100 video clips, we assessed visual, semantic, and temporal alignment for the seven augmentation types — 18 of 21 evaluation conditions exceeded 80% accuracy. A usability study with 12 participants compared the system against a no-augmentation baseline and a 2D augmentation method, and found higher immersion and enjoyment. Interviews with 8 film creators pointed to the potential of stepping into the film’s world, alongside concerns about control and authorship that motivate authoring tools for creators.