Map2Video: Guiding Real-World-Grounded AI Video Generation

Problem — AI video generation isn’t grounded in the real world

AI video generation lowers the barrier to video creation, but existing tools give little support for grounding content in real-world locations — a requirement in applications like film previsualization and scenario generation. In practice, users fall back on verbose textual descriptions or reference images from on-site photos or street view imagery (SVI), which are cumbersome and unreliable across shots. Our formative study with five filmmakers surfaced challenges in shot composition, character motion, and camera control when using SVI with an image-to-video tool.

Solution — guiding generation from the map

Map2Video guides SVI-grounded AI video generation through map-based character trajectory annotation and direct manipulation of cameras and character masks. Both are defined in geodetic coordinates and projected onto panoramic screen coordinates, so what the user lays out on the map stays consistent with what the model generates. Following familiar filmmaking practices like location scouting and rehearsal, users pick a location, place actors and cameras in the street view, sketch movement paths, refine camera motion, and generate the shot.

Evaluation

In a study with 12 filmmakers, Map2Video provided stronger controllability than an image-to-video baseline — for both scene replication and open-ended creative exploration — while reducing cognitive effort and improving spatial consistency across multiple shots.

Work done as an intern at Fujitsu Research of America.