AI agents rebuilt San Francisco's Union Square in 3D for $33—but visual details still fooled them

PhiloLabs ran an experiment to push the limits of AI coding agents beyond simply producing working software. Researchers set multiple Claude Fable 5.1 agents to work on a challenging task: reconstructing San Francisco's Union Square as an interactive 3D scene using real geographic data and reference photographs.

The result was striking. Within two hours, the agents had assembled a functioning Three.js recreation containing 453 building footprints, 75 custom façades, 129 named stores, 220 moving pedestrians, and 109 vehicles—including the famous cable cars on Powell Street. The cost: roughly $33 in API charges for approximately 8 million tokens of computation.

However, getting the code to run was just the beginning. The real challenge lay in catching visual errors that traditional automated tests would miss entirely.

Building a vision loop#

To evaluate visual accuracy, PhiloLabs integrated Playwright into the development cycle. Rather than relying solely on unit tests, they had Playwright capture screenshots from 34 fixed camera angles throughout the scene and compare them directly to photographs of the actual location.

This approach yielded 147 detailed comparison sheets, making subtle errors visible. A building might technically sit at the correct coordinates but possess wrong proportions. A storefront could end up on the opposite side of the street. Using consistent camera positions also made it easy to track which problems persisted or appeared in successive iterations.

Agents critiquing agents#

The workflow then shifted to review. Specialized agents examined the comparison materials, with some focusing on architectural and geographic accuracy while others assessed technical art and interactivity. They produced nine detailed reports identifying issues.

This division of labor highlights something important: not every mistake fits neatly into testable logic. Verifying that a building occupies the correct GPS coordinates is straightforward. Determining whether the overall streetscape feels like Union Square requires subjective judgment that neither unit tests nor simple visual inspection can fully capture.

Incomplete data, incomplete solutions#

The agents weren't working from finished 3D models. Instead, they synthesized open geographic databases—primarily and USGS elevation data—alongside photographs and local research to generate geometry and textures.