After a presentation at SIGGRAPH Asia in Hong Kong, I met Johnnyfly. He wanted me to try some hardware his team was developing. It didn’t exist yet, but he thought I might find something interesting to do with it. Johnnyfly described the device as a kind of “magic wand” for capturing pieces of the physical world.
A few months later, the hardware was real.
He’d seen my monster drawings, and he was interested in what an artist might do with the technology. Kosmo provided a spatial camera for me to test. Could it become part of everyday creative work?
A little reality, brought inside
Kosmo captures physical places and objects as Gaussian splats. If you haven’t heard of splats, imagine a scene built from many tiny, overlapping shapes, each carrying information about its position, shape, color, and opacity. Together, they reconstruct the appearance of the captured space … including the appearance of its materials, lighting, and shadows.
The technology wasn’t entirely new, but the form factor was ... now I had a small, "magic wand" to try it with. And another piece was falling into place: Gaussian splat support in RenderMan.
One of the scans I created was the plaza at Pixar, with the lamp and ball outside the Steve Jobs Building.
You might recognize them.
CapturedSplatsI could bring that capture into Houdini’s Solaris environment through USD and render it with RenderMan. I didn’t have to rebuild the plaza as a conventional polygonal scene. I could just render splats.
Until recently, I’d mostly been reading about this way of representing the world. Now these pieces … Kosmo, splats, Houdini, and RenderMan … had come together into a pipeline I could actually use.
Reality → splat → render.
So I get to experiment.
What else can it do?
Once the splats were in Houdini, I could work with them as point data. Familiar Houdini tools let me move them around, change their scale, or adjust their opacity. Simple experiments to get a feel for what I could control.
That led me toward stylization.
What would happen if I pushed the captured detail toward something more painterly? My first treatments affected everything. The building, trees, lamp, and ball all began to break into textured marks. Interesting, but not quite it.
Using a camera-based falloff, I could keep the ball clear while letting the background become more abstract. Now I could choose where the image needed detail and where it could loosen up, but I wanted more control.
Naturally, I brought in more new tech, agents to drive Houdini, allowing me to art-direct and automate iterations. I created a design document for the stylization and used that as a guide for creating painterly foliage, objects receding into the distance, brush strokes that followed the geometry … and used those directions to guide procedural changes to the scene.
Each render became something to critique, paint over, and refine. Working this way let me concentrate on the image while building up an adjustable treatment of the capture, with room to keep exploring.
From there, I developed a fuller painterly treatment of the plaza. The place remained recognizable, but the surfaces, edges, and color took on a different character.
CapturedPainterlyI didn’t know exactly where this would go when Johnnyfly first approached me. Putting the pieces together gave me a way to find out.
Just Getting Splatting
The Lab, an internal group at Pixar whose mission is to explore experimental workflows, asked me to scan a cardboard frame for a project. The scan picked up the folded edges, corrugation, handmade knobs …
And it even got the glue!
And that experiment was already suggesting several new ones. Including scanning and stylizing the entire Lab … full of art supplies, tools, toys, and even a Jaws pinball machine. The result was a very detailed splat, and a time capsule. Adding stylization nicely captured the creative chaos.
That’s what interests me when new tools come together. I couldn’t have made these images three months ago … it took a spatial camera, support for Gaussian splats, and new ways to work with agents on art direction. Discovering how they fit together was the experiment.
Sometimes a random conversation gives you something new to work with. Following up with Johnnyfly gave me a reason to explore. Now seeing what I can make is the interesting part.