In 2017, everyone seemed to be getting into VR. I was dubious, it looked weird ... but since everyone was doing it, I decided to get a VR headset and check it out myself. After not understanding the buzz, I realized I could sculpt in VR, and that changed how I work today.
I had bought an Oculus Rift mostly because VR had become difficult to ignore. I put on the headset, tried the demos one tries in a new headset, and eventually found Oculus Medium, a sculpting program that let me make forms directly in the space around me.
My first VR sculpt took about thirty minutes. It was a long orange creature with tiny arms, uneven horns, large teeth, and the general expression of something that had just remembered an urgent appointment.
It did not look much like a computer model. It looked like Play-Doh.
I was encouraged.
Moving the camera with my head
Learning a new 3D program usually begins with learning just how to navigate. How do I rotate the camera? How do I select a blasted sphere? Which combination of mouse buttons, modifier keys, and Vulcan grip do I need so I can zoom?
In Medium, if I wanted to look around an object, I moved my head. Simple. I already know how my head works. Once I understood the basic tools, the interface mostly got out of the way.
I was ill prepared because I didn't know how to sculpt. My background was in drawing and 2D illustration, where I had accumulated many years of making fast monsters on menus, illustration, paintings, and anything else that remained still long enough. I assumed that learning to sculpt would require building an entirely new set of instincts.
Instead, all that experience doodling transferred into VR.
The creature from that first half hour had the handmade shapes of my drawings. More surprising, it had my drawing style. I had spent years working in traditional 3D applications without finding a natural way to get that quality into a model. In VR it appeared almost immediately.
This was not evidence that I had secretly been a sculptor all along. It was evidence that the border between drawing and sculpting was less official than I thought.
I made more creatures. I entered VR sculpting battles in front of audiences, complete with theatrical lighting and smoke machines. I won. I lost. Art battles are intense. The smoke machines performed best of all.
Getting the creatures out
Making a monster inside a headset was fun, but I really wanted to know whether it could survive contact with the rest of my workflow.
Medium's geometry was voxel based, but it could export polygonal meshes ... stubbornly at the time, but it could do it after some serious retopology. That meant I could move a sculpt into Maya, give it materials, light it, and render it with RenderMan. Substance Painter made the material stage unusually fast. Instead of spending most of a day building a complex network for a prototype, I could start with a strong procedural material, adjust it to the model, and get a finished 2K RenderMan image in less than an hour.
The first creature looked good in brass.
That changed the experiment. The headset was no longer a place where I made VR things. It had become the front end of a pipeline that connected the speed of a doodle to my familiar production tools.
This pipeline was not something I had planned. I only worked it out after I decided I needed to try out the headset.
Now the doodles looked real
The creatures began in the same place most of mine had always begun: in notebooks, on scraps of paper, and during my travel ... on restaurant menus, which I'd give to the staff and no ... and occasionally get something back.
I drew my menu drawings quickly. Big teeth, big eyes. Part of their appeal depended on the speed, they were usually done before anyone noticed. I could finish a menu drawing in five minutes.
Medium let me keep that gesture while giving it volume.
I could import one of the drawings, place it beside me in the VR workspace, and use it as a reference without treating it as a blueprint. I could pull a jaw forward, push an eye back into a skull, or walk around the creature to see what the drawing had neglected to explain. The model remained loose, but it now occupied space. It had a back, which was new information.
The sculpts also behaved more like handmade objects than conventional computer models. Their surfaces were uneven. A horn pulled with the Move Tool retained evidence of the hand gesture. They felt modeled rather than assembled.
Once I brought them into Maya and RenderMan, that feeling became much stronger. Lighting gave the forms scale. A rough material made one look as though it had been shaped from clay. Brass made another feel heavy enough to damage a table. In Substance Painter I could use curvature, ambient occlusion, and other information from the surface to place wear, dust, and variation where those things might naturally collect.
These are renders of VR sculpts, not photographs of manufactured objects.
The results were digital, but they felt tangible. I wanted to pick them up.
That was a strange reversal. Computer graphics had spent decades getting very good at making immaculate surfaces. I had finally found a workflow that preserved the wobble of a fast drawing, then used physically based materials and lighting to make the wobble feel real.
The creatures could also be rigged and posed like any other model. I could adjust a material, move a light, turn one around, and discover a view I had never drawn. A doodle was no longer a single image. It had become a thing, a real monster.
For a while, photoreal rendering was the natural destination. The models looked convincing in metal, clay, painted surfaces, and materials I had not imagined when I drew them. Then stylized rendering arrived, and the route doubled back.
Back to ink
In 2020, RenderMan was being developed with a new system for nonphotorealistic rendering, Stylized Looks. The VR models were good test subjects because their geometry already had a handmade character.
- Doodle
- Medium
- Maya + RenderMan
- Doodle again
For one test, I used a drawing from my 2019 Inktober series as the target. I imported the drawing into Medium, placed it beside the workspace, and even sampled colors from it. I built the scene in roughly seven hours. In Maya, RenderMan combined material networks, additional outputs, and display filters into the final illustrative treatment.
The important part was that the look could be adjusted interactively. It was not a filter placed over a finished image after the fact. Lighting, materials, geometry, line work, and graphic treatments remained part of the same live rendering session.
The resulting pictures had a family resemblance to the drawings that started the process. The resemblance was not exact, but encouraging. The 3D forms kept the lopsided, handmade quality of the sketches, while the renderer gave me a new set of ways to push and rearrange it.
I got into VR initially because everyone else was trying it. A few years later, I was using it to make ink bottles, dragons, and a growing population of creatures that could become a convincing render or circle back into another drawing.
It was also the point where several previously unrelated parts of my work finally met, and the first orange monster still looks like Play-Doh.