Inigo Quilez art

Inigo Quilez is the founder of Shadertoy and one of the most influential human on this planet when it comes to procedural generation and rendering techniques. In this discussion we go over Pixar, general rendering ideas, mathematics, the history of Shadertoy and so on. To be honest, this was a fanboy moment to discuss with Inigo Quilez, so I decided publish the written format.

Antoine

I think the best way to start the interview, is by letting you introduce yourself and talk about your work in rendering!

Inigo

Yeah as you said I’ve been working mostly on rendering since the beginning of 90’s. I was born in San Sebastian small city in the Basque country between Spain and France. I worked in virtual reality in the early 2000’s in Belgium in Brussels that’s where I learned some French back then it was with huge Silicon Graphics machines and you needed like a hundred thousand dollar equipment to do VR but that was what we got then. I went to work at Pixar where I did work on offline renderings. I learned a lot about lighting and Composition and more of the artistic side of computer graphics not so much on the algorithmic stuff. Then, I worked in Oculus where I did a bit of both where I was doing film and we were trying to discover the language of cinematography in a real-time context and in a virtual reality environment but I was doing both low-level mean and high-level art direction and things and then I created a painting program called Quill for painting in virtual reality and animating and then I became the lead for that project for about four years at, first oculus and then Facebook which is our parent company so some other

In my spare time I’ve been doing lot of computer graphics as well first in the demo scene which is this underground community of computer graphics and artists people in Europe mostly and I joined in 1998 and I was focused mostly on making computer graphics with very small executables where you try to generate as much data visuals,textures,meshes and sound from as little amount of code as possible and then, related to that I created Shadertoy about seven years ago, although the first version was ten years ago. I made sure that I kept that spirit of doing a lot of things with very little amount of code but I made it in a more less underground way where I make it well exposed so that people can have a username and they can associate a work to their name not necessarily a nickname and also where the code is free and available right from the beginning as soon as you publish a piece the code is there for everyone to analyze and then I’ve been also doing a lot of educational content where I have a website where I publish all ideas or experiments. Well not all of them because I don’t have time to write every single thing I do but I try to publish about all the major things that I find them are some discoveries never big ones but little discoveries sometimes are things I learned and I put it all online for people to learn.\ 

Antoine

On your website there’s a lot of stuff, you’ve been publishing since the late nineties. Your are an electrical engineer on your academic formation, how much does it serve you? I know a lot of people ask you this question and interview but I just wanted to discuss on this.

Inigo

It’s a good question because everything I know about computer graphics and programming it’s something I learned on my own by either reading or looking at somebody else’s code which in turn in turn motivated me creating ShaderToy to kind of give back again to the community because I learned everything from the Internet but the math I was studying there didn’t include anything related to number theory or permutations, combinatoric none of that. I geometry of course and calculus, mostly Fourier transforms Laplace transforms calculus and things like that so it does help because anything that has to do with data compression or things like that I can work on those environments. I guess the main thing that studying not computer graphics helped me is to approach things differently I suppose because I never learned how to do anything in a book so I had to find my way and that means that first I had to reinvent the wheel many times you know like I had to discover by myself things that I could have just read about but that meant I got used to discovering new things and giving a try giving a code to problems instead of just looking at how to solve à

It also meant that I found variations to some solutions that normally wouldn’t be implemented the way I did so overall I think I am in a place where I am because I don’t have that formal background in computer graphics but I do have it in electrical engineering.

I’m ready to try things in different ways, I guess of problems differently, I just land on the solution that everyone has landed, I found and I hit some little gems here and there.

Antoine

You’ve talked about ShaderToy which is your main side-project. I would like to discuss the graphic culture and general rendering and then after we’re gonna go to some more geeky and nerdy stuff. Shadertoy is a platofrm I use daily, and realized there is a lot of love and huge respect for sharing. Was it the idea when creating the platform 10 years ago?

Inigo

It was since day -1! I had first created a small ShaderToy and it was called ThatWay in 2009 with the first WebGL implementation that Firefox released. I reached out to people I knew and I got them to send me shaders by email and I would publish them myself in that website. I was hunting somewhere from my main websites URL but I had to ask people to send me the code by themselves.

Then two years later a friend of mine, Ricardo caballo, Spanish as well, demosceners as well who made ThreeJS , made his own version of ShaderToy his own which was called GLSL Sandbox and that got very popular because people would upload shaders on their own without having to send him emails like they had to do with my side. The problem with his page was I think that there was no attribution, no one knew who was the owner of the shader. I thought that was a pity that the sense of community was lost it was just anonymous people sending often times duplicated shaders without opportunities to talk about the shader, discuss about it. So that’s when I joined with my actual partner and colleague, partner in crime for this business which is Paul Jeremias who works at Pixar now and together we built the new ShaderToy, with community in mind. You need to enter your name and your email to register, we don’t use it for anything, but for letting you know that someone commented on your Shader. This way you can build a solution of shader and you would be able to build a reputation like “This is my name, these are my shaders”. So yeah the idea was to build a community for sure! The fear was will people respond to this proposal will they want to join in when there are so many other communities already! You have the demoscene community with their own websites and portals, you have a trillion things but it did work mostly I think because the first round of shaders were injected by people that I hand-picked before the website was public so I emailed 20 colleagues from the game industry, the film industry, from the demoscene and I asked them for a favor which was to publish two or three shaders in the website. So by the time we opened and went public there was already a good collection of very interesting shaders and I think that was the hook to start getting everyone else to join in.

Antoine

That’s a very nice backstory and does it still evolves nowadays?

Inigo

Paul and I, which we meet every Tuesday night: we have dinner and then we we add new features or fix bugs in shader toy so yeah it’s a slow pace in that it’s not our job it’s not a full-time thing, there is no ambition to grow it or sell it or monetize it or making it into a business although it might have to happen at some point if we want to give better features to the users like storing their own data and so on we might have to pay servers which means we might have to start maybe thinking about how to monetize. Maybe not for everyone but maybe for some users, I don’t know anyway because we don’t have big plans for it or we don’t depend on it to make a living we are taking it easy and enjoying the ride versus having to rush. In that sense we add features as we please and as we find them interesting and the other thing is that there are many features that some hardcore users are desperate to have myself included…

Antoine

What kind of features these people want?

Inigo

We all want to have more control on the multipass system when you write a shader in different passes. We want to control not only wich resolution they are, but also what pixel format they have: integer vs float, or how often they refresh do every pass does run thus every pass run every frame or every other frame or it runs three times per frame? So all those controls have to happen if we want to improve the algorithms but there are other features which are less interesting or less sexy but which are more useful for the broader user base of ShaderToy: like being able to follow creator that you like or getting to get notifications when they upload a new shader or sending messages across users so they can discuss and get in touch for a business or whatever they want to do. So there’s always this tension between what I want as a user and what I want as a product manager.

Antoine

You also have worked at Pixar on offline rendering system, was it Path tracing rendering back then?

Inigo

At the time it was not… I think Pixar switched to full path traced and ray-tracing rendering just three years ago. I left five years ago so no at the time it was rasterization.

Antoine

Oh so all the movies like Cars, they were made using rasterization?

Inigo

Yeah yeah yeah, it’s a variant variation of rasterization where it’s not exactly the same as how the GPU rasterizes things which is mostly in screen space. They split measures into tiny tiny micro polygons as they call them in object space and then they splat those into the screen but it’s rasterization meaning you cannot just cast a ray to query global information about the scene like “is there a shadow here or is there an object blocking the light?” so although you could do it I was very expensive. 99% of things were just shadow maps based or point clouds based to bake ambient occlusion and things like that. I think they kept using rasterization until The Good Dinosaur, wich was rasterized but some shadows were ray traced and then Monster Inc changed to path tracing.

Antoine

You worked a lot on environnement in your career, your work on creating terrain is mindblowing! What is your creative process when creating large environnement and how does it defers from creating characters?

Inigo

On the terrains or landscapes versus characters I suppose which is how you normally divide every team that ismaking a game or film. I’ve traditionally been doing sets or landscapes of terrains or environments because it’s the easiest for me. I grew up in the Basque Country in the Northern Spain, there is a lot of mountains I spent my childhood either hiking or skiing so I know how those things look When I make a landscape, I always work from memory I don’t look at reference pictures or anything! Not because I shouldn’t which is the right thing to do but just for the [came of it], I do it like a painting.

Then characters is something I have never done before a year ago I guess. I took it as a challenge, of course I can keep learning and doing better at landscapes a lot of my landscapes out there are not the best at all became easy enough to do that even refining my process and getting better that it would be a bit of a refinement thing where I would get slightly better with every the more effort I put. I would get smaller and smaller quantities of getting better it would be like a diminishing returns thing. While making characters: it’s an open field which I have never touched so I can do a lot of progress very quickly! Again not that I’m gonna reach a super high level but I can get better than I am right now \very quickly so that’s exciting because it’s a new thing for me.

I have to learn what a face looks like, what are the components that make a face how they move the expressions so yeah I’m taking it a bit of a challenge. I made that yellow jumping guy which is like 5 blobbs you know it’s not very complex but it was animated which I had never done before! I made a boy as well in ShaderToy, procedurally with ellipsoids and parabolas and a few other things and he’s there you’re staring at the camera, but he doesn’t blink. So I have a yellow guy which is very simple but is animated and I have a surfer boy who is complex but not animated and now I want to do both. I want to do something complex and animate it. That’s the shader I’m working on now when I have a bit of time. The process in that case unlike the landscapes and the terrains which I can make from memory is that for characters I definitely have to look online to see how about that other people do characters: both for films, for caricatures and so on. So I try to you watch a few Youtube Videos and see what they have to say about it. I’m also not very systematic as in “I’m going to spend one week now learning about character design” because I like to put my hands to work and get things going and discovering things by my own by

But every now and then I do go online to learn a bit. I would add that there’s a third type of shaders or pictures or paintings or whatever you want to call them, that I do which are just abstract! They are not character or landscape but just something that is inspired by some geometrical configuration that I found while trying to discover some equation or whatever!

Then if that’s the case, if I I’m working on a cubic equation or the SDF of a triangle, whatever it is, then chances are I might take that little discovery and apply it in a shader which just tries to be beautiful or stylish but uses that mathematical concepts. And normally they end up being very very abstract. I also do those ones, although I’m not that good at that!

Antoine

When you work on the SDF function, do you consider it being more art or math?

Inigo

It is a combination of both actually! We could specify for the listeners what an SDF is. It’s a formula and a mathematical expression that encodes the shape and the geometry of some figure like a triangle or a star or a circle. So instead of defining those shapes with vertices of polygons as you would do normally in traditional computer graphics the SDF is a purely mathematical representation, no polygons, it is just an equation. You code those and through different algorithms becomes a shape on the screen. Unlike what some of the people following me on ShaderToy assume, I am not that good at math.

Antoine

What do you mean?

Inigo

I’m not so good at math as in I don’t see in 3d very well actually so there are people who you give them a Rubik’s Cube and they can solve it or they can understand how you should solve it and they can do it or if you ask me what’s the intersection of a plane and a cone when the plane is more than 45 degrees, it’s not immediately obvious to me that that’s a hyperbola or a parabola or an ellipse. I have to write down the things and work out the maths and then I have an answer. It’s not like I can see that much, but a little bit of course.

So when I have to do an SDF most of the times I do the drawing and soon enough I convert it to actual equation that I put in the paper.

The very first times I would go through the process of deriving everything through equations and getting a very complex equation and then trying to simplify it to runs efficiently in the computer. But now with experience I have learn to find the shortcuts before it’s too late, meaning I can find the symmetries of the problem. For example, you know like a triangle if it’s a isocele triangle then the right side and the left side are the same therefore you don’t need to compute both you can just compute one of them and copy that result or the maths to the other side. So those shortcuts I can take them right away now just from experience like you know when you have done a thing a billion times… The is the trick basically.

Antoine

Do your find yourself working a lot the SDF on paper nowadays, or you pretty much derive them all

Inigo

Yeah, every other week or so I have a new SDF and the reason is that I I’m not in a quest really to find or to write the brightest sDF equations but with any new shader thing that I’m writing I oftentimes have to derive a new SDF because I need it for the thing I’m doing for example so it’s always motivated by the picture, or the art I am working on.

So if I’m working on a new character and she’s gonna be moving her face and talking, I needed to sculpt the shape of her mouth and her jaw. An ellipsoid wasn’t giving me enough of a sharp shape to it and a square was too sharp so I had to find new SDF which allows me to do more looking like a rhombus shape and then extrude and round it. So I did the equations… And again, now because I’ve done already so many of them I can do it in a few minutes and in five minutes it is done and then I publish it as an independent SDF and I put it in my website in the catalog of SDF and then I move on to working onthere on the shader.

The driving force is always the artistic thing. Now I’m learning to do characters or trying to learn so that’s the goal and you want to impress people with a character that they have never seen in real time as a mathematical equation in ShaderToy or in the form of a mathematical equation so that’s the the goal.

The process process of going through it: it’s a big mix of maths and also those shortcuts that I take which are either learned by having done so many of the thems or sometimes by intuitions. Sometimes I am like: “It feels like these SDF should be be able to do it with one only one square root” or “it feels like there is too many computations here,in the end of the day this shape is just five lines, so why should I have more than five divisions for example.” So it’s a mic of math and intuitions.

Antoine

How long does it take you to make a complete scene?

Inigo

It depends a lot on the on the painting! Those who comes from already learned process, those comes very quickly. It could be from between an hour to 2 hours, then I publish them. Others who are more more technical, it could be a couple of hours. So in the case of the Yellow Jumping Guy, it was around 10 hours, split between two session on 5 hours. I normally even if I had ten hours in a row which I don’t anymore because now I have a family, while before I could do that not anymore I cannot do those beautiful session of whole day coding. Even if I had the time I wouldn’t because I’m taking a break and coming back the day after to the shader the week after helps a lot to see things that you didn’t see while you were deep into the process of making it!

But the next one I am making now is a totally different story! Because every step of it is different, I have to learn something like just for

Sometimes suddenly, you go to bed thinking “Woah! That shader that I am makingis looking good it’s gonna be amazing” than the the next day is like “Woah, those lights are bad…”. So it’s good to take a break.

But the next one I am making now is a totally different story! Because every step of it is different, I have to learn something like just for the slot in the shape I spent easily an hour just finding what were the shapes that I needed to use to to give good definition to her face. I went online to check how animator does it at DreamWork, what are the muscles around the eye etc… So in the end it might be a total of 10-15 hours I don’t know but, it’s gonna be spread over a month or more, in the small chunks of time because that also gives me time to think about what I’m going with! it because

Right now in this shader, I’m just putting in place the main pieces which is the modeling it’s not finished so you have to put the mouth and I want to change the proportions but I’m 70% there in terms of lighting! I’m I started with generic light rigged that with a key light and a bounce light but now I’m starting to put about a second backlight I’m starting to slowly transform the generic light rig into something that is more motivated by an environment or a setting. For character, I take time to think of the setting: “is she in an office? is she hacking into a computer that she shouldn’t be hacking into or is she making a shader at home?”. I take my time to think about what she is doing, to give a more accurate feeling to the scene.

I like to take the time to think about all those little details, so when I sit down in front of the computer I can actually foccus on turning those reflections I had into code…

Antoine

A lot of your scenes have really different vibes to them, some are more PBR driven, while other have more cartoonish look. Do you follow any rules, such as: how the light should behave in a certain type of scenes, use PBR rules etc… ?

Inigo

So, it’s a good question! I do a blend we’re in I use principles of PBR but not necessarily, but I don’t go full-on PBR correctness and so I would say that even if the style is cartoonish it’s always good to start from some physical principles. So these days, we are already used to working in linear space doing our lighting in linear space and then doing only in the end a gamma transformation and perhaps some color gain control or something but it wasn’t true a few years ago right…

Five years ago this was not knew. So when that thing kicked in and we are all learned of the benefits of doing light in a linear space and how much realistic things look like, I embraced it, right away so now I never again made a shader without a square root in the end for linear to perceptual gamma space conversion. And I never did my lighting in any other space that wasn’t linear and that helps a lot! Because again all the things things look more plausible even if you are doing something cartoony…

Well there are two philosophies here I guess but I’m taking the Pixar and Disney philosophy which is when you make a cartoon which is what they do they keep shapes stylized you don’t go crazy with adding wrinkles to the face or anything. Super simple shapes just enough to convey expressions and feelings, materials are super simple as well they don’t go crazy with how much scratches things have, or if they do they make them very tiny. They keep the surface very clean, but that doesn’t mean that the lighting is not realistic. In the way it bounces around the shadows the shape the penumbra’s, all that is realistic because that helps us understand depths and the relative positioning of things.

I think that’s the reason but that’s there that’s the formula, some other people go crazy with lighting as well and they do super weird things and that’s fine! But I’m taking their approach I think it’s a successful combo: stylized shapes, plausible lighting.

So either you’re doing photorealism stuff or you are doing that cartoony approach where everything is cartoony except for the light. Having lighting that is physically based is a good idea so what I do is I do the basic things of physically-based stuff like linear space and so on… I do more the PBR stuff on the lighting, even for cartoonish scene, but I do not go crazy on it, I don’t know by heart the micro facets models. I just do whatever it takes to make the result pretty.

And actually, oftentimes you might find me doing things where I take the n dot l term which is the basic lighting term for a diffuse lighting situation and I might just multiply it by 0.5 and add 0.5 to kind of shift thing around.

Antoine

I see this a lot in your shaders, the “multiplied by .5 + .5” trick!

Inigo

I mean you can choose those coefficients to decide how much the lighting wraps to the back of an object even though the light source is not hitting there you can lick some lighting in there and fill the darks which is very useful. You could do it by adding more lights to the scene and having fill lights everywhere but that’s easy and looks very looks beautiful and it’s very useful for a cartoonish or painterly situations.

Actually, I saw someone doing that on Twitter the other day and he was like: “Oh this is amazing, a big discovery, oh this is amazing this is what I invented” and I was like that was just since the 80s by everyone who ever made computer graphics although now it’s discouraged. He or she, I can’t remember, was doing it in Unity and we may have a complain here about these game engines, I believe I haven’t used them but it looks to me like they forced you into a physically based world or territory…

Antoine

Unity is now implementing a whole new scriptable render pipeline, to allow custom flexibility! It is very powerfull and driven by Sebastien Lagarde

Inigo

Nice, that flexibility is great, altough, this is a bit of a brainstorm thing now or I’m making this idea to go but that’s great that you can crack open an engine and customize it for your needs if you need game is to be cartoony then do the cartoony shader that we were talking about like adding point five to the diffuse or whatever.

But still means you have to go out of the easy path: which is just using the thing that comes by default and that’s a pity at the same way they have a PBR default I would say, I would love that they had a cartoony default where all these tricks that we all know about: adding .5 to the diffuse, colorizing the occlusion and apply in a color ramp to the penumbra of a shadow, all those things have been used for example by Pixar before they went path tracing and global illumination! The movies I made were full of that because we didn’t have GI right so if a shadow was looking too dark you would go to that shadow and multiply it or add a bit of blue to it and then it looks great, and if it looks too sharp then you would take a smooth step and separate the dark from the bright.

In order to do those things: the lighting artists had the tools! They didn’t have to hack the renderer! They already had all these non-physically based controls ready with nice UI. I wish that these engines had as well, the same way they have all the default PBR’s nicely implemented, that they had another set of non-photorealistic things with all those tools that the industry learned about since the eighties and have them available for those who want to make games that are either cartoony in look or need to run very efficiently say on an Oculus Quest, where every trick you can use to hit a giving look helps.

On Occulus, you don’t have the performance to start casting rays or doing shadow maps or doing the PBR thing. So yes, if I was there I would propose it: “Hey guys, can we put a teram of five people to work on a nice set of tools for those who need something different!”.

Antoine

Do you think the cinema industry is more carefull on following the correct way to handle PBR, and the video game industry is more hacky to hit these high FPS?

Inigo

I would say, I almost feel like it might be the other way around! I’m not in the game industry, I have never worked in a game studio, I have never played a video game and I never used a game engine on a daily basis.

But I have read most of the papers from GDC and all the presentations on the blogs by the game developers that’s how I know Sebastian Lagarde. I kind of think I know what’s going on and I feel like there is this trend where everything has to be correct and not so much there are not so many rooms for hack, at least in lighting and rendering. I’m sure they hack everything else and that’s why I was saying that it would be nice that these tools gave you ways to hack. Hacking is a good thing for the tool not a thing to avoid!

I think that films they are all about hacking! First of all you can hack because every shot is a different thing, every scene is a different set of files. It’s not like in a game where everything has to look good from every angle, you don’t know where the user is gonna be in the film you know these.

So you basically hack everything per shot basically, and a shot is three seconds of film. Every shots has a different set of lights, different tweaks every shader has its own parameters that have tweaks for that one shot, so it’s a lot of hacking! Even from a lighting perspective, even real films not even CG films like you put bounce cards for the lighting to be better in the context. You put a bounce light to fill their wholes, and that bounce light shouldn’t exist in reality there is no floating white umbrellas in the street, walking around. But you hack it, and that’s fine!

So films is all about hacking about hacking and I feel game is trying to be more correct. Which is kind of the opposite of what the intuition should tell us, they cannot afford that kind of realism, films can and they don’t! I mean those who argue that now do you use PBR and path tracing so that’s realistic

But I always think, as I was saying before, that bringing plausible behavior of light which is realistic in the way it bounces in a non realistic situation, which is having bounced cards and things…

Antoine

So there is a lot of hacks in the movie industry!

Inigo

Yeah absolutely! You can decide you know, you have your PBR and path-traced stuff and that helped a lot getting a first look for your shot. So before, when there was a new shot on the film you would have an artist work for two weeks placing five hundred point lights to kind of fake global illumination. And then you could have a first look of the scene and then you could start working on the art like: “where do the highlights go” or “where do the rim lights go”.

But with PBR, physically based lighting and path tracing you hit a button and in five minutes you have something that looks great! So you saved two weeks, but it’s far from being final quality!. So what you do when the show needs to be approved is you have your global illumination solution but then you tweak it and you say things like: “You know what the bounce light is not in her face, is not bright enough let’s make the bounce of object A twice as strong!”. So you can can fake physicality!

Example: the light bounces depends on the incoming angle, the outcomig angle, the PBR, the material parameters! You have the equation and that tells you how much irradiance is getting out or whatever! But in film you tweak it! You say “okay it doesn’t look good enough, let’s times two!”. You multiply everything or that object, if PBR by two, now it’s emitting more light and it’s receiving, physicality is broken but it looks good so…

Antoine

That’s very interesting stuff! I did not knew that. It’s very nice to know this!

Inigo

Crossing my fingers, that I don’t receive an email tomorrow from Pixar!

Antoine

Haha, if there is anything you want to remove before we publish, just let me know! The other day I was making an interview with a guy from Ubisoft, and he was going about very particular algorithms used by the studio and I was like: “Wait a minute, this is not public!”

Inigo

We are good, we are good! What I am saying is about the film industry in general, not from a particular studio! Just tweak, you need to make the picture look good, because everything is in the service of the story, right? Example if the moment is happy, but the shadow in the face of the character are dark and dramatic, those things don’t match so you have to pump it up and put a fake light or a bounce card or just take some reflectance function and make it brighter!

Antoine

Are you a movie fan?

Inigo

I like making films, I don’t watch them. I don’t watch TV neither. I think last year, 2019, I watched only one film!

Antoine

Wich one!?

Inigo

I watched the Pixar release! I have friends working there so they invited me to see the films when they get released. It was Toy Story 4. So I saw that one and then at home we don’t do Netflix or anything like that.

I saw a couple of episodes of The Marvelous Mrs. Maisel. The story is not that strong, I mean it’s interesting, it’s good for keeping us hooked… But the lighting, the costume design and color grading was beautiful. I’m not a story person that much, either.

I like things that looks pretty!

Antoine

If you are curious, about “beautiful things” you should check out the work of Wes Anderson. When you watch his movies, you are somehow more attracted by the look than the story itself. Anyways, we are a bit moving away of rendering and time is passing, so I would like to ask you about VR!

You worked in the VR industry and movie! You worked on Dear Angelica, wich is a wonderfull piece of art… Where do you find the inspiration?

Inigo

This is a team work! In this one, Dear Angelica, in terms of visuals besides the technical aspects which I took care of, the visuals themself are 90 percent the work of the artist who painted it and and the director who had the idea for the story and and led her to paint all those things.

So my role in that movie was VFX supervisor.The idea was: we want to make a movie or a film which is VR meaning is all around you, you are in it not watching it from the distance and we are not going to do something cartoony as in a CG DreamWorks film like we had done before with Henry! We needed to do something that is like illustrations and drawings, and asking ourselves: “how do we do that?”

That’s where I created Quill, and VR drawing and animation tool. So my role there was to provide solution like making drawings in VR a reality and what does it mean to. So we tried things, from having to do the drawings in space, to having 2.5 D drawings you know drawings with depth but that still always looked very CG, you could see that it was 2d drawings in billboards basically.

drawings you know drawings with depth but that still always looked very CG, you could see that it was 2d drawings in billboards basically.

So I brought my role there again was basically enabling the look of the film, for then the artist to go wild and bring their creativity to it. And it turned out to be beautiful, I think.

Antoine

Being a huge movie watcher, I can say that the final result is really doing the thing, I cried in my VR when watching it… What is the main challenge to write VR code?

Inigo

The challenge is clearly performance! How much you can do, especially now that VR is becoming mobile you no longer have a PC necessarily! You don’t have a 2080 by your side to run your program. You have to run on a mobile phone hardware. So i’s really about performances.

There is a few things you can do, the first one that should always be done is: Pick your visual style wisely! Don’t try to do photorealism sutff in VR. And to link it to what we were saying earlier, as engine should provide default not-pbr and cartoony tools and looks. Don’t try to do what the hardware cannot do, if you try, you will end up with something that does not really look good.

If you do, you’re gonna have to sacrifice too many things: the resolution of your shadow Maps, how many of them you have, your specular… Everything’s gonna look ugly and weird, i’s gonna land in no mans land! If it’s not cartoony the viewers expect realistic but then it is not good enough to be fully realistic and you are in a weird place, weird spot, where it looks like a game from the 90s or something.

Antoine

When it’s movie or short films, are all those things precomputed? Example on my Occulus GO, I cannot really move in space, except from turning my head in 360 degrees, do you pre-compute a lot of the data that are rendered or is it all real time

Inigo

No, it is in real time, because the user can be anywhere. I mean, with the GO, wich is the one you have, you don’t have positional tracking, all you can do is to look around. So in that case, you could pre-compute things. You could render the whole movie in a cubemap or in a stereo cube map, but that’s the past!

Now VR is positionally tracked, meaning you can move anywhere! Not only that you can walk, meaning you can stand up from your chair and explore every part of the film, but also meaning that if you are sitting on your chair and moving your head left to right, you need an effect of parallax. So you have to render in real-time.

So the tricks: again the first one choose your style and then I would say secondly hack the rendering techniques that work for that style! You have to render polygon spin there’s no way around it, but maybe things in the distance can be rendered with billboards or 2.5 D billboards or as point clouds or maybe, you can render the left eye and reproject it on the right eye and only fill the gaps of the reprojection. But this may only works for flat surfaces because you can easily fill the gap, so if your style is cartoony and everything is shaded flat you can do it but maybe if you had a lot of texture maybe that does not work anymore!

So again: everything start with what you want to see visually and work the techniques, the rendering algorithms and the shaders backwards to what is the artistic goal. And hack a lot!

Antoine

Do you think using analytical formulas or ray-marching can help to reduced cost, example on the shadows?

Inigo

Traditionally if you want a soft shadow you have to render a shadow map and dopercentage close filtering, and sample it many times. You totally cannot do this on a VR device, it is way to expensive. But if you replace, for example, the object with sphere, than you can compute exact shadow in a very cheap way, only with a few multiplications and one division or something.

I gave a talk at Siggraph about this, wich is my philosphy. Sometimes it’s worth replacing the problem wich is casting the shadow of my character by another simpler problem, wich is casting the shadow of spheres instead of a character. Then solve that problem analytically and perfectly.

So it’s like: making an approximated solution to an exact problem which is a crappy shadow map for a character versus an exact solution to an approximated problem which is perfect shadows for the spheres that approximate the character. And again, I think a lot of these solutions has to be done these days. Example, if you have a mobile device, and you are rendering at 90 FPS you have 11 miliseconds to render your scene twice.

Luckily the latest device, like the Occulus Quest is not 90 frame per seconds, it’s 72, I think.

Antoine

Do you still do VR stuff?

Inigo

No, I am taking a break! I am foccusing now on ShaderToy and videos for Youtube, and live streaming.

Antoine

You have moved freelance few months, do you like it, do you miss the industry?

Inigo

Well I am not doing so much work, I am taking a small vacation, altought I took a couple of small gigs. And I don’t know, I mean, not working is awesome!

I know it won’t last forever, especially here in San Francisco, it is really expensive, this place is good to burns some money.

For the industry: I miss it and I do not miss it… I miss being in a team, like when I was making Quill, the impact of the work we were doing was huge. We were shifting the industry of animation, to start using VR tools, so that was exciting. But when you lead a team, you have to do things that may not be your cup of tea.

While what I do know, on a daily work is great, I get immersed in my shaders!

Antoine

Thanks a lot Inigo! It was such an honnor to talk with you. Are you going to Siggraph this year?

Inigo

I might go, this year is in Washington I think. So I don’t know if I will go, because now with my daugther I don’t really leave home for many days. Washington is a bit further than Vancouver or LA. But I will do Sigraph Asia, probably.

Antoine

Ok, well let’s keep in touch! If you coming to Washington for Sigraph 2020, let’s go have a coffee!

Inigo

Yeah sure!

Antoine

Thanks a lot for joining! Au revoir!

Inigo

Yeah, it was fun, we discussed a lot of things, Thank you! Au revoir!

Leave a Reply

Your email address will not be published. Required fields are marked *