Welcome to episode 30 of Lost in Immersion, your weekly 45-minute stream about innovation. As VR and AR veterans, we will discuss the latest news of the immersive industry. Seb won't be joining us today because he's working, so it's just the two of us. So we'll see if maybe it will be a shorter version of our podcast, but we'll see. We have a lot of things to discuss, so Fabien, if you want to start, please. Yeah, thanks. Episode 30. Okay, so we are on September 26, and tomorrow is the MetaConnect conference, where we think that the MetaQuest 3 will be released. And there have been a few leaked videos and rumors about the details of the MetaQuest 3 hardware, and most importantly, the quality of the Mixed Reality passthrough mode. So there's a bit of background, there is a high expectation on this device, because it's supposed to be the most affordable Mixed Reality headset currently on the market. So we can see here that if this video is accurate, it's actually quite impressive. The quality that can be seen through the... Let me play it again. So you can see that application that's called CamFire. It's a collaboration working application. If you look at the video, which is the passthrough video, it's very difficult to see the usual distortions that can be seen on a passthrough video. So again, if this is really accurate, I think it's very encouraging to see such a good quality on the MetaQuest 3. So we'll know more tomorrow. That was the main thing that I wanted to discuss, is the quality. And related to that, if the quality is actually matching the expectation, I'm curious to know what you think is... Has the MetaQuest 3 the potential to become really like a big hit? Maybe, I don't know, even more than the MetaQuest 2, because it will be the first affordable AR, MR device. And how the mainstream users will adopt this technology. Because for now, most of the normal users have only used VR instead of Mixed Reality. So yeah, I'm curious to know what you think on these two points, and we can discuss that. So this is the first time I've seen this video, because I went through this. But yeah, as you mentioned, there are lots of expectations towards Meta, as it would be the first new headset release after the announcement of the Apple Vision Pro. So of course, the mindset of the global community and people in general is really oriented towards Mixed Reality slash Augmented Reality. We know that it was not specifically the target for Meta at some points to do this kind of Augmented Reality representation. It came from the fact that people modified the Guardian on the Quest 1, 2, and the Quest Pro as well. And we know that as it was not meant to do Mixed Reality, the quality and distortion was pretty bad, to be honest. So this is the first time that we'll see what the dev team of Meta can do with this Mixed Reality goal in mind. So very interesting to see that the... Well, first of all, the first video, the very first video we've seen, we saw about the Quest Pro with Mark Zuckerberg, was completely oriented towards the Mixed Reality experience. So we can hope that they did all they could for the headset to work that way. Indeed, the video is very interesting. We can see that all the basic collaboration tools that we've seen in VR for the past years are present. It kind of looks like the kind of experience we could have wanted with the HoloLens, despite the fact that there is a keyhole effect with the HoloLens 1 and 2, and there won't be one, hopefully, with the Meta. However, I may be a bit cautious about the final rendering, because I really don't know how they capture this video through the headset. How can they say that it is from the user's eye? I think this is more a spectator view that we've seen on the HoloLens as well. Of course, the result is better because you don't have the limitation of the field of view. Very interesting, but I'm not quite excited now. I'm really waiting for the real distortion and user use case to see if it's working as well as it is shown right now. It's also interesting to see that, I think we mentioned that in the latest podcast, or maybe two weeks ago, it was that Meta has released a video of Meta for Business, so the use of the MetaQuest Pro for Business, and here it's a business use, it's not a game. So, like a collaboration experience. I wonder if it's an intentional leak or not. It's interesting to see that more and more industrial usage of Meta devices are starting to rise. I'm actually pretty excited for the release tomorrow. Yeah, because also this is the first time we've seen real footage of it. As we mentioned previously, they didn't make a lot of communication about this headset, which is very strange at some point. We haven't questioned the fact that the headset was ready at some point, because they didn't show much until now. So, maybe they are doing their Apple way of announcing new products without any much information and doing the big reveals the day of the release. Well, we'll see, but the fact that you are mentioning the professional use case is a very good point. We know that Meta is searching for community and new users. We also know that with their previous headsets, they needed to do something more innovative, because doing a Quest 3 that would be the same as a Quest 2, but just with a little more resolution, maybe better lenses, it wouldn't have been enough to capture or to gain a better or bigger community. So, they had to do this new feature, which is Mixed Reality. And of course, they maybe now know that the professional community is the strongest one, and this is where they could make some real money as well. Because when you have a big player, a big industrial player, liking a device, it would be by thousands or millions of units, and they can make the whole game a whole better by bringing a lot of development power as well to develop new features and new applications for the headsets. So, very interesting to see how the community is welcoming this. I think that we mentioned the expectation, but I guess they don't have to do the perfect headsets yet to be successful, because the expectation is so high towards the Apple Vision Pro. We just need something that works like the Apple Vision Pro, and with good performances, maybe not the perfect graphics at some point. But yeah, I think the bar is not too high for them to be successful right now. And we mentioned the Lynx R1 as well, with their missed, I guess a missed opportunity, because they were alone on the market for quite some months now, maybe three or four months. And they could have killed the game at some point. But yeah, they were not able to deliver the high number of headsets that they could have sold. So, that's a pity for them. I hope they recover and be able to deliver large batches of headsets. And it will be very interesting to see the comparison between the Metaquest 3 and the Lynx R1 at some point as well. Yeah, I think if someone is not happy about the timing of this release, it's a Lynx indeed. Because they just started shipping, I think, one month ago, if I'm correct. So, as you were saying, if they were able to ship maybe one year ago or six months ago, they would have a much larger timeline media presence as well. So yeah, it will be interesting to see the hand tracking also on the Quest 3, because the Lynx has a hardware device to track the hands. So, that will be interesting to see. And the controllers as well on the Quest 3. So, hopefully, they are as good as the... I mean, the latency is as good as the other devices because they are not the ring on top of it. So, hopefully, tracking is good. So, yeah, a lot of things to... Probably, we will discuss that next week. A lot of things to expect and to assess which one is the best. And also something that I just remembered is, if we talk about business use case, the device management is very important. So, hopefully, Meta has some kind of capabilities for device management on a large scale. If you want to manage 20, 40, 50 devices, how will that look on the Quest 3? Yeah, especially if you want to do collaborative application as they are showcasing in the previous video and this one as well. Just a bit of information. I saw that the CEO of Lynx talked about their production power and they said that they will be able to deliver all the previous orders of the... It was a Kickstarter, but yeah, it's a collaborative campaign. They will be able to deliver by the end of the year. So, they are targeting December for the first backers. And then they have to address their waiting list. So, maybe they will be able to ship everything by mid-2024, which is very, very late at my point. So, if you are trying to order a Lynx R1 now, you won't have any expectation to get this before mid-2024. And so, if you are in a hurry and if Meta is answering your need, maybe you just have one choice at this point, it will be Meta or nothing. So, yeah, it's a pity, as I said, for Lynx. The other thing, yeah, if we can list all the things that we'd like to... We have a lot of questions about the MetaQuest 3, as you mentioned, the controllers, the tracking, and as well the fact that the cameras are not directly in the same line as URIs. We discussed it a lot during our last, the other podcast. So, we'll see if it is possible to eliminate all the distortion, because we know that a lot of people in the community says it's quite impossible to have a perfect video see-through technology if the cameras are not aligned with the URIs on the hardware part. So, very interesting to see if Meta developers find a solution to reduce this distortion and produce a high-quality rendering. So, for my part, this is where I'm really, really expecting Meta to provide a good software application. And, yeah, the distortion will be the main part of this device, to know if it's something that you can work on, or if it's not. So, this is my main expectation for this headset, because, as I mentioned, I would like to try the LinkSR one, so I'll be waiting for some reviews as well. And the choice is between those two headsets for my next hardware device for me to work on. So, we'll see in the upcoming weeks to see which one I'll prefer. Yeah, we have one, so Seb will test it for the next few days and weeks, and we'll get a long topic on that, I think. So, keep you posted, we'll have a LinkSR one review in the following weeks. So, for my part, I'll be showcasing what I've done on the brand new Gaussian Splitting algorithm. You may have heard that some people consider it as a new revolution for 3D scans, so I tried it, and I would like to present you what are my feedback and reviews about this. But first, I just have to launch a bit of console prompt and so on. So, just while I'm doing this, maybe, could you give us your insight about the Unity situation? We heard that they kind of backtracked on a lot of things that they announced for the licensing fee and licensing costs, especially the kind of very weird way of counting the installation and making the developer pay for the user that is installing and then installing and then installing. That could have been very bad for especially small to medium studios. We know that the VR community is especially impacted, would have been especially impacted with this new way of paying some fees and licenses. So, do you have anything to say about this, Fabien? Yes. So, I don't have the exact details of this new pricing, but there are a few things very interesting about it. So, first, the announcement of this rollback and updated pricing structure was not done by the CEO. So, that gives a hint as to how the communication was handled and how the leading team is maybe has lost simply and lost trust from the developers. So, after this blog post announcing the updated pricing, I was pretty surprised and actually quite happy to see that the reaction and feedbacks from the community of developers was actually mostly positive. So, the worries that many studios had to first face many unexpected costs and maybe to have to switch to a new engine, so to change the pipeline and everything that is around that change of technology. So, I think that relief was really seen into the comments, but still I think the trust of the development community into the leading committee in Unity is lost and will take quite a long time to heal. So, yeah, that's my feedback from what I've seen and I've heard from other developers as well is that mostly it's quite a good news. Yeah, maybe I went to LinkedIn and some forums as well and the community is divided into two camps now, the ones that are relieved to see that Unity is making some efforts and then saying that yeah, well, we can continue to work with Unity. So, we'll see what they are doing next. And then there are the other part where, as you mentioned, for them the trust is completely lost. They've seen that Unity from one day to another can change completely the rules without any good communication, if we can say that. So, they are feeling the threat of this. So, a lot of, I don't know if you've seen on YouTube as well, there are now lots of tutorials and also some reviews of what Godot can do. There is a big push on how to convert your Unity project to Godot or Unity project to your Unreal. So, I don't know if it's an opportunity for some streamers or YouTube video creator to create some content and exploit at some point or use the fact that a lot of people are trying to move from Unity to another platform. So, yeah, Unity won't die tomorrow because of this, because they have lots of very strong partnerships, especially with Apple for the Apple Vision Pro development exclusivity. But yeah, there may be some impact of this in the upcoming months and years. This is a global image of the company that is impacted. So, it can mean impact on the commercial side, but also on the employment side, because people are maybe not that willing to join the Unity company when you've seen this kind of miscommunication. And it's not a good message to developers that want to work for Unity. We've seen that here with Ubisoft as well, after all the drama about the toxicity of the workplace. And just for the information right now, in Canada, Ubisoft wants their workers to come back to work, physical work, two days per week. And a lot of people are now afraid of this toxicity to come back because as everybody was working remotely, this kind of situation was not there anymore. And people are, yeah, a bit scared or afraid that it might come back in the next month. So, you can see that the whole image of a company is very, very important. So, yeah, we'll see how it will be done, but Unity also has a lot of pressure for them to make money at some point, because they never did for the past 15 or so years. So, at some point, they will need to make money and make profit as well. So, this is maybe the most threatening part of this situation is that they need to make money fast and need to prove that it's a sustainable company. And with this kind of event, it will be very hard for them to do so in the short term. Okay, so I'm ready. So, I'll share this. So, this is your video. And this is mine. Okay. So, as I mentioned, I tried the Gaussian splitting algorithm. I did two tests because it took me a lot of time. So, we've seen last week that some people use the 360 cameras to make the capture of their environment. So, I decided to try that because I've never done this before. So, I got a Gear 360 camera that I got for $50 on Marketplace. The interesting thing about this, it can do 4K video recording. So, very interesting for the price, especially. So, I did a very quick turn around. I walked around my backyard recording this. It was a one-minute shot of this, of the environment, and then started the conversion and all this joy to make it compatible with the Gaussian splitting. So, just to show you the result, as I'm streaming, maybe there will be some lag, but yeah, it's really, it's a real-time rendering. It's completely fluid on my computer right now. So, the main issues I encounter with this is that when you are doing 360 capture, so you have the number of frames of your video to consider. So, this is, as I said, a one-minute clip. So, regarding how many pictures you want to use, it will create a lot of data as an input. So, my first test was, in my mind, I was like, I'm going all in. So, I made the, not the maximum, but yeah, a very large amount of sample of 4K images from my video stream, my video clip. And then I used an algorithm to convert all these 360 images. So, the equirectangular rendering of this, I converted them to normal pictures. So, from each picture of the video, I got eight different pictures of this. So, it's like a cube map or so, a rendering. And those, I converted them with cold map to get the points of interest and the position of the virtual cameras. And then I tried to train the Gaussian spectrum algorithm, the artificial intelligence one. And the main issue I got is that every time I got the memory shortage, well, I didn't have enough memory for it to do. So, just for you to know, my first sample was like 8,000 pictures. Then I went to 2,800. And finally, for my laptop, I discovered that I can't go much higher than 300. It must be the limit. So, this is, I could have, I guess if I had a bigger computer, at some point, I could have done a better result for this. Given that you have two main way of teaching your AI. You have the 7,000K and the 30,000K. So, this is the number of iteration. And for this one, if I wanted to do the 3,000K, it would have taken me like 68 hours of training. And for the 7K, it took me about 24 hours. So, also, this is a very greedy algorithm. And you have to, so, I hope my electrical bill won't be too high for this because it run days and nights. So, I guess the result for this is not as beautiful or perfect as we've seen on other streamers or podcasts as well. But given the situation, if we compare it to classic 3D scan, I think it's way better. And if it's compared to Nerf, Nerf is not real time. And I'm not sure I would have got this kind of result as well. So, basically, the experience is good, especially when you are seeing the trees and so on. The 360 video advantage is that you are getting the whole environment all at once. It's very easy to capture. You don't have to move around your cameras for hours or at least a very long time to get all the different angles. So, the efficiency is here. However, the quality is not because you have a large amount of data and you are limited by your computational power. So, this is the first sample I have. I have another one. I tried something different. So, what's your feedback or what do you think of this, Fabien? Well, it's very interesting to see your feedback and to hear that it indeed takes a lot of memory and time to try that. One question I have is how did you remove yourself from the video? Did you hold the 360 and walk around? Yeah, I had a selfie stick and I put the camera just on top of my head. And by design, if I can say, the fact that the 360 cameras is not filming great at the bottom and the top, well, I'm completely erased from every shot. So, that is very cool. Cool. And I guess the fact that, you know, when it's doing the matching points between all the pictures as well, as I may have been maybe some arm or foot or parts of me in the picture, by doing the matching points, this is erased at some point because it's not there in every image. Yeah. Okay. So, that's it for this one. Just let me switch back there. Stop the other. So, just for an introduction, the other one was done with a classic camera. So, 8K, 8K pictures. And this is, I would mention, it's not a scan, but it's a 3D scan of, like, just telephones. This is some old data I got from previous photogrammetric work. So, let me launch this. Okay. And share here. Okay. And there. So, you should be seeing a telephone right now. And so, this is a 3D rendering. We move just a little bit. So, you can see this is a splatting algorithm. This is not the pictures. And you can see that it's, for my part, I've never seen something like this in NERF or 3D scan. So, very, very, very impressing, this. So, just for some data, I just have 150 pictures for this. So, I took them all around the telephone and the table. You can see that you can read the text, like, quite easily on this. So, it works with classical, well, with normal pictures. It's, to my point, it's less complicated to get a good result with a classic 3D scan when you just, you really have to manage your input data if you want some very good point cloud. This is not the case here. I guess I could have taken, like, pictures all around and it would have gotten me a good result as well, which is very interesting to see is that the algorithm is providing a point cloud. So, it's a .ply. And I tried to visualize it in Mesh Lab. And, yeah, the point cloud is not that dense or precise. And if I wanted to use this point cloud for 3D scan or photogrammetric algorithm, it would have been a mess. It would have, yeah, I wouldn't have gotten anything from this. And it's very impressive to see that from a very few point cloud point, it can get this kind of results. So, it's real time. The next steps, maybe for next week, will be to see how it works in VR. I do other tests as well with the 360 cameras, knowing that my limits is now 300 pictures to get maybe smaller rooms or for my use cases, if I can capture high quality, like, small to medium rooms with this, it would be a game changer for a lot of my works. Okay, that's, that's, that's indeed really, the quality is really impressive. And have you heard or seen any white who have animations, or like using a video instead of image? Oh, as an input, you say? Actually, both is can you use a video as an input? Or can it output an animation like someone moving? Yeah, like in the NERF one, you can. I've seen people doing this. Because it's real time. So, it's just a screen capture at some point of your movement or your keys that you are placing in the, in the, in the 3D rendering. And for video as an input, yes, because this is what I did for the 360 one. Basically, it's taking your, your video and rendering it as images, you just have to specify the frame per seconds. So, as I said, for the 360 input, I had to do like two frames per second. Because it, as I said, it's two frames per second for one minute and then multiplied by eight. So it grows very, very fast. So, the 360 one is a good way of getting the whole environment very quickly. Because you just have to move around your cameras and you've got all the angles. However, yeah, you have to have a very huge computer to do so. When I'm seeing the other example that we presented last week, I guess you have, I'm just working on laptops. So, at some point, it's a limitation that is made by the hardware. However, I have a RTX 3080, which is not that bad. So, on this, yeah, I didn't mention that for this rendering, with 150 pictures, it took about 30 minutes, the whole process. So you can see that when the data set is small or manageable, it's a very quick way of getting your 3D rendering. However, yeah, the 360 one, there are lots of cameras and lots of matching points as well, as the algorithm is taking ages to render. So, I'm very curious to master this, to know how to master this, to know how people got their very, very beautiful shots that they got and they're showcasing right now on YouTube. Especially the 360 capture with the Insta360 cameras that are 8K or 4K for some of them. I'm very curious to know how they managed to get this kind of quality. We know that it's possible with a small amount of data. So, I now have to try it on a larger one and try to master the 360 capture. So, lots of very interesting innovation here. For me, I consider it, yeah, it's a very big step, maybe not a revolution, as some are talking about. It's a very, very big step towards, yeah, a quick and efficient way of getting 3D environment. Now we have to know what we can do with this. We kind of see the limitation with the time of computation and the number of input data that you can use depending on the computer you're using. So, this is one limitation for, I think, but yeah, compared to what we had before, the volumetric or 3D, very classical 3D scan or even the NERF, it's a big, big step forward. One last question is you mentioned about the training process that is quite heavy. What about the rendering itself? Is it something that can run on a standard device? Yeah, here on my laptop, I have all the video running at the same time. Well, the laptop is blowing a lot of air right now, but the rendering itself is very, very efficient. I'm very surprised by it as well. So, the main issue here is not the rendering itself. I'm curious to see how it is managed in Unity as well. This is the next step for me to get it into VR. But no, the rendering is the useless part here. This is mainly the training one and the limitation of your graphical card memory because I have 16 gig and it reaches it very, very quickly. One thing that is very interesting is that it computes the memory needs at the start of the algorithm. So, you don't have to wait like 60 hours or so to know that you would be reaching. It would have been a mess, to be honest. You let your algorithm run for hours and hours and at some point, it just crashed. I never had any crash here. There are lots of messages that are giving you intel of what is going on. So, you know basically the number of hours that it would be taking and the numbers of hours that is still ongoing. So, very interesting to see those feedbacks. I really, really appreciate it. Another piece of information is that it's really not user-friendly at this point. You have lots of different add-ons and parts to make it work all together. It's all in prompts. So, maybe if someone wants to create some user interface, it could be done very easily. But right now, it's not the case. I had the chance to have all these different modules already installed from my previous NERF experiences. So, I already got Cruda, Nakonda, and Coldmaps and all those different plugins and pieces of software. So, it took me not that much time to get it running. Basically, it's the same workflow as the NERF one, but with more information. You have more control of what you are doing. So, it's very enjoyable. And the result is better. You can move around. You can see the great results there. Cool. Nice. So, yeah. Do you have any more things that you would like to discuss about this Gaussian spline? I guess it will be an ongoing topic for the following weeks because it's very interesting to see what you can do and how it can be used in a professional way. Especially, can you take measures? How you can work with collision? I know that you can have some... Can you have like a very low quality or low poly mesh that you can put around? Yeah. All these kind of technical things I would like to explore. Yeah. And how it will compete with the Apple Vision.
