Welcome to episode 51 of Lost in Immersion, your weekly 45-minute stream about innovation. As VR and AR veterans, we will discuss the latest news of the immersive industry. Let's go. Fabien, welcome, as well as Seb, and please talk to us about your new subject. Hello. Okay, so the topic I want to talk about today is how headsets are understanding the environment they are in, and two related news, one on the meta side and one test that I did on the Apple Vision Pro. So, a couple of days ago, the Reality Lab, so the division at Meta that does all the research around metaverse and XR and headsets and so on, they released a new model. Actually, it's not released yet, they just released the research paper on a new AI model that could run on the headset to help understand the space that it's in. So, it's supposed to work on glasses, so my guess is that the model will run on the phone that it's connected to, or on the Quest, and to have a really, really complete understanding if we are looking at windows, if we are looking at chairs, table, and so on, the walls, of course. So, that will help developers to create new type of experiences, and maybe we can discuss the actual applications a bit later. So, if it works as a showcase, it's a very, very nice model. So, it's not released yet. Hopefully, it will be embedded into the next headset or maybe the next update. I'm not sure. So, that's the paper for Meta. That reminded me that when the Apple Vision Pro started to be released a couple of weeks ago, we saw a video that we found a bit strange that was showing the mesh and the understanding of how the Vision Pro was actually understanding the scene that it's looking at. So, I did a bit of research now that I have the Vision Pro, and it's actually true. There is some debug flag that you can activate when the Vision Pro is connected to the developer Xcode. So, you can see here me testing this in my space. So, there are a lot of things going on, and it kind of takes a lot of time for the Vision Pro to forget things. Like you can see here, I have two or three doors at the place where there was only one. But that being said, I was kind of surprised by how stable the mesh is when you are looking, moving around. Because on the Vision Pro, when an app is opened, there is a bit of like a wobble. It's very tiny, but I think it's there. On the other hand, the mesh itself is very, very stable. So, I don't know, maybe there is some kind of automatic delay that is set up on the window to have a smooth experience. I'm not sure. So, yeah, this is actually pretty impressive, and you can see me trying to confuse it with a mirror, and it gets confused, of course, but that's to be expected. And then a bit later, I'm trying to see how good it is at updating. So, let's see. That was here. So, yes, I'm closing the door. And so, yeah, it takes a bit of time to detect that the door is not there anymore. And as you can see, the mesh, it remembers the whole room. So, my whole apartment is recognized. You can see the other room kind of at the back. And so, I didn't mention it, but you can see the text of the segmentation, looking at is it a wall, is it a ceiling, a table? And I tried a bit later in the video. Okay. So, I tried to turn off the light in the corridor, and after a while, it lost the tracking, as you can see. So, just like, boom. And so, yeah, now it's back when I have light, and I tried to move the chair, and it was actually pretty fast to detect it and to see that it's a seat. And so, yeah, overall, pretty matrix-like experience, but it was very interesting. So, yeah, that's it. So, the two ways that the headsets are looking at our environment one, the Apple one, which is actually already implemented, and the one that Beta is doing, which hopefully will come soon. So, yeah. Seb, what do you think about all this? Well, it's great that it's improving in time and getting better and better. And that's something I tested a lot with the HoloLens 2. And here, what you are showing is that it seems to be much more quicker to update in time. There was the same kind of meshing solution with the HoloLens 2, but they had only one processor for this. I guess having a specific processor on the Vision Pro, taking care of that, meshing the things, and not bothering with the rendering that much, just focusing on that part and doing the segmentation. That's awesome and needed. Then I think you were asking what we can do with that. I think it's a matter of implementing scenarios where you need to instantiate things in a specific place in an unknown room or randomly position something that makes sense in your room on the specific table or on the floor or on the wall, completely depending on your scenario and what you want to tell to your user. And I guess with AI, you will be able also to have random things appearing and replacing your door, your wall. Although, when we look at the shape of the door, for example, it doesn't go perfectly at each corner of the door. I think Meta seems to be a bit more accurate in that, but I haven't seen on the Meta version if they're accurate when you open the door, how fast they can update the mesh and the segmentation to tell you, okay, the door is now in this position. Yeah, I didn't see any moving objects in the Meta video. And one thing that seems very interesting in the Vision Pro is that it seems like your mesh is very accurate and dense. If we look at your bed or even the wall, the amount of triangles that are there, that's kind of amazing. And doing that on the quest, you see your FPS going really down very quickly. Here, do you see any, depending on the scan that you do, do you see any change? No, I didn't see any lag during the process now. And it's always updating in real time, right? Yeah, I think this is actually, yeah, we are not seeing it, but it's always there. Yeah. Do you see a lot, do you see the mesh moving all the time to readjust, or is it done at a specific pace? In the HoloLens 2, it was like this. You were seeing every five seconds or three seconds, depending on your settings, how fast it was beaming and scanning the room and updating the mesh. Yeah, I don't know. I don't know. I don't know. I don't know. I don't know. To me, it seems like always computing, it was super fast, but sometimes it kind of like, as the door near my desk, it kind of didn't pick up that the door was actually now closed. And so it's a closet door, and so it's a bit smaller than a normal door, but yeah, there are a lot of things. So I guess if we were to do an experience there, for example, in my office, and we were used to use doors, there will be issues in the actual experience, yeah. And just as a joke, it's pretty difficult to open the door, yeah, here, because I'm seeing behind, and I can open the door with knowing what's behind. I don't really know how to explain it, but visually, it was a bit strange to do. I know exactly the feeling, yeah. I did it with the HoloLens 2. I did a version of my own place to check out how it will look after doing some reworking, and looking at the wall being removed and seeing another room behind, yeah, it was weird. Yeah. You have to project yourself. Yeah. Okay. Guillaume? Yeah, I was just checking. In fact, this kind of application has already been released on the Room Plan app by Apple, which was made for the LiDAR integrated in the iPhones lately. They already have this kind of feature, recognizing wall, ceiling, floor, and some kind of furnitures for you to have your 3D scan blueprints of your room, of your house. So, this is clearly an extension of this feature, making it completely real time. So, I guess there is room for improvement, as you mentioned. It's not that real time yet. There's a lot of false positives, and it's hard to re-update the mesh. But we can see the strategy of Apple behind that, that is continuing. In fact, it's not coming from nowhere. That said, I guess the Apple Vision Pro part doesn't integrate the AI layer that MetaQuest is mentioning in this paper. So, I guess this is where the innovation is. We've seen in the past weeks that it can be done on the Gaussian splatting as well. There was a paper released in that way. So, I guess this is clearly where it's going, the 3D mesh that is better and better, and the AI layer for us to get very precise recognition of what we are seeing. All for what you can ask is that four years now with the HoloLens and all these devices that can do these 3D scans of the environment, basically all we have is an empty shell of our environment, and we know that it's very hard to create something that has meaning. Or, yeah, you're just filtering with the orientation of the mesh that you're seeing, is it vertical or horizontal, and you're trying to create some content with that. I guess once we have full recognition of what we are seeing, we will be going a step further in the kind of application we can do in augmented reality. This is something that was lacking for the past decades, I guess. So, very, very enthusiastic about this. Really, I can't wait to test these kind of features on the MetaQuest 3 to see how it works, if it's as efficient as they are showcasing right now. But, yeah, I guess we are going in the right direction anyway. Yeah, and I agree with you with the fact that we need to have more precise to know where the keyboard is, where the mouse is, where the phone is, to do more precise interaction with the real environment. They were showcasing some you know, connected IoT as well, when you are switching your light on, and you can see that the light is connected to the object or different elements. So, it must be precise and, yeah, even recognizing the smallest objects that are present in your environment, if you want to have complete interaction with them. Yeah, sure. Yeah, and I would think like, for example, with AI, let's assume that you have a new connected light that you just bought, and you put it where you want, and the device automatically recognizes the brand and connects to the API and automatically set up the time. I don't think we are that far from there. It's a bit futuristic, but, yeah. When you can see how precise it can recognize objects or context in 2D images, I guess it's just a matter of training and time for the AI to be able to do the same in 3D. Okay. Cool. So, I'm not sure if you have anything else to mention on that topic, but maybe that's it. Okay, let's move on to Seb. Yes. So, this week, I wanted to talk about the Lumen headset, which is a glasses, like they say, that empowers the blind. So, the idea is to use the same technology that there is in the HoloLens or the Vision Pro, so tracking the space. And with AI and segmentation, being able to guide someone that is blind in their day-to-day transportation, to help them to move from one place to another. It seems that this company is backed up by high-end companies, so they have funding, definitely, and their headset seems to be quite well-made for a long... with a lot of cameras and with a big battery, so being able to really wear it all day long and being guided all day long. So, yeah, it seems interesting, this kind of direction, too, for this kind of person. They are saying that a guide dog is very expensive to train, and there is not that much... it's not scalable, not easy to have trained dogs done for all the blind people. So, using this kind of headset could help solving an issue for the blind people to, yeah, to move around. I don't know if you saw the news. I don't have to say except that it's nice. They presented this device during the last CES. They had a very small booth hidden somewhere, and it's very funny because back in these days, they were asking for funds because they were telling us that their headset was great, but they didn't have any investors for it to be produced. So, I guess they nailed it during the CES, and they found some backup for them to be able to release the headset because back... it's not that far. It's like beginning of this year. They didn't have this much professionalism in their presentation. It was very startup-like with not much funding as well, and yeah, very interesting to see that they found what they were seeking for, and I hope they can release something. I guess the problem is still the same, unfortunately, with this kind of device, is that the audience is not that big. Maybe they would find some maybe mainstream variation for everyone to use them and then increase their market because otherwise, that device would be very expensive for them to be able to live, basically. But as it is using haptic feedback on the forehead, maybe there will be some application in the classic VR headset world. We'll see if it's possible. Yeah, it's really a very nice technology and a really great idea to use these technologies to be able to empower people. I also wonder, as you did Guillaume, about the price and adoption of such a device. I wonder maybe at some point they will think about a smaller device that uses the smartphone as a powerhouse, to have maybe a smaller form factor similar to the metaglasses, for example, or other glasses like that. Maybe there will be a middle ground, but I would be curious to know, just as a thought experiment, how much it costs to train a guide dog, compared to the price of a headset, and maybe it's not that expensive. Yeah. No, that's what they were saying. I think the glasses is around 3... I need to check those facts, but I think from what I remember, it was like 5k for a dog to be trained, and like 3.5 for the headset. So... Yeah. From what they were saying, yeah, there is a... And also the availability, it would be quicker to get the headset than to get the dog, because you need to wait a lot of time for the dog to be trained before you got it, and then you need to train yourself with him to make sure you understand each other. So, what do you think we will have first? These headsets or an AI that can help dog speak? These headsets. These headsets. And the fourth factor here is they are using a lot of cameras, but I think it's for the precision on the guidance that they give. I'm not sure that with a pin, a small pin that have only one camera, or one small glasses with only two cameras, I'm not sure about the quality that they will get yet. I think here they are using DEVSCAM and exactly the same kind of technology that we find in the HoloLens 2, when we look at the kind of camera bar that they have on it. So, I think to trust this kind of technology, you need to have a precise experience wearing it, otherwise it won't be adopted. So, I guess that's what they are focusing on. I will be keen to test it. I hope they will be at Laval Virtual in France. We'll see. And that's it for me. Okay. Let's just let me check this. So, I'll start with that. Okay. So, I want to talk about stereoscopic cameras. So, as you may know or not, there have been a lot of different models of stereoscopic cameras through the years. However, most of them, really, as are some older models, it's very rare to find them in mainstream cameras for us to get stereoscopic ones. There are some industrial cameras that have been done, and also some 3D, 360 cameras that can do both monoscopic and stereoscopic rendering. However, one of the news is that Canon is projecting on releasing a complete set of 3D stereoscopic cameras. So, I found it very interesting to see that manufacturers like Canon is embracing the 3D, they are calling it as well VR cameras for VR or for stereoscopic rendering or capture. And it has been announced a few months after the Apple Vision Pro that is now, we know, capable of viewing this kind of content. So, I don't know if this is a sign that something is going on and the Apple Vision Pro or if the Apple Vision Pro has some partnership with big manufacturers for them to push this kind of technology and to provide a great way of capturing stereoscopic content for them to see. So, we'll see that. But to my knowledge, this is the first time that a big name like Canon is releasing this kind of very well-thought device. So, you can see this on a classic camera, you can have this kind of lenses. And what I found very interesting is this kind of form factor. Apparently, we don't know if the camera on the side could be flipped so that you can have a 360 camera that would be able to do stereoscopic on 360 depending on the use case you'll be filming. So, anyway, it's a very cool and very new form factor for this kind of device. It's very compact as well. So, it is still on the prototype side. So, I guess maybe next year we'll have a very real device to review or to try. So, I guess this is it. And just as a reminder, during the first wave in 2015-2016 when everything was stereoscopic and 3D, we had this kind of phone. One of them was the LG Optimus 3D and you have these stereoscopic captors in the back. But I guess it didn't work because it completely disappeared. And maybe it will be back on the next Apple phones as well. I don't know if this already is the case. I'm not an Apple guy, to be honest. So, I don't really follow what the new features are on the Apple phones. But yeah, what do you think about this? So, yeah, it's a bit strange, I would say, to see this move. But I saw that in the press release, they are talking about live streaming. So, maybe now that very fast connections are more common, streaming this kind of experiences either 360 or 180 stereo would be actually doable. Something that was not doable maybe like 10 years ago. So, that's one that I see maybe some actual use case there, yeah. Okay. On my side, I think Nikon has done some cameras like that, some 360 cameras that are on both sides. I'm sharing my screen. And so, it's not the first time a big player is going to 360, but I think Canon is always going with the highest quality possible when they release something. So, I hope they will release some shots and some previews of what we can get with this kind of device. Because right now, the Insta... I didn't see a lot of difference between Nikon and Insta360 cameras, for example, in terms of quality of what you can get inside the headset. And I guess the Vision Pro now opens the opportunity to look at the content in a bigger resolution than what we were able to do with the Quest 3, for example. So, yeah, it could be... Because right now, looking at 360 on the Quest 3, for example, is still a bit crappy, I would say. Or they use really high-end cameras with a setup that use like 32 cameras on a specific rig and they compute the video afterwards on a high-end computer. So, it costs a lot to make high-quality 360 videos. So, if those kind of players can provide a nicer and easier way to do this with this kind of quality, that could be great. Yeah. Well, I guess that, to be honest, right now, if you want to do some VR 180 or maybe 360 as well, you have to buy two of these kind of cameras and put them side by side and separated with the IE separation, the IPD. But yeah, it's basically not very efficient and the quality is not there. So, I guess if the separation is built in and the software provided can process all the captured raw data, of course, it would be better than what we have now. But very interesting to see that the 3D is coming back. We made some jokes at some point that the 3D movies and 3D glasses will come back at some point. But yeah, surely it will be. I guess it would be the case. Maybe now that we have 4K, 8K TV, the 3D will be better as well, maybe. But yeah, we can foresee that it will be back at some point, the second wave. Okay, so we have something more to add to this. Okay, so that's it for today. Thank you guys and see you next week for another episode of Lost in Immersion. See you guys. See you. Thanks. Bye. Thanks.
