Welcome to episode 42 and the first of 2024 of Lost in Immersion, your weekly 45-minute stream about innovation. As VR and AR veterans, we will discuss the latest news of the immersive industry. So, hello guys. Welcome to 2024. I hope you had a very nice holiday time and that you are bringing us a lot of news for this beginning of the year. So, Fabien, if you want to start. Yeah, thanks. Happy New Year. So, today what I want to have a look with you guys is a LinkedIn post from the director of product at Niantic. So, Niantic is the editor of Pokemon Go and they are also like FWall, which is a web AR SDK. And Tom Emmerich is the very active voice on AR. So, he did 24 predictions for the year and what I like to do is to select a few of them and we can discuss these together. Let's start with the first one. He says that spatial computing, the term spatial computing will knock off generative AI as the peak of expectations. There is a Gartner peak of technology expectations and he says like spatial computing will lead in 2024. My guess is that he thinks that the release of the Apple Vision Pro next month is going to create that hype bubble. So, I'm curious to know what you think. If you think it's a correct expectation. We can start with you, Seb. Sure, it's definitely a trend. I'm sure it's going to move forward with the launch of the Apple Vision Pro, which is scheduled in early February in the US. So, yeah, I'm sure it's going to be a hype at launch. We will see a lot of communication around it and experience being developed with it. So, yeah, I agree with him that it's going to be a trend this year with this new device that is supposed to be a very high quality and mainstream compared to the Vario, which is only used in business use case. So, yeah, I'm sure there will be more influencers and people talking about it like on mainstream. What about you, Guillaume? Yeah, I don't know about the technology itself, but the term spatial computing is already trending like crazy right now. I think this is Seb that will be speaking about a new headset and they are simply using the spatial computing term. So, I guess we don't have to wait for the Apple Vision Pro to be released to see that effect. Yeah, it's just like the metaverse back in the days. It's a buzzword that is simply completely replacing augmented reality, maybe even virtual reality and on the side metaverse as well. So, I guess the prediction is easy to make because it's already the case. The biggest question is the Apple Vision Pro and other because we know that Samsung also is preparing something and meta as well. So, maybe more for the end of 2024, but we'll see if the video and the release strategy of the Apple Vision Pro will work once all the users, I guess it will be mainly developers at this point and professional, we'll see what they are doing and if their strategy also for the Apple developer, because we know that if you are a certified Apple developer, you can have access to the Apple Vision Pro already for about four months, I guess, and you can go to California to test your app. So, we'll see if this strategy works and if there are a lot of applications available and if people are happy with it, because we all know that the headset is one thing, but the applications are another one and it's a main piece of the puzzle for it to work. Yeah, I think as you were saying, the buzz word and the reality are really two different things and he's doing the comparison between generative AI as a buzz and special computing as a buzz. So, we just need GPT-5 to make a buzz and that prediction will fail, but the reality of the technology is maybe quite different. Okay, next. Prediction number 16, reliving memories become a killer app for mixed reality as it allows consumers to be present in their past. So, again, I think this is really pushed by the special videos of the Apple Vision Pro and maybe also thanks to all the innovations that were made last year with NERF and Gaussian splatting and such. So, yeah, what do you think, Seb? Well, I think we are talking about it since the beginning of the podcast, almost the beginning of the podcast, so that's definitely something coming and we were saying that all the virtual environment needs a 3D creation tool, easy 3D creation tool, and that's definitely coming with the new tools to generate an accurate 3D model of a real object. So, yeah, that's also something I think will be trendy this year. And every month you can see a huge improvement in this technology in the speed you can generate them or the speed you can, the frame rate you can get by looking at them afterwards. So, yeah, it's moving quickly. We are on it since a long time and we can see that it's moving forward faster and faster, so that's definitely something I feel will be also trendy this year. Yeah, we already talked about this. I guess the memory thing is way wider than we think because we talked about the fact that we can now duplicate barely but surely as a complete identity of someone. We saw that some companies are trying to do some clones for people that passed away so that you can still interact with them in virtual world. We know that there are some, there is a large part of the population that are willing to use this and if we think about what you said, Seb, about the possibility for us to generate 3D very easily and combined with gen AI that can be impersonating someone based on a few photos or a few memories and create a new digital persona of someone, we are bringing up to the table all the digital twin of person or people and we know that some tech companies are waiting for this to be a reality. So, of course, it would become something and I think it's way bigger than we are imaging right now. Okay, yeah. It's actually linked to another prediction that he makes about, as you both said, hyper-realistic avatars. Okay, next. Mixed reality energizes the fitness industry as a new opportunity to get people moving at home and the gym. I'm, on my side, a bit curious about this one because everyone who has tested headsets knows that doing fitness, like high intensity fitness, is maybe a bit difficult. Of course, movements, simple movements like you can do on Beat Saber or the boxing apps are pretty cool. So, yeah, maybe he's betting on people actually using that for a short period of time for doing fitness. So, yeah. What do you think, Seb? Well, with the amount of headsets that has been sold now, we can see that there's more and more people having this kind of headset and you need it for boxing or training. It's become, I think, a good opportunity for the reseller of a bike, you know, a trainer bike that you install in your place, in your home. It starts to be really interesting to not only see a screen in front of you with a video that was done a couple of years ago now and it's become easy to install a tablet in front of your own trainer and then bike and feel that you are biking. With the headset now, you can be more immersed. So, I think this kind of experience can be quite awesome and maybe also it can allow you to do a ride in a virtual environment that is nicer than going outside around your place. So, that's interesting. Now, if it's going to be trendy this year, I'm not sure. It's really a matter of having hardware and stuff connected to the headset to really do a physical training that is meaningful. For me, what is missing right now is the interesting stuff to do on your own trainer wearing the headset. That's what can simulate a real experience. So, that's really what will make me wear a headset to do a physical training. Yes, I guess about the virtual reality survey we analyzed a few weeks back, I guess, training and not healthcare, but all of the coaching and sports at home applications are the third application in VR right now. So, it's already something that is noticeable. I guess, once again, maybe Niantic has more information than we have because we know that there was a rumor about Apple making the whole health and sports something big for their Apple Vision Pro. We didn't see that much during their presentation, but I guess they are working hard to make something with this. And obviously, if the Apple Vision Pro and Apple is doing something that is successful on the sports side, I guess, would be a success, of course, because we know that Apple is not into games. They are probably not that much in the VR application as well. They are not pushing it so much and they are trying to work with other studios to make these. So, the only way for Apple to make a killer app would be through the coaching and sports applications. So, I think maybe he's giving us a hint here or he's confirming that Apple is going that way in the upcoming months. Let's see. Yeah. Okay. So, thanks. There are many more. So, 24 in total predictions. So, we put the link in the show notes so you can have a look. Yeah, that was a fun one. That was a fun one. All right. On my side, I wanted to talk about the announcement that has been made for ATS about mixed reality headset. The first thing that was announced is a new chipset from Qualcomm, the XR2 Plus Gen 2, which allows to have more cameras to track the eyes of the user and the face of the user mainly. That's the improvement and also increase the quality and the amount of pixels you can display at the time. And the first announcement was from Sony. It's the – I don't know if they released a name for their headset, but they're talking about special content creation on the headset. And the way they design it, it's quite interesting in terms of how they design the controllers because the controllers, when you are wearing it, you have like this point. I don't know if you saw my mouse on the screen, but this part of the controller is like an index that you can point and have precise things happen at the end point of the controller. And like here, we can see how the user is handling the controller. So, it's really different from what exists out there right now. And they are focusing only on the business use case for this headset. So, quite interesting to see Sony moving forward in this market and maybe improving a lot what they did with the VR2 headset for the PS5, which is not selling that much apparently. So, yeah, it's interesting to see that they are going in another direction like all the other players going back to business use case with a headset that is more expensive than PSVR2. And yeah, it's interesting to see that they're going in another direction, although the way the headset is done is really as the other headset done the same way. But the camera seems to be in front of the eyes, so maybe less distortion on screen. So, that was the first one. There is also the X-Reel R2 Ultra which is a new glasses from X-Reel. They did not announce which processor they are using in this one, but it's supposed to be at $700. And they are talking about a competitor to the Vision Pro, which seems weird for me because it seems like it's really a small glasses. So, I don't know how they can fit the amount of hardware that is inside the Vision Pro. Inside the Vision Pro. We'll see. I guess we'll have some feedback on use case and usage at CES. And the last one is from Phantom Technology, which the headset is called Cassie. And they're talking about a headset only for AI, mostly special AI assistant to help you, I guess, with business use case too on detecting object and guiding you on what to do wearing the glasses. So, yeah, a lot of new headsets are being announced there at CES. And mostly right now they are talking about real use case. So, here we can see the user asking to know where he left his button. And I guess... You're not sharing the right window. Yeah. Right. Sorry. So, I was thinking here we see a user wearing the headset and asking for where he left his button in the room. And so, the headset is guiding the user to where he last saw the button. So, it's kind of a headset understanding what you do at any time, any point, and guiding you. And you can ask a question directly to the AI. So, it can reply to you and you can be guided in doing some interaction with the real world. So, we talked already about adding AI to the mixed reality experiences. And now, yeah, it's coming. So, yeah, I guess it will be really interesting in seeing how it looks inside the headset, what is your field of view, and how fast the AI will reply to your question. So, yeah. Fabien, I don't know what you think about the headsets. Yeah, there are many very interesting things. First is if we look at the big tech players, I think they now all have either released headsets or announced headsets, Google and Samsung, Apple, Meta, Sony. It's definitely something that is on their mind of all the big tech players. What is interesting also on the Sony one is they seem to have, I don't know, Seb, if you have more details about that, they seem to have a ring that you wear. So, it's really interesting to see the, I don't want to say innovation, but yeah, let's say the innovation that they provide on the controller part and how this will be used. So, yeah, very interesting. Yeah, just to note, don't you think that the Sony headset is like a clone of the Lynx headset? They are very look-alike about the design and the fact that the screen can go up. I really like that feature as a developer for us to keep the headset on and just have to flip the screen. It's very convenient. But yeah, it really looks like a Lynx knockoff. So, very interesting to see that their design is used, but I'm very curious to see the XR capability of this headset and about the lenses they are using. So, we will see that. Very interesting also to see that Sony is finally coming to the professional side. I think this is a first for the VR, AR and mixed reality side. So, maybe something as previously, maybe they have more hints about the market and what is going on, or maybe they will push this spatial computing because this is how they are presenting it. So, maybe they are feeling that Apple is leading the way very obviously towards the spatial computing side, and maybe we are simply reliving the 2015-2016 era where all the manufacturers are doing VR because it's a trend and they are not doing much about the applications. So, we'll see if this time is a good one. And very interesting to see that everyone is working this way. And finally, about the AI glasses, of course, it's great, but once again, about privacy and what it's analysed, they are just scanning your whole environment in real time and, I guess, gathering data as well. So, it's very interesting to see the use case of this device. Not sure that the hardware part will be the final form of it, but the software one is quite interesting. And we know that companies that are doing both software and hardware usually are not making it that far and just losing one side at some point. They have to make a choice because it's too hard to make hardware and software at the same time. So, we'll see which side they'll be choosing. Regarding all the competitors on the smart glasses or AR glasses, I guess they will be doing the software one. So, we'll see. Yeah, for me, I want to also mention the data stressing to see that there are really two directions. We were thinking that everyone will go to pass-through technology using cameras to really display that inside the screen or inside the headset. But actually, see-through is still something coming. So, that's interesting to see that it's not out of the way because I still think that right now the cameras are not providing a really good video quality inside the headset, specifically in a dark environment where glasses with see-through, it's working perfectly. So, that's one point. The other thing you were mentioning that the design is the same as the Lynx. I don't think they are using the same kind of optic and lens because the one using Lynx has been bought by Apple. So, now they are owning the technology. Lynx can still use it. Yeah, but Sony is owning the screens for the Apple Vision Pro. So, there could be some arrangement here. Yeah, that's true. But yeah, I wanted to mention again the fact that Sony is going another direction with the controller. So, I think that's an interesting direction. It needs to be tested, of course, but the fact that you have one controller with a finger or precise location at the end of the fingertip of the controller and a ring on the other side, you can maybe have some interesting interaction there. If also you have the hand tracking, yeah, that really can be interesting to see other usage. Now, it will make the development trickier because, of course, the development will need to be specific for this one. So, we'll see how it goes and how useful it is to have this kind of controller instead of the standard one. One word about what we saw about the actual glasses compared to the Apple Vision Pro. I think we are already seeing a similar trend as the generative AI. People who are not experts in the field try to jump on the trend and write articles or posts about it and confuse things. So, my bet for 2024 is that we will see a lot of confusions between terms and comparison between devices that are, as you were saying, totally different. So, yeah, you can get back to me on that at the end of the year. Thanks. I think that's it on that subject. If you want to move forward again. Yeah, just to add, did you see that the Apple Vision Pro will be delivered with a special head strap that is going on the top of your head? So, the first thing is that it is offered, which is very rare for Apple to include something. So, if they are including it, it should be necessary for the headsets, I guess. It confirms that the weight or the weight distribution of the headset is not that good. So, they had to add this extra for people to be using the headset. So, very interesting to see that. So, about my topic. I just want to speak once again about the Gaussian splatting and what is going on in this whole field, because now I guess we can talk about the Gaussian splatting field. It's just not one algorithm. It's a whole ecosystem of research. So, the latest project is LungSplat. And the idea here is to be able to create a segmentation of the Gaussian splatting cloud and make it very, very accurate compared to the ancient methods. So, it's very interesting to see that even if a point cloud-like that is the Gaussian splatting, you can act like it is a real mesh and generate this kind of 3D segmentation. So, I guess the next step here is to add some artificial intelligence to make the AI recognize what kind of 3D object you're working with. So, the whole project is available right now on GitHub. So, I didn't try it because it was published like a few days ago. I didn't have the chance to try that, but it sounds very interesting. But once again, it's a first step towards what we talked earlier of the global generalization of the 3D scan and 3D generation help with AI and this kind of segmentation. We know that, I guess this is a Luma AI. They are very strong on the Gaussian splatting part as well, and they are already proposing 3D segmentation of Gaussian splatting. So, I don't know if they are using something like this or if there will be a grading to this technology. But once you've scanned some object with Gaussian splatting, they are able to do the segmentation, extract only the object. So, very interesting to see that companies, not also researchers, companies are using those technologies to provide new ways of 3D scan and generate 3D content very fast and very efficiently. What do you think? On my side, I wonder what would be the use case. Because if it's done afterwards, because Gaussian splatting is not in real time, what would be the idea? Is it to have occlusion behind object and display a 3D object inside a 3D scene and be able to have an occlusion behind object and see an object, a 3D virtual object object going behind? Or have an interaction with some part of the 3D? I don't know. I don't know if you have any idea on that, on what could be the use case for this segmentation. Well, the first one would be to do the contour of your 3D object. Instead of doing it by hand, right now, the Gaussian splatting and filtering which blobs are integrated or not in your 3D object. So, basically, to extract 3D assets and bring them to another place. Because we know that you can merge different Gaussian splatting cloud and make a whole scene with this. And the second use case I'm seeing is about a 3D object recognition. So, just like the AI glasses did, where can you show me, I don't know, an Xbox controller in your scene and you can extract it automatically or point it in the Gaussian splatting. So, if we think a bit forward about those technologies, we know that you can basically now generate 3D splatting in real time-ish. But I guess it would be in the upcoming months. So, you can add this on top of the 3D Gaussian splatting generation and add all this 3D knowledge or analysis of your environment. So, it would be very interesting. But yeah, we are far away from this perfect solution because it's still very demanding in GPU power. So, it's a futuristic view. But yeah, we can see what the... This is basically what we are doing right now with 2D segmentation that can be applied now to 3D in an efficient way. Yeah. I can see, for example, I don't know, for a building inspection or for industry or usage, like having a drone going very quickly into a zone and then from the Gaussian splatting, being able to query things where are these, is this at the correct place, this kind of queries that might be more efficient in the 3D world than in the 2D world. It's very interesting to see how fast, as you were saying, how fast these technologies are progressing and how accurate this seems to be. If you look at the screen that you are currently sharing, it's pretty impressive. It's a huge use of all the algorithms in 2D segmentation that have been used for years now, and they are all putting those together, applied to 3D. So, very interesting to see the reuse or the use of all the other algorithms and put them with the new technologies and see what they can do with the 3D scan and 3D generation. If you can simplify the way to have only one product and extract that from the point cloud instead of having the table and the whole point cloud around it being displayed and you need to do manually an eraser to remove the point you don't want, is that automatic? They can start to do the way of making a real product in 3D faster. Definitely. So, do you have anything more to add for this 2D episode? I guess the CES is going on, so we should probably have more news next week. Keep you posted. This is a wrap up for today. I thank you guys for this first episode of 2024 and see you guys next week for the next one. Thank you guys. Bye.
