Welcome to episode 65 of Lost in Immersion, your weekly 45-minute stream about innovation. As VR and AR veterans, we will discuss the latest news of the immersive industry. Hello, guys. Hello. So, what's new for this week? Okay, so I have two topics today. The first one is about meta. So, I think... So, I don't have the update on my quest yet, but it's currently rolling out to everyone. I think it's one of the biggest updates in terms of features that they released because it's actually starting to bring spatial computing capabilities to the quest. And when I say spatial computing, I mean in the Apple Vision Pro meaning of the spatial computing. So, currently, before that update in the quest, the apps, the 2D apps that you see, they are only on a kind of like a rail, I would say, in front of you. And it's very limited. So, you can only see them in that space. So, now, they added another feature that lets you take out apps and put them... So, it's only three apps. So, it's not like... Well, it's better than none. And so, yeah, I would be very curious to see how it works, how it tracks. Like, is it staying very nicely and steadily in the space? And if I remove my headset and put it back again, will it stay and appear again at the same position? You know, all of this kind of user experience feature, I would be very curious to test it to see if they thought it through. But yeah. So, that's the main update of the v67. And the other one that was announced a couple of months back is the independent developers games and apps can now be seen in the horizon feed. So, that makes content discovery much easier for independent developers. So, yeah. Oh, something I forgot. I'm just seeing right now on the special computing feature in Meta is they have a focus mode. So, on apps, you can focus on it and it seems to darken the other areas of the view. So, that can be nice, you know, for watching a movie or focusing on one app at a time. So, yeah. That's it for this topic. What do you think, Seb? Yeah, nice to see that they are still making progress every month and trying to embed the new features to make the headset more day-to-day usable and with more content. I wonder if they open up the, because right now you need to scan like a room that is five by five meter maximum. With this functionality, did they remove that boundary limitation so you can go to different places and place different 2D app in your different room that makes sense where you want to use them or get the information that you want? Like in the kitchen, I mean, you know, cooking app always there. And when you go into your room to where you're handling your work, you have your own app there. Yeah. That would be something I would like to test if they did not announce it, but it's embedded or if they did it differently where you have to have a five by five meter scan of each room and do that manually every time. But yeah, nice progress. Yeah. Yeah, sorry. I don't know. I didn't see the details in the videos that I saw and in the website. But yeah, I think you are totally right. It's a nice feature, nice feature, sorry, but is it like really usable? That would be a really difference. Guillaume? Yeah, we can see that spatial computing is the latest trend. Everybody is talking about it and it seems to be the must-have for VR right now. However, I really wonder if this kind of features are really used by the users themselves. I didn't see any enjoyment or global enthusiasm about this. It seems like a more technical, not trip, but yeah. You must have this feature to be like with the top guys in VR or mixed reality if you want it right now. I'm very, yeah, you know, they are putting a lot of effort in this and I don't think this is the answer to the main quest that is to find the killer app that will bring more user and stability to the community through months and years. I found it a bit like lost efforts in there. There may be, yeah, it's just communication at this point. I don't see it to be used. I don't have any example of people that are currently working in spatial computing despite the fact that you have this nice video of people doing CAD models and working on very complex industrial projects. But I don't know anybody that is currently using it despite the fact that they tested it. They are not using it on a daily basis. So, yeah, I find it a lot of effort for just a communication feature unless they found something that will unlock this unfamous killer app that we are all seeking. Yeah, I think that it could become the killer app when, if, when they release the smart glasses that we talked about last week that are potentially coming in the fall. Because, yeah, having a small form factor that can be worn all day long and having apps that are, as you said, Seb, like I have one in the kitchen, I have one in the bedroom, I have one in the living room, you know, all this kind of usage can be really cool. I think they called it Augments. Yeah, the widget you can put in your space. Yeah, that's it. So, yeah, it's more, once again, a concept feature that is developed right now, but it's not on the right device. Like everything in AR, we still don't have the device we need. Okay, yeah, good point. But they proved that they are building a strong platform on which each of their devices are compatible every time they release it. So, yeah, they are preparing maybe the field for the new device so they have apps and contents when it's released. Mm-hmm, okay. Okay, and my second topic for today was, so Luma, we already talked about this company quite a lot in the podcast, you know, they were doing a scan, like from the smartphone, you were able to scan. They did Gaussian splatting or Nerf, I can't remember, and they did as well like 3D object generation. And now, like since a couple of weeks already, they released Dream Machine, which is a video generation platform. And it's actually a very good competitor to Sora from OpenAI because, so first, it's already open to beta for everyone. Sora is still in a private beta. And one feature that is pretty cool is the ability to... A prompt. They encourage you to write the type of camera motion that you want to have. And it seems like it's one of the biggest differentiators for Luma. And so, because you have keyframes, people already had fun with it. So they feed some very famous pictures to Luma and have the video generated. So I saw this quite funny one with all the memes that you know that are now alive. This one's pretty creepy. And so they have also more cinematic capabilities like that. And I forgot to mention as well that once you have generated a video, you can extend it. So it will generate another one taking into account what was done previously. And so I think this is how they managed to generate these longer videos. So yeah, I wanted to try it with my own prompts. But I did too early before the podcast and it's not generated yet. So you know, maybe I will showcase them next week. So yeah, this is a dream machine from Luma. Seb, what do you think? Yeah, it's very, very nice. As for the picture generation, the text is always an issue. We can see it on the sign outside. So there's a couple of things that will be the same issue on this first generation. Of AI tool. For picture, there is a couple of things that are coming. But I think the overall rendering that you get from only two pictures, one in, one out. It's quite amazing. It was great to see Sora too. But here you can really test it. It's in a couple of minutes, you get quite a nice result. So yeah, but it's another way to prompt for new pictures. Yeah, I really liked, as you mentioned, like you noticed the text. And I really liked also that they showcased also the limitation. You see, so the morphing. So when an object kind of morphs into another when it shouldn't be. Text, as you said. Which is called Runway Gen 3, which is specialized in text generation. And you can see very good results on different kind of text animations that you could find on After Effects, for example, like Liquidy or Sparkly text. It can be done through this video generation AI. It has pretty much the same result as Dream Machine, but it doesn't have this keyframe feature. And more about the Dream Machine, as they announced like last week, that you could have like, you can modify certain region of your AI generated video. Like, for example, the character is a woman and you would like a goblin instead of it. You just have to inpaint the area of the character and they will generate another the goblin instead of the woman you generated. So it opens up the capability of modifying your content without having this try and fail method that we had with the image generation AI that is now corrected with the inpainting features that is, like, on every generated AI, image generated AI engine. So very interesting to see that. And when, well, when you are using your own pictures, well, you have what you need. But when you are just using the prompt, there is something that is... It's got lots of questions, because the result is really, it really looks like something that you've already seen in a movie. And you can ask yourself if, what degrees of, you know, copyright infringement or what the part of the movie they use for training that is used inside of this. And it's very, yeah, it makes you ask a lot of questions about this copyright thing and what kind of data they use for the training. So I guess there will be some, maybe not trial, but yeah, lots of questions about how you can use this kind of content. Especially when you're doing it through prompt generation. And we can see that there is this kind of issue with music generation, with UDO and Suno that has been sued by the major music company, because they think that they are not creating something, they are just copying the stuff that they use for training. So in that way, this trial will be very interesting for the future of AI, because it will, you know, tell us if the Gen-I we are using right now is really creating something, or it's just like outputting what it had in the training input. So something that will be, will have to follow up because, yeah, basically the future of Gen-I is in that version here. Okay. Cool. Interesting. I saw something just very quickly as well. It's related, but another topic is NVIDIA, they released a tool, an open source tool for like fake data generation. So, you know, instead of having training on real data, they have a tool that generates fake data that you can train on. So, you know, I'm not really sure exactly how this would be used, but it opens up a lot of new usage for everyone. Yeah, the question of the training data is something that is very important right now, especially when you're doing, you're using data with personal information. There is a lot of work for the anonymization of the data, and it took so much time that some people are saying that it would be easier and quicker to generate them. So this is exactly what NVIDIA is doing here. But it's generated based on what? They still have a model? Yeah, yeah, sure. At some point, it's a chicken and egg issue here. Okay, cool. I think that's it for me. So, Seb? Yes. So for me, I wanted to start with two subjects. One is Gaussian splatting, and one is NERF. Both are still progressing on their own pace. For Gaussian splatting, there is a new paper that came out about spotless splats. Not a tough one to say. And the idea is that you can use the same data as you are using for generating your Gaussian splat, but its AI capabilities is to remove the user and other content around the subject that you are filming. So here, for example, you really keep only the bike and the street compared to the other system where, depending on your position, you get a lot of artifact coming from what was happening in the street when you were shooting the scene. So they have a couple of examples. It's quite impressive the way they clean the scene completely compared to another, the previous version of the Gaussian splat. So that's usually the kind of artifact you fight with, and you have to clean completely your scenes to avoid to have this kind of mesh splat in space that are misplaced. Here, with their cleaning system, that gets much nicer to look at. So I don't know if you want to comment on this first one, Guillaume, Fabien? Yeah, well, it's a big step towards automatization of this kind of capture. For all of us that have played with Gaussian splatting, we know that those artifacts and unwanted splats on the environment, it's something that is very, very important. It's something that is quite a pain, and usually you just have to select a very small area of the rendering to get precisely the object you are targeting. So yeah, I find it a very nice improvement to Gaussian splatting right now. Tom? Yeah, same. All right. I think we say it almost every week, it's amazing to see how fast this is progressing. And the big question is how fast this will start to replace a well-established 3D pipeline. I mean, it already does, I think, on some applications. So yeah. About that, I made some researches why this Gaussian splatting is still not used for geospatial content, for example, or application. The main issue here is that the precision of the Gaussian splatting is not there yet, because when you are doing conventional 3D scans, it attains 10 times ratio in precision, like you are in millimeters for the conventional 3D scan and more in the centimeters field for the Gaussian splat. So they are starting to use this for just the rendering, because of course it's better than a classical point cloud. But once they want the precision, they have to go back to a lighter or conventional 3D scans methods. So what they are trying to do right now in the industry is to merge the two techniques. So it doesn't solve the price issue, but it solves that you have a precise 3D rendering with nice render. So this is the main way that this is a reward they are taking for the use of Gaussian splatting. And the NERF one, I know that you are, it's the next topic, but yeah, the NERF one seems to be on the downside of their evolution, but NERF is older than Gaussian splatting as well. It has been used in the past years and months, and now Gaussian splatting is slowly replacing the NERF one. So we'll see what's going. But yeah, when you're looking at it, Gaussian splatting is very, very young, a very, very young technique. So we can't expect it to be completely used in the industry yet. Although my second subject was about NERF, because there is still some news and updates around it. And this new paper is hybrid NERF, and they are showing quite a nice use case in an industry where they are moving inside the industry and showcasing the different rendering that they get with different techniques. And their hybrid NERF seems to produce much more quality in terms of what you get. I think the best video for showing that is this one. This one. And that's not the one I wanted. That's the one I wanted. So on this one, they compare the frame rate and the quality of what they get. And if you stop the video here and look at it here at 72, 82 FPS, at least you can read what is written on the device. So the resolution, the pixel, and the quality that you get is much more better. So I guess the precision is also way better. And at a decent frame rate to look at. So NERF for me is maybe not over. Maybe there is some progress on both sides that is interesting, depending on the use case. Well, the battle is on because NERF is trying to get the real-time that they don't have and Gaussian splatting have. They have it. And the Gaussian splatting is trying to get the precision and better results that you can get with NERF. So there is this approach that some researchers are trying to merge the NERF algorithm with the Gaussian splatting one, despite the fact that it's actually not the same. So I don't know how they will do this. But yeah, you have their pros and cons and they're trying to have the same nice result in real-time. And yeah, it's very interesting to see this current battle between NERF and Gaussian splatting. You already have two teams there. And they are using the same datasets. Which is great to compare the progress on both, yeah. Fabien, do you have any feedback? Um, not much to add, but yeah, we spoke about like 3D pipeline when we talked about Gaussian splatting. But the use case that they show here, like for industrial maintenance, building inspection, I think like moving from video to 3D, like 3D, like spatial understanding can be a really nice use case for this kind of, like a very fast capture that can be analyzed later. I'm sure like they are already using computer vision a lot, but it would be nice to see actually if computer vision can be used on the NERF or on a Gaussian splatting. Well, can be used on the rendered video, but more in like understanding the spatial data. Anyway, just like thinking out loud. Like providing that to your AI tool and ask them to find the tools in the workshop and something like that. Well, yeah, I think the segmentation is already working, so it's some kind of image processing here. Yeah, yeah, yeah. Cool. All right. And the last news is about Meta that starts releasing its 3D gen AI tool to do text to 3D generation and to actually share the video of their progress. And it's allowing also to have your own, you can generate the 3D and then change the texture and iterate on your texture. So it seems quite powerful and I didn't get access yet. I don't think it's available right now. But as soon as it will be, we'll definitely run some tests on that. And I wonder how they will embed that in one of their experiments on the quest, so you can draw something in the air and generate something with that shape that you did. Could be a great tool for non-3D artists to quickly generate something based on the shape of an object that they have around them. So same, Guillaume, if you want to comment on that. Yeah, once again, we can see that the progress made in 3D generation are great in just a few weeks. If you can remember what we showcased last year about this, it's night and day. We can see apparently the mesh is something like low poly mesh with a nice texture and no molds on it. It's the kind of techniques that were used by Luma AI, for example, and other 3D generation. What is very impressive is the auto rigging, apparently, if it's correct. I don't know why they mentioned the Mixamo one, but yeah, apparently you can do auto rigging, which is something great. And there, apparently, you have animation as well. So very curious to see that. But one thing is that most of the assets that they are generating are quite random. I don't know what kind of data they are using for their training data set, because when you are speaking about 3D generation on the industrial or professional side, I would be seeing something more like furnitures or tools and maybe workstations as well. And it doesn't seem to be the case for those generations. But you would say that on the marketing side, it's better to show pugs in metal than a nice sofa. You can sell more. But yeah, I'm curious to see what kind of objects you can generate and how realistic they are. Because yeah, random assets, you can use them on funky games, but the use of these kind of assets is not that big, I guess. Yeah, just one thing about Mixamo. They use Mixamo for the auto rigging, so it's not in their tool. It's written like this. Auto rigging and animation based on with Mixamo. Okay, thanks for the clarification. No problem. Last time, you were going to comment. Yeah. So yeah, as you said, Guillaume, for industry projects, it's very difficult for an AI to know the exact type of machine that you want to build in 3D. And I see a lot of opportunities, I think, for yeah, as you both mentioned, like games, independent game developers, or also kids, apps for kids. Like they can bring their vivid imagination into life with this kind of experience. That could be a really cool app for, you know, digital playground, sorry, for kids. Yes, plenty of opportunities like Guillaume said, I think. I hope it's not the only, like the perfect one that they get, because it's trained on the same kind of model like this. I hope they have a lot of capability with that. We'll see when it's available and see what we can render with that. Based on our test, the previous generation was more guided on one kind of theme and format, and we were always kind of getting the same 3D output. So yeah, that's it for me, I think. Guillaume, you can move on your subject now. Okay, thanks. Well, I guess if we, one of us, talked about Meta, so this would be the case for me as well, as Metanand, that they would like to bring generative AI to content creators, especially for the Horizon one, because we, maybe some of us, forgot about the Horizon platform, which is still on, and they are still working on it. Unfortunately, they still have the same issue, meaning that they don't have as much users as they would like it to have. So as we know, and we announced, and now it is official that Meta said they are not good as using, as doing games and content for their platform. So they are targeting now small developers, indies, and small studios as well, for them to be able to create something that would interest users to go on their Horizon platform. So Fabien said that it is now easier for us to discover Horizon content. You can see Sfou said that they are building AI 3D generated models for us to create assets. And this is exactly the kind of goal that they are aiming at, is that they want us to be able to create content directly to Horizon and create experience that would be great enough for people to come. So this is not a last effort, but it's maybe nearly to that, next to that. But we can see that this is exactly the same method that is used by Roblox. They are now providing AI generated model. Well, you can generate 3D models like the easiest way through AI. I don't know where they are in their deployment program on this, but yeah, basically they are all relying on the community to create something fun and great. And it is exactly the kind of model that has been used by VRChat for years now. And we can see that is, to me, it's the most successful platform for VR in the chat or community in a sense. So they are just copying VRChat with generated, with gen I tools for the community to be faster and more efficient. So I guess this is a great step towards their, maybe not success, but the improvement of the Horizon platform that they just admit that they are not good at it. And they are now relying on the community to do so. Fabien? Yeah, it's funny. I was exactly about to talk about Roblox. I had a quick look. Indeed, there is this world generation ability. They also have a coding assistant in Roblox. And you can ask questions about documentation on the AI assistant. So we also last week or two weeks ago, I forgot, Seb, you talked about Snapchat. And I think it's kind of similar feature that also is coming to Snapchat where AI is used to help content creators. Yeah, I think it's a, so, you know, like when we were doing AR 15 years ago, we had to have two computers and now anyone can do it using Snapchat. And then we see the democratization of the accessibility to being able to create virtual worlds. So it's pretty cool, I think. Yeah. Tim? Yeah, that's what was kind of what we were expecting. I think then working on that on both AI and VR and MR games. Yeah, definitely. That's something that needs to move on. They need to move on, yeah, quickly. Because other will. And I think Apple is quite risky on that. But they should also announce something quite soon, I guess. Because that's also one of the issues with the Vision Pro is that there is not a lot of content to look at right now. So I think this kind of ability inside the headset could help. Yeah, the only issue I would see with the GNI tools for content creators or the community in a large way is that maybe we will limit ourselves to the simplest use case or games. I don't know how you can create complex interaction or complex assets using GNI. Meaning that I don't want it to be lowered. The level of expectation would be lower. But on the other hand, you can see that most of the successful apps on mobile games or even in VR are often the simplest one. So, well, I guess this is the issue with GNI right now. This is something I thought about. Is that we don't have time to work with the GNI in an extensive way and to see the limit of it. We are always using it for a few hours. You can see that it's working. It creates results and so on. But using it in a real project type generation material is very hard to do right now. Because when you are investing time in the GNI model, you can see that two weeks after or months after another one is better and you have to relearn everything. So it's very, very hard now to know the exact capability of GNI, especially in 3D or video or audio as well. I'm feeling that we are always scratching the surface. But we don't know exactly how deep it is. And yeah, it's kind of frustrating right now. I don't know if you noticed this, but apparently OpenAI is slowing their pace also in the release of their different AI. Maybe it's something that is coming now in 2024 and 2025. And people are now slowing down the pace for us to be able to get some expertise on a given AI and see what you can do and work with it for real. And that's just like, okay, it helps me doing something in 10 minutes instead of two hours. But would it be possible to do the complete job or something more complex? And we don't simply have the time for this. I don't know if this is the same for you guys. Completely the same. We are doing a lot of investigation in AI picture generation. And like you said, every week there is a new model, a new way of doing things that needs to be tested and compared to the previous one. And yeah, that takes a lot of time just for that. And so going to all directions, trying all the things that are coming, start to be hard to follow. Okay. Any last words? Yeah, I don't know if you can re-share real quick my screen. It is one of the generation that I started on Luma has completed. And here I just entered our logo, the logo of the podcast. And I've said something like headbanging on the sound of music. Okay. So you see the results. The results. So, you know, it's a very, very simple test. I didn't put a long prompt. But, you know, looks okay. The background is, yeah. Background is crazy, yeah. Okay, thanks. So that's it for today. We'll see you guys next week for another episode of Lost in Immersion. Thanks. Sounds good. Thanks, guys.
