Toru Lin - Embodied Intelligence from Autonomous Experience

Key Takeaways

[music] Yeah, talk about the title of the talk today as embody intelligence from autonomous experience. Um, which I'll introduce more later and maybe go to the third slide. So yeah, I work on robot learning. I am both a final year PhD at UC Berkeley and also a visiting researcher at Meta. And when it comes to my research, I develop real robot systems and with work spanning dextrous manipulation, local manipulation and whole body control. And also I draw from a wide range of data sources and learning based methods. So my work has covered approaches including tele operation, centuroo and learning from videos. And for learning methods, I also work with imitation learning as well as reinforcement learning. Okay. Next you figure out the play. Okay. So for today's agenda as the title suggests um it will be I'll be talking about learning from autonomous experience. So I will start by talking about what exactly it is and why we should care. Um and I link to the core challenges that I think um and how my research resolves some of the challenges. And then finally there will be time for Q&A but also as mentioned feel free to interrupt anytime. So you we've seen a lot of recent advances. Um I think all of us are aware of all the large language models happening. Um also great video generation models as well as most recently a bunch of advances in robotics. Um we're seeing a lot of humanoids doing different tasks like folding clothes, making coffee and making boxes. We also have autonomous driving car running on the road of San Francisco. Next, so the standard recipe today if you think about all this advances is basically converging. Um, everyone is using big data and supervised learning and this is basically mimicking human behaviors. Next, um, but if you think about it more carefully, none of this is actually bitter lesson. Um, and what is bitter lesson? I I guess I don't need an introduction for this. And I also recommend this video between Rich Sudden um and a great podcaster um on why he thinks about this is none of this is beautiful lesson. So I won't go into more details here and I will just mention one important reason why um I mentioned this because what is the bigger lesson? It basically says long-term progress should come from methods that scale effectively with compute. And that's clearly not the case with the current approach that we're taking. If you look at this um survey from the year of 2023, it was already saying that high quality data was running out and it's just even more so the case today. And that is just when it comes to large language models. Um, if you think about the harder modalities like robotics, the data is only only getting more expensive and more scarce. Next, and another thing that I want all of us to think about is how differently humans and animals really learn. So, compared to the currently popular paradigm of learning from big data, we learn in a fundamentally different way, which is from experience autonomously. Um so if you observe how a children or how children learn in general you see a fundamentally different process where there's a lot of active interaction with the world. Um they're guided that by rewards and goals rather than these teleoperated trajectories and in the first place there is no such thing as teleoperated data from another embodiment. everyone of us learn from a very personalized um and very interactive experience with the environment. Um so going back to why the previous approach was not beer lesson and why this one could be another promising way. If you think about it um our own interactive experience with the world can give us infinite end domain data and it basically leads to a chain of neverending learning. And of course this will lead to long-term scalability. It also happens that there exist a natural algorithmic framework that already fits into the story which is reinforcement learning. So I'll talk more about it later today. Next. So now I've gone over what learning from autonomous experiences, why is it important? Um I'll just go straight to the core challenges and my research next. So again um I'm going to talk more about RL um hopefully I don't need to cover more basics of it and I'll just skip this part. So um assuming that everyone have a basic understanding of what RL is um if you think about RL application into robotics there are a lot of natural technical challenges that arise in this process that's also quite different from the case with large language models or videos models. First of all there's the hardware bottleneck. Um this is this means that all the RL application to robotics is bounded by real world time and also real world hardware. One person can only operate one hardware at a time. you can't really do a lot of parallelization for data collection um like how you do with collection of text data and training of text data and that means the data is scarce and it also means exploration in the sense of RL is going to be really expensive because you can't have a hardware just running 24 hours a day um it will just start breaking the environment and also breaking itself and also lastly for robotics there's something interesting that I don't see many people discussing which is that the objective here is hard to define. Um there isn't anything universal like the transformer objective. Um and it's it's basically a question with no clear answer. Next. Yeah. So given these technical challenges naturally um well what's our hope? So our key question is how to build a practical robot systems that only need a small down data can learn from autonomous exploration efficiently and with generalizable task objectives. So this is our vision and now I will talk more about how my research addresses some of these challenges here. Um yeah linking to different parts of my publication which I will go over now. Next. So the first challenge, exploration from scratch is very hard. Um, next. So what is exploration? It just means that in the basic reinforcement learning loop, you need some way to get your own experience to begin with. And that process of exploring the world to get your experience for learning is called exploration. And why is it hard to explore from scratch? Because um for us for children in general we can learn fast from autonomous trend arrows because we have the biological structures and the things that's built in from evolution to protect us for safety and also just guide us for exploration. So immediately we know there are these things these surfaces that we need to touch and there are this kind of motion that we want to use. But for robotics this is definitely not the case. If you just throw in a bunch of random weights and let it start exploring from there, the outcome is going to be catastrophic. So, how do we fasttrack this evolution process in robotics? Next, so one way that one of my previous research work um proposed is to simply collect multiensory policies from human guidance. So in the simple term uh in the simplest term this is basically a very intuitive teleoperation interface that allowed us to quickly collect a lot of data with a complex robot system that come with a humanlike form. Um you can see in this picture we use two UR 5 arms equipped with two multifinger hands so that it basically looks like an upper torso of a human body. Um and given that we um connected with a VR headset top. So basically a one person with no prior experience in data collection can just quickly ramp up and collect 100 trajectories within an hour or so. Uh next so having this setup um give us just really fancy teleoperation capabilities to the extent that we can use the simple VR headset to teleoperate the robot to even play a game on a controller. So now you can see how this tool can allow us to collect a lot of data for diverse household tasks and train manipulation policies from there. Next. Yeah. So this is an example of four policies that we've trained with the data collected on this setup and also with the tactile sensor on the fingertip. Um so all these are autonomous policy um played at 1x speed. Next. Yeah. And building on top of this, we also extend um this setup to perform more challenging tasks on more challenging humanoid embodiment. Um doing things like dishwasher loading and also whiteboard wiping. Yeah. So going back to the challenge that we're tackling as I mentioned um exploration from scratch is not easy on this real robot hardware. So basically with these teleoperation interfaces and policy learning from multiensory data we're able to fasttrack evolution um by just quickly ramping up the robot to these basic command SKUs. So next I'll go into this second challenge of defining a learning environment um and reward um for so for this part I'll actually go into the RL challenges u next so if you think about how humans and animals learn our sensory motor skills really evolve from one a complex and well-designed world and second a clear objective of survival and how are we going to translate it to robots? How do we design a world for them to actually learn SKUs that are useful to us and how do we define objectives that can be numerical optimized other than just saying the simple survival as goal. So in this section I'll present my work um that touches upon practical RL with centuro methods. Next. So, first work I want to introduce um is this publication we have at Carl 2024 titled twisting less off with two hands. Um the TLDDR in this is we train two robot hands to twist bottleless using purely deep reinforcement learning and center. Um and the result is dextrous and robust and generalizable behaviors that is from a scalable approach. Um next, so just an overview of what this whole pipeline is about. Um when I talk about Centtorial RL, so on a high level is pretty straightforward. You have a definition of the design. You have a reward design. You have definition of the task. You choose a simulator you like. Um for example, ISAC gym in our case. You learn the policy somehow in the simulator and then you deploy in the real world. And for this we're completely dealing with lowest level control. So the observation is just proprioception and object state and the action is also the low-level torque commands. Next. So on the first glance it sounds really easy but as I mentioned there are a lot of practical challenges when it comes to engineering this whole world and reward for successfully completing these kind of robotic tasks. And the first challenge is physical modeling. And in this case specifically, a very natural question for physical modeling is how do you model the bottle? How do you model the complex um twisting joints that's on the bottle cap between the bottle cap and the bottle thread? So traditionally there was a really hard graphics problem and in our case we found that instead of directly tackling the faithful modeling problem, we can use a more hacky way um hacky work um by simulating the pressure you get when you try to twist open or twist um or close the bottle up. So here um I'm not sure how familiar the audience is um with Isac Gym, but basically you can define this artificial joint in Isac gym so that it allows you to add artificial force to any rigid bodies of choice and that will allowed us to simulate bottle twisting um without solving the graphics problem. And next, so another challenge that we ran into is how to design reward for this camp task. So in a lot of prior um robotics work, the reward design were a lot more similar uh simp a lot more simple because um first of all the dimensionality of the robots are much lower. People used to use the seven dove arms with a single parallel jaw gripper. But in our case um you can see even though there are only two hands each hands is already 16 degrees of freedom which makes them 32 degrees of freedom in total. Um this is already close to some of the humanoids. And with this kind of highdimensional um robot embodiment designing the right reward becomes a lot more cheeky. So um it becomes even harder because the task itself is not that straightforward like how do you define numerical reward for twisting bottleless and in this work and also in the next work that I will introduce we realized that there's actually a universal formula for defining a lot of reward for manipulation tasks because you can decompose manipulation as basically two things. One is the contact state which means whatever your finger is coming into contact with um some part of the object that you're handling and the other one is object state um which is whatever velocity or position you want your object to be. So by combining just these two terms, the contact reward and the object state reward, we proposed this really general recipe for defining reward for tasks ranging from twisting and later on um you'll see things like handover, picking up and other manipulation tasks. Next um and finally just another slide on the challenges with Centural. Um so as mentioned you need to first train a policy and simulator and transfer to the real world. Um but the biggest challenge here actually lies between the gap um and simulation and real world. For example, um in the real world, we can easily set up a camera to point at the scene and capture videos of whatever um robots and objects that you're handling. But in a simulator, this is a lot less straightforward. But without that modality, it's also hard to reason about the object state. So here, what we propose is to take an intermediate state um to translate between the simulator modality and the real world modality. So the real world here we're using is an RGB camera um with depth and the simulator states we're using are something a lot more simpler which is just the object state um and in this case very specifically is a center of mass of the bottl and the bottle body. So the way we're doing this translation is we take the RGB image um we use segmentation mass to segment out the object and then we use depths and also approximated center of mass of the mass to um calculate the corresponding 3D coordinates in the simulator and for the actual policy that's information that we feed into it and that's how we're able to perform this central transfer. Next. Um, so going back, yeah, that's the overall pipeline and technical challenges that we're able to resolve in this whole pipeline. And the result are these highly challenging dextrous manipulation capabilities that we got from these robots without using any human data or any real world data. Yeah. So if you think about it, this is pretty impressive um with robustness. Yeah. Okay, you can go to the next slide. Yeah, if you think about it, it's pretty amazing that you can go from zero human data um zero real world data to this kind of performance even achieving robustness under different kind of disturbances. Next. And all these objects shown here are also completely out of distribution, which means that these are the kind of objects that the policy during training never saw in a simulator. Um, for a simple reason because how do you even model a EP bottle realistically in a simulator, right? Yeah. So we performed a lot of tests like this and found that with domain randomization our model was able to achieve these out of domain uh out of out of distribution generalization for free um even just with simulator training. Next, and yeah, building on to the work that I just mentioned, um, again, we move on to another more complex platform, which is this humanoid robot. Um, again with multifinger hands and two arms. And we applied the same principles for other things I mentioned um in physical modeling and reward design and then simp transfer. and again resolve the tango challenges there and successfully perform the central transfer without using any human data and any real world data. So again you can see these policies generalized to diverse objects in the real world um even though these objects are the ones that they have never seen in the simulator and yeah feel free to go next. Okay, I will skip this technical part because there's the paper and if you're interested, you can feel free to read more in the paper and I'll just show more results on the capabilities and different generalization and robustness of the policy here. Next. Yeah, this is a bmanual handover task that was particularly difficult to reward design for because you can see a lot of contact, specifically making and breaking of contact happening here. And yeah, that's the definition of a heart manipulation task. Next, lifting of different boxes um of different weights, mass and sizes. Um next, also grasping of a lot of different objects. Next. Yeah. So something that I found particularly interesting was that you can see how the fingers are actually adapting to different positions and shapes of the objects. So even though on the left and the right um these are the exact same objects, you can see the emergence yeah different graphs that the robot actually using to handle them differently when they're in different positions. So this show that the policy is not overfitting by just commanding a power grasp or some other kind of average grasp. It's actually seeing the environment, analyzing it and adapting it as policy online. Next, and also there's this generalization that comes with pre to a lot of objects that are hard to grasp including really small and really out of distribution objects. You can show the right hand side video as well. Next and again robustness against random disturbances. All right. Yeah. So, I actually tried very hard um to the extent that I was really worried about breaking the robot. But again, really impressed that it was able to handle all those disturbances. And just a technical note here, I think the robot was able to get these robustness because we're training in them in the simulator. And in a simulator we can randomize the environment by adding this random forces of different magnitude and that allowed it to transfer the policy to a real world with the same sort of robustness. And going back to the technical challenges that I mentioned at the beginning. So learning in assimilation really offer us a good way to circumvent the hardware issues because as I mentioned you just can't have the robot run and explore in the real world for free um it's going to break the environment and the hardware but you don't have that issue in simulator. Yeah. So now that I've covered the first two parts um one on imitation learning and calibration and the other one on reinforcement learning and symptoou the last challenge that I want to mention it's very specific to robotics is how do we actually build a continuously improving system um next so um so I present two major classes of existing approaches um I think a lot of these robot learning works in today's literature really fall into either one of these categories. Um but we actually don't see a lot of them combining these two and if you think about it carefully I think each of these approach really has distinct advantages and also disadvantages. So for the first approach um is mostly taking a human curated data set. This allowed us to get some really impressive demos at a really fast speed but then it's not really scalable in the long run because it's not bitter or less impute as I mentioned initially and then we have this class of approach which is about RL from scratch um which is about how you define a static reward function and then you learn from the reward function online either in simulator or in the real world but then you run into the core problem of RL which is that is not efficient and being not efficient enough is a killer in robotics because yeah data is expensive hardware expensive everything is just harder than than things that are completely virtual. So going back to our application to robotics um the key question here is how do we enable an actually scalable and efficient continuous learning loop. Can we instead of working on these two approaches distinctly can we try to combine them and have them bootstrap themselves um with human feedback. So here I'm going to introduce um an additional work actually I think one or I forgot whether I put one or two but we can start from the first one here um showing how we can actually combine these two in a quite elegant weight. So first work I want to talk about as hover um versatile neuro whole body controller for humanoid robots. So this is different from the work that I mentioned earlier because it's u complete whole body control and there's no dextrous manipulation element to it but the same principle could transfer. So on a high level um this work is about learning a versatile whole body controller by distilling multiple specialist simulation learn RL policy into a general policy that transfer to the real world and um very specifically I will just mention what this policy do. Um, as you can see in the bottom picture, you have a humanoid. Um, and what is not shown in the picture is that you actually have a human controller behind this humanoid. And a human controller, um, can use interfaces ranging from VR to RGB, video, and other stuff. Um, perform some motion and get the motion faithfully translated to the humanoid. So, that's what the policy does. Next and I think what's the most elegant part uh the most elegant part of this work to me are two things. So one um in this work we realized that for whole body control you can really use trajectory following along as a general reward formulation. So all these policy um have been trained with data not specifically collected for teleoperation but just a generic graphics data set um where you have for example human walking or punching in the air or dancing or doing other stuff as continuous trajectory and the only objective we need is for the humanoid to follow these trajectories. um and that allowed us to train these diverse motions that are commonly available um in daily use. And the second biggest takeaway I have from this work um that I thought is really nice is how you can use an RL policy as data generators to train policy. Um and that's a bootstrapping element that I mentioned earlier. Um so just to zoom out a little bit the overall pipeline of this work um works like this. So first of all you have a set of different RL policies trained with different objectives. Um so for example we have the same motion of dancing or following a specific clip and you can imagine that all these even though all this motion follow the same trajectory following objective the input they get can be very different. It can be for example if you have a VR headset the human command is going to be hat pulses only whereas if you have a joystick the human command is going to be a root velocity on the base of humanoid. However, um even though there are all these commands, the way that we can train a generalist policy is that first we train RL policy separately um that take in different control commands from humans and then we distute them into the same set of imitation learn policy. So in this whole process, the row of RL policy is actually a data generator. So if you want to think about it in a more abstract way, you can think of each RL policy as your human tele operator collecting the data for you. And once you have the data collected from this RL policy, you treat them as a data set and then you train the final policy. Yeah, I'll just let that sink in for a bit and then the next slide. Um and this is another work that takes in uh not exactly the same idea but there's also this element of how you can train the RL policy to begin with and then use teleoperation and imitation learning on top of it to bootstrap the policy into something more powerful. So if you click on the video um you can see that so first we train these dextrous manipulation policy using completely RL from scratch um doing things like screwd driving also on the right hand side um there's this hand tightening a nut so the interesting part of this work is that even though the hand motion the finger motions themselves are purely RL trained the wrist motion comes from top operation and imitation machine learning. So this is another example of how you can use RL policy as a base element to generate the data for you um for things that are really hard to be teleoperated and then on top of it you can have human control to provide high level guidance for example the wrist movement and um things specific to a task objective to lessen the burden on RL exploration and once you combine these two um you basically get a more powerful policy policy or the data for a more powerful policy and again you can um disuse this data or you can train this data into a more powerful policy and the loop goes on. So this is a work that we just archived a few weeks ago um and again I won't go into the details but feel free to check it out on archive next. Yeah. So now I've gone over all these different technical challenges and how different parts of my work deal with this. Um, next I I think I'll just leave it to the audience and feel free to ask more questions and again thank you so much for listening. >> Yeah, sure. Thank you so much. Um, yeah, I think it was a wonderful talk and it was I think really insightful. Um, yeah, anyone's free to come off of mute, ask a question. Um, if you want to turn your video on, that's great, too. Um, I'll just leave the the floor open. >> Hi, just to um question from the last slide mainly because that's the most fresh part in my mind right now. But you mentioned um combining two models, one trained based on policy for the more >> um harder text actions and one train based on imitation policy. So are you using two separate models as well and then combining them later? Uh >> yeah. So the interesting thing is that uh maybe I'll explain a little bit more about the pipeline. So um there's only one model at a time and this is the way it works. So first um we train RL policy to perform these inhand manipulation and in this case it's similar to our twisting policy where you see this fingers moving um to either twist this screwdriver or the nut. So that's one model. Um that's the RL model. And what we do next is that we take this RL model out. Um we deploy them on a real hardware and then we try to control the risk with teleoperation. And that will be um what I mentioned about um in the first part of my work. Yeah. That's no that's not a model. It's just a teleoperation interface. >> Got it. Got it. Makes sense. >> Yeah. And with that process um it give us a way to collect data for a more powerful policy um with both a wrist movement and a finger movement. And that's a data set that we can distute into another brand new more powerful policy. >> Makes sense. Makes sense. >> Uh sorry. Uh okay. Thank you very much. I've got a question. Uh [clears throat] so for for the the second part of the your presentation actually you made mention of domain randomization. Uh so in this lesion the randomization is is in the disturbances on the object. Yes. >> Yes. So domain randomization um what I mentioned specifically was about um how we deal with the robustness um and the disturbances is random forces that we can add to both the object and the human or robot itself. But for the whole training pipeline we also add in other kinds of domain randomization. Um for example we randomize the mass and size and surface friction of the object. uh we randomize the noise and observation and action. Um we randomize for example torque friction. So there are a lot of other parameters that go into domain randomization just to achieve this level of generalization. So, so how do you test for robustness since all of these is already in the training uh uh uh like it's it's already in the training set itself but probably there's no overfeitting but there's no robustness because this has been seen before. >> Yeah. Uh that's actually a good question. Um and I'll try to answer it rigorously. So I think the key uh intuition is that first of all we don't have a simil we don't have a faithful model of the world. So there's actually no way for us to know whether the simulation distribution is actually completely the same as the real world distribution. >> And that's by definition um some generalization happening there because there are going to be objects that are not covered completely um by these distribution of parameters we have in simulator but we don't know for sure. >> Um but I think again that's also a very good point. Um and I think that goes into the philosophical question of what is generalization really? Um because when you if you think about the generalization um yeah hashtag generalization happening in large language models personally I think a lot of it is also overfitting a lot of it is um there is the compositionality and emergent extrapolation but I also believe it could simply because a lot of the input data and output data already exists in the training set >> um and I think it's the same In robotics, if you think about domain randomization, you can also see as a way um of covering as much of the test distribution distribution as possible in the training distribution because if you just keep adding in more domain randomization, you're going to get a larger distribution of robot dynamics of object dynamics and there's a higher chance that your policy will be able to be sent to real transferred. Um but again there's no rigorous way to test it and we just have this blanket term called generalization to call this emerging capability. >> Yeah. All right. All right. Thank you very much. Cheers. >> Yeah. Thanks. >> Good question. >> Yeah. [clears throat] Go ahead. >> Yeah, Duru. Uh, thank you first of all for the great presentation. So, I had a quick question from your first paper. I remember an image where uh you had depicted the different ways of reward functions, right? >> Yeah. Uh I wanted to ask like uh on the left hand side that left hand you me you mentioned two reward functions. So is it like those three reward functions are very localized to the left hand only or is it like will it work the other way around as well? What I mean is >> sorry which break was a little bit lost on which work and which specific reward function are referring to? >> Yeah. Yeah. It's the first paper. So maybe if Google can go there. Uh >> this one. >> No no no. Ah this one but not this page. This is the first one I believe. >> This one. >> Uh, no. Go ahead. Maybe. >> I think this is the first one that >> Yeah, >> I mentioned. >> No, it was that twisting hand. That one. >> Okay. >> Okay. Then the first one and the RL part. >> Yeah. This one. This one. >> Yeah. So, you can see how uh the loss functions here for the left hand is like it says okay, the left hand reference contact wise. So is it like the left hand only uh holds the object and the right hand only rotates the thing to open it >> or can it work? >> Yeah. So in this case we specifically define it this way because we designed our real world hardware setup to perform the task in this way. >> Um but because everything is it's just a matter of symmetry. So you can com you can flip the whole thing. Yeah. You can change the left hand refers contact point to right hand and vice versa. And then you can switch the orientation of the bottle. And that's actually something that we've tried and the policy just work as beautifully. >> Oh, so it kind of work. >> Yeah. So if you just flip everything around, um, basically this underlying principle is completely the same. the training process is the same and it just if you just run a training the model will learn to perform the same thing but in an opposite direction and something interesting that I want to mention is that uh we also try changing the longitude of the reward so by doing that we also we can also control whether you're twisting close or open the bottl because it's basically a matter of changing directions Okay, cool. >> I had one more question from the second paper uh which was uh uh probably the next one. Okay, maybe the haha this one. This one. So, this dropping of object, right? >> Mhm. >> Uh let's say that person, that robot, uh it's dropping an object into a certain space, right? Uh let's say that basket. Yep. Uhhuh. So will it be able to take care of it if let's say the basket is mobile? The basket is moving. Let's say >> in this case no because it didn't appear in the training task. It would be an interesting Yeah, it will be an interesting task though. And I can imagine you need to do some work in setting up the simulator environment. >> Yeah. So I can imagine you can have something like a car um a moving object in a simulator and then maybe your your object tip will be the same. um like instead of chasing a fixed goal position, you're chasing a moving goal position. And the training will probably be harder, but I think it's also doable. >> Yeah. So, it will probably something >> uh sorry. >> Yeah, it'll be an interesting task to try. >> Yeah. Yeah. It'll probably be something like where it has to think ahead of its time and we'll have to >> maybe predict a projectile. Yeah, I think the key bottleneck in that case will actually be the hardware performance. So for this we are running everything in 10 hertz and um I think it'll probably be a max of 20 hertz because um first of all the perception pipeline actually runs fairly slowly and also the robot controller if you run it at a higher speed is also easier to get overheated and just cause dangerous movement. So I think um in theory that task is possible in practice it would just require more engineering work. >> Yeah. Yeah. >> Yeah. >> Yeah. Yeah. Thanks. Thanks for your >> I had one quick follow up on that too. So um the so this specific work for the sim to real reinforcement learning vision based dextrous manipulation >> um this is also RL in SAM correct? >> Yep. >> Okay. >> Yeah. So one one question I have is like how how much randomization did you do for object placement like initial uh initial object placement? Um I think one thing I found uh working with uh with the research group I work with is that uh even sometimes gener generalizing um some of these grass poses just just by moving the object to different places >> on the uh in in the like you know on the desk >> here or on the table here uh introduces like like reasonable challenges like you interesting. Yeah. So if I think of this um I have a very specific set of numbers. So I think it's about 20 cm in X-axis, about 10 cm in Yaxis, and about 10 centimeters of variation in Z-axis. Um and those are very specific numbers because it's also related to the workspace of the robot. Because you can imagine if we try to set a code position that's too far away, the arms simply cannot reach it >> and that will disrupt the RL training, >> right? And the the these are 4A robots, right? And they the you know, typically it's just the upper body. >> Yeah. I'm curious if you ever tried like a crossbody placement. Uh so, you know, left arm reaches across to the right. Um even though it's in its workspace, but the right arm is closer. Um yeah, I'm just curious. >> Yeah, I I did not think about it because it was just very unnatural. Um yeah, but I think it's possible. Um I think the bigger concern I had at that time was um for this model, we did not really factor in the possibility of collision including self collision. >> So I just wanted the motion to stay as far away from any self collision as possible. So I did not try that. Yeah. >> Yeah. These these are expensive. So I guess >> didn't want to break up. >> Yeah. >> Maybe I could ask one more question. >> Yeah. >> So yeah for the collision actually the the collision avoidance is in the robot itself and not in your reward function. Yes. This is what you mean. >> No. No. Exactly. So I think there's only self collision. um penalty for for the sake of like not um well in simulator it's more about penalizing your robot such that it gets to the correct go state faster. It's about performing a task. Um but there isn't an explicit concern of safety during the training process because I imagine if you want to factor in um your simulator training then it will be engineering uh engineering wise it'll be harder because you might want to calculate for example the velocity at which you're hitting a robot and the possible damage it causes and the magnitude of force and so on and so forth. Yeah. So I did not really use any of those information during training and as a result it doesn't really learn to safely um protect itself against collision. >> Ah so so so it's just uh you try to do this by making sure that like in the mechanical setup of the robot it it doesn't happen. This is what you did. Yeah. >> Yeah. Ah great. All right. All right. All right. Cheers. Thank you. >> Thanks. We probably have time for maybe one more question if anyone has one. So like 10 seconds to ask. If not, we can wrap up. >> Hey, one last question again. Yeah. So this is something of a survey I'm doing with uh every researcher I meet. It's like uh with all these um uh you know top models accessible to everybody, what does your daily coding text look like and like what fraction of your daily code do you write manually and what percentage of your code does AI. >> Um sorry what's the first question again? >> Yeah like uh not to over complicate it but like what fraction of your code every day and is written by AI. Hey, so I think the other part was what's your your your coding text stack? >> Oh, I see. Uh, it's my hand and my brain and VS Code. Well, to be fair, I think when when I work on this project, it was cloud wasn't a thing yet. Um, and I was very late to the party. Um, which I regret actually. Uh, nowadays I use cloud a lot. Um, but for this project specifically, it was my hand and my brain and my VS code and maybe autocomplete. >> Oh, that's that's nice to hear. Yeah. >> Well, um, yeah, good last question. Um, Tor, thank you so much for joining us and thank you everybody for the the good questions. Um, yeah. Um we'll have another session in another couple weeks. So stay tuned in the Coher Labs uh Discord group. Um and yeah, if anybody's interested in uh connecting with Toro, just reach out to me and I can connect you and um or you reach out to her directly either way. But um yeah, everyone, thank you so much for joining. Toro, thank you so much for your your presentation. It was super insightful. >> Yeah, thanks for resolving the technical challenge. Hopefully there weren't too many uh nexts that you had to say. >> Yeah. >> Thanks. Have a nice day. Bye. >> All right. >> Thanks all. [music] >> [music]