Dev Community Live: Run OpenClaw Agents Safely - Cloud AI, Zero Data Exposure

Key Takeaways

What is up everybody? Welcome to our first community stream, Dev Community Live. We are joined by some excellent excellent people who are going to tell us a little bit about some cool tech. Uh and those folks are Andrew from Protobia AI and Q from IBM. Welcome guys. Uh so so excited to have you here. We're going to we're going to keep it pretty pretty normal uh for us, which means we're going to launch straight into a demo. And then after we see what we're going to learn about and talk about today, we are going to we're going to actually uh dive into the details. So, take take us away. Show us show us what what we what we have in store. I think you might be muted. Yeah. It it it bit me even even on the live stream. You love that. Um yeah, okay. So, I mean, obviously Open Claw, this is Open Claw dashboard if you haven't uh worked with Open Claw yet. Man, there's so much cool stuff that we're seeing people do with Open Claw um and it's going crazy. And of course, there's a lot of discussion around security. So, I think that um it's a very really interesting time in terms of this particular technology we're going to be talking about and how you can bring an additional layer of security to your Open Claw agents. So, what you're looking at here, this is again the Open Claw dashboard, and we have two agents configured. In this case, one of them we're calling Safe Claw. I'm going to take you to the GitHub repo for that in just a minute. But, uh the idea is that I have a dedicated agent that has access to a model that it can access in a secure way, but not locally. You do not have to have uh be running a model locally in order to secure it. And that's really the magic of Protopia uh Protopia AI. I want to throw that in at the end there. Um so, again, you should check out the website, but what we're going to go through today is I'm going to show you um in this agent uh right now, we're we're using uh VLM IBM tech uh you know, open source uh with Quint 332B. And this with this model, we're using the Protopia technology to connect securely. And I'm going to go into the details of that in just a minute, but let's jump straight into the the uh the GitHub repo. This is uh open source where you can go and exper- experiment with this, use this for your own agents, and [clears throat] um the agents that you have you within your Open Claw setup. So, uh we go into a lot of we we love this cute little uh this cute uh our our little cute uh protected uh Protopia Claw. But, um we're going to jump down straight into some of the the details. So, there's an obvious sort of easy financial data, but the real value, I think what the the real differentiator, the new value that Open Claw introduced to us in terms of agents is a notion of just really a timer or cron, basically an agent that wakes up on periodically or that can react to events. You know, if we just sort of look at where agents were, that I feel like that's the really big shift that happened with Open Claw's people thought, "Oh my gosh, we can put this on on a timer to do things." And one of those things that we're going to be showing you is like say for example, you wanted to have a constantly updating portfolio, you know, what up update your position on where you are in a portfolio, your portfolio. So, we're using public data plus your private data, and you don't want to necessarily send that to every LLM. In this case, we want to say, "I want to be able to use a secured LLM, and I really don't want to run it locally." Now, you could run it locally, but I think that, you know, using more powerful models really brings with it a lot of benefits, and sometimes those models are just too big to run on your local machine. So, let's go take a look at exactly what data in here we're working with. So, I'm going to jump into the examples directory. So, for that particular demo, we're going into the investment portfolio. So, how did I set this up? Actually, I should have gone to my crons first. So, this is how you set these up. When you want to set up a regularly recurring task in Open Claw, we're going to set up we're going to go show you how to work with the portfolio monitor. This is actually really easy to work with. This is the command that you're going to run. Super straightforward. Open Claw cron add, we already provide this as just a shell script that you can execute, but the real the real meat of this is right down here. All we're doing is saying, "Look in this investment portfolio folder." Now, that is within your um the the workspace that you've established on a sort of per agent basis. Remember I showed you at the beginning, we're running with two agents. Each of them can have their own uh you know, workspace, which is kind of the sandboxed environment that they can work on. And in this case, we we in in the demo, we have instructions on how to copy it into that workspace after you have created the agent. Then we're going to go next and I'm going to show you where this task is. So, all we've done is run this to set up the cron job. But let me show you how easy it is to define one of these uh tasks. So, I'm going to go to that same one, this investment portfolio. All you have to do is set up this task MD. Uh you know, it could technically be anything. Just make sure you refer to it with those natural language instructions. And in here, you can see this is just regular markdown, clear instructions uh for the my private data, where it's going to pull from. That was also in this the directory above. I'm going to point to that again, uh just to show you that this is sensitive information. This is my portfolio as a CSV. So, I don't want to have this uh just being sent to any LLM provider. I want to be able to know that where I'm sending this it stays completely secure. And I don't mean just encrypted on the wire, I mean I don't want my LLM model provider to be able to uh retain this data for training, even if I tell them not to. You never know. Uh this technology gives me that, you know, peace of mind that uh they're not training on my personal data. So again, all we did is set up this and it refers to that portfolio CSV there. And that's it. You know, I have the cron that I I configured in the other file I showed you. It refers to this task. And that task refers to the private information that I want to send off to the LLM. Now, what's happening with this data? When the cron runs, now we've integrated um in this particular demo with Slack. You can integrate with uh you know, I I've some of my one of my open claw setups is using Discord. Um you know, you can obviously set it up with your you know, iMessage or text message and so on and so forth. But in this case, we've got Slack. And so, every time this cron runs, and I think we've scheduled it on like a 15-minute loop, um it's going to go look at my data, my private data. It's going to go do a search for um all of the stocks that I have a position on, and it's going to submit this through uh through Protopia's technology to another to an LLM that's secured using their technology, and uh and then give me back my response, which you can see here. So, you might be wondering uh how is it secured? Well, let me just show you um in the command prompt here. So, I'm just going to run a command that I already had. So, uh let me see. Here. Let me see if it'll go to the beginning of the um Am I on my Feel like I I think it's a different tab. When did I Wait, did I How did I switch tabs? Give me a second here. Um Not that one, not that one, not that one. Um cuz I swear it was this one. Just a second. This is how demos work, right? Am I on the right >> live demo gods in action, guys. >> This is man. And you know, you know, I tested this all the way >> And we know the audience loves to see it. It's uh Oh my god, yeah. >> It means it's real. It is real. Um Let me think. Uh SSH. Yeah, this is what I did. So, uh we're we're running this demo. All of the the claws that you're seeing here are running on DGX Spark. Actually it's a DGX Spark that is at Andrew's house. So I'm SSH from Houston to Andrew's and let me see if I can get my I had a a version stating class proxy. Where did I put that? I had a command prompt in my little Here we go. In my chat history. Call out to paste if you don't use them. They're my favorite sort of history. So you could see at at the very bottom here, this is the the command I'm going to make move this up a little bit so that you can see it off the bottom. Um There we go. So I'm going to get make this a little bit longer. Um I need to know as much as possible about this user's user. So so that I can tailor my sales to their needs. Again, not necessarily information that I Wait, this is This isn't going to work because I didn't um uh source my Uh Andrew, is it in this directory or do I need to go into another directory? Uh it should be in the safe claw directory. Thank you. Source. See I had this all teed up. Um v and v and activate. Great. Now do this again. Did it party forget my history? Give me Christmas. Okay. I'm there I'm going to put a little more text on here so you can see what is more of what's happening. Tell me more about this user or so that I can tailor my sales approach to their interests. So, um all this is doing is I wanted to show you the what you're inputting to the um to the uh the Protopia AI technology and what it how it is presenting it to the upstream model. So, let's look at this. I provided this was my prompt. Safe claw job blah blah blah. This is the potentially sensitive text that I um want to send to that LLM. The first stage of processing in any LLM is to embed that text into all of these floating point numbers. So, here is actually what uh we what is computed to pass into a model. That's every single model standard inputs. We got all these float floating point numbers. Um what the Protopia technology is doing is uh converting those you know, adjusting all of those floating point embeddings. This one point number is in the embeddings to a uh a value which is semantically the same to the model but different from the initial embedding space. And what that does is it creates uh text that looks like this. And that is not reversible. It's a one-way hash, one-way transformation. But what's fascinating is that when we pipe this straight into the model, the model can understand it just like we had passed in the original embeddings. However, it's not reversible. So, if my model provider is logging it, they're not going to be able to make anything out of this. So, it's a very interesting situation where they're hosting something and we're passing into the internals of that. So, uh of that model and they can't log it, they can't get anything out of this. So, it's it's a very interesting uh approach. This is called stained glass transform. So, I'm going to show you uh sort of a quick visual of what we just talked what we just did. This is on the Protopia site. So, you can see this is us, the client. This is the stained glass transform proxy, which like I said, that's running either on my Well, in our in our case, it's running on our DTX Spark. And so, it does help to accelerate. This is still a model. It's just a very small piece of a model. And uh so, uh it can run effectively on a CPU, but it definitely benefits from running on a GPU. So, uh shout out to Nvidia for this. But from this point, it leaves my local environment and goes to an upstream inference server. And like I it passes in right here in the middle is where that transformation is, where that uh the stochastic embedding is what we call it, is created and sent into the model, the model handles it, and sends back the response. The stained glass uh transform proxy receives it and then hands it back to me. From the client standpoint, it's like it's just talking to a regular model. As a matter of fact, the API that is presented by VLLM of the model is still just a regular OpenAI compatible model. Uh or you know, OpenAI OpenAI compatible um API. So, all of this is transparent to the client, but we get this really cool effect by separating out these pieces of the model and performing that uh stochastic um embedding, that transformation of the embeddings. We get that really great security. So, um I'm going to pause there. Um Andrew, do you do you have anything you want to correct me on? Uh I I want to clarify one thing, right? So, if you wouldn't mind pulling up the terminal window one more time. So, when we apply Stingray transform, we take your text, we do the tokenization, we do the embedding within that Stingray transform proxy. Like you mentioned running on the Spark. Um those embeddings are what is getting passed into the model. So, that reconstruction down beneath it is not that text is not getting passed into the model. We're working in this much larger input embedding space. And I want to also clarify is that the input embedding space is something that all language models work on. Um the transformer blocks of a large language model, they don't work on text directly. They work on this internal embedding space, and that's what we're sending it. Cool. Good clarification. So, All right. So, what happens now? So, like I said, we we've already submitted this this information. We've got back the secure response. How does this work with Open Claw? Now, I'm going to go back to Open Claw control panel. Now you know that we have this Safe Claw that is configured to use this this model, in this case Qwen 332B, and we are partnering with Nvidia to bring out a new very in the near very near term term term future Stingray transform that is aligned to NeMo tron 3 nano and super, which was just released today. Definitely go check that out. It's a incredible model. Um looking forward to I haven't run it on the Spark yet, but I'm really looking forward to using a quantized version on the Spark. But, so this model is using this in this case, I'm sorry, this agent is using this model. And I can configure my main model to run to use, you know, whatever model that I want. You know, in a perfect world, maybe I would have my main and my Safe Claw using the same Uh, same model that is secured behind Open Claw uh, I'm sorry, by Protopia by Protopia AI. It's up to you. I think that over time we're going to see uh, that it's very common practice to run many many different agents and having them uh, using a model is appropriate for the different workload that they might carry. In this case, I think in the demos that I'm going to show take you through in just a second, you know, we're primarily doing some uh, I think one of the most interesting one is a classification job. And so, we're going to uh, go back to this to the GitHub repo here and um, we just went through example two here. I showed you the portfolio monitoring, combining going out using uh, the Brave tool Brave uh, search tool combined with uh, my my private information that has my my position uh, financially. But, let's go down to the PII scanner. I think it's a really common use case that we we're presented with this sort of chicken and egg situation. If you go back to let's say the situation where I only have, you know, one model, but I'd like to know or one agent, I don't want to send sensitive information to that model, but I still need a way to know which information is sensitive or not. How am I going to do that? You know, I need to be able to at least classify it. And I think that that's a really going to be a really common use case whether you're dealing with which emails do I want to handle a certain way? Which files do I want to handle a certain way? I think that like it with Claude co-work or those tools right now, it's kind of like you just got to trust it. Um, I just don't think that that's the future. I think a multi-agent multi-model future is the most likely. And so, having uh, a a clear class a clear secure classification strategy is going to be I really think uh, very commonplace. So, uh, here is an example of using this secure classification before we decide whether or not we want to use a different model. In this case, we're just bucketing it. Uh we're taking uh a bunch of files which have come from all over the place, whether it's from our email or files on my local file system, just whatever tasks that I have delegated to my OpenClaw and it's running across various it may be text, CSV, JSON, and it needs to classify it before it takes the next step of acting on it. And so, I'm just going to hop over into uh Transmit here just to show these are two of these fold These are actually I'm using um SSH or SFTP'd into this machine. Let me refresh it and see. We've been running this for a little bit now. Um and so, you can see it's it's the job has been progressing. And so, let's go take a look and see if how our classifier is doing or, you know, how Quinn is doing. So, um yeah, this looks like uh this is not PII. So, let's go look at one of these files down here. Yeah, this definitely has sensitive information in it. So, the the classification routine, again, which was all routed through uh through my stained glass proxy to a secure model, and that model is not hosted The full model is not hosted on the DGX Spark. Only the proxy is hosted there. The full model is uh being hosted on Modal right now. This is really And that's part of the way we've set up the GitHub is if you want to get started, uh you can uh use um Modal to to um get going as well. But in Anyway, you can see here um we're we're running the classification task. Let's go back into OpenClaw and sort of see what those cron jobs look like cuz as a, you know, a a decent, although admittedly a little bit confusing, interface. Uh so, I can see here's here at the top of the listing are the various uh cron jobs that I have configured. And then I can click history, and it will filter it down below out of the out of my view for some crazy reason. I'll have to scroll past all my other jobs. And now I get into run history. And uh this was the this was the uh financial history of the financial uh position job. And then down here, did I click on the right thing? Oh, I did. Clicked the wrong one. History here. And then it filters the listing. And I need to scroll down to run history. And here we have this open run chat. So again, this is in the uh Open Claw uh web UI. And you might be wondering why haven't we updated? Well, it turns out that the most recent version has a little bug in terms of the way cron jobs work. We haven't upgraded. I'm sure they'll get it fixed quickly. But you can scroll I can scroll through here. I can see the whole history, all of the details of what this did, all the tool calls that were that were performed. And I can see, for example, how did it move or how did it uh you know, move these particular files and where did they land? Like in this case, what's your what's show? That's not the right one. Also not Here we go. Move. Where did it land? Data Garden PII scanner pending. Okay, it truncated it. Um oh, wait, it's over here. Still truncated. Of course, when I did this in the demo our our pre-prep demo, it it was short enough that I could still see it. But the point important point is it I can see how it moved the data around for me between those two different directories. Uh we also, as I mentioned earlier, that the final output of this we integrate with Slack. So, um I can scroll to here. Here is the output the final output of that job. Um you can see that we just processed five files in this case and um it it classified the types of PII and gives me it's sort of a final um, uh, you know, report. And so, this is just running all day, and we, uh, so, Protopia has this as a shared channel, so, anybody can go in here and sort of see how this is progressing. This obviously for our test purposes, but for you individually running your own private, uh, OpenClaw installation, this is still a very convenient way to sort of, uh, be able to go back and sort of look at a log of operations and and what it did as those cron jobs were running throughout the day. Um, okay, so, back to the, uh, so, I took you through, let's see here, we went through the, yeah, so, that's pretty much everything, actually. We went through how to build the cron, we went through how to build the tasks, went through what it's doing, and how we're leveraging, uh, Protopia's technology to give us that security of knowing, uh, that we're using a secure secured model. So, whether you're working with your maybe data at work that, uh, you you is not safe to be putting on the regular internet or running to a regular model, you can still use a hosted model using Protopia's technology to be able to do these kinds of sensitive operations for that. So, I'm going to pause there. Andrew, uh, what what have I missed? Uh, yeah, I think the important thing here is what has enabled this OpenClaw agent that we call SafeClaw to work is that Singularity Transform and the Singularity Transform proxy, right? So, that Singularity Transform proxy in summary is running locally on a DGX Spark, but it can run on commodity hardware, commodity GPUs. Um, it's running a very small model that is taking in the text, creating these embeddings that are protected, um, that are sent up to some upstream inference server. So, that allows you to use the really big compute that you can find in the cloud through your cloud providers, but you're able to use the data that you wouldn't normally send to those places as a result. And because this transform proxy is OpenAI compatible, it's compatible with Open Claw, it's compatible with other agentic frameworks. Uh pretty much anywhere you could use the Open A SDK um or anywhere you'd use the LM, you can also use this proxy um as well. So what what what would be like uh any any I I feel like, you know, Protopia needs more model hosting providers to partner with and model creators. So what would be your ask of anybody Anybody on the call who has anybody with any relationship or work with any of those? Yeah, for sure. Um we have a page on our website. Um it's Protopia.ai/safeclaw. Um if you could bring up the link. And then that if you go to that link right there, um you will be able to vote for what model hosting providers you would love to see this technology be compatible with as well as any um models or other frameworks up there. Um there's also a drawing for a DGX Spark um that's there. Um so if you participate in that, you'll be able to be in that drawing. And then um on there is also a link to the repo that we were able to see on GitHub today. Um so when you put in your information on that page, you'll basically get a sign up that you'll be able to get access to a version of the demo that we've seen today. All right. We have a question in the chat and uh you know, it's from uh Muhammad Salman Hussein and the question is, "How do you validate the same class embeddings are truly not invertible against reconstruction in membership inference attacks?" Big fancy words, uh but the important part, "While keeping model accuracy and inference latency acceptable." So how do we like uh like, you know, there's a classic There's no free lunch, right? Like how close to a free lunch is something like this stained glass approach? Uh sure. Yeah, so um given only observations of the transformed data um on the on the compute provider side, the research that we've done, and you can see some of the preprints that we put on archive about that. It shows that techniques that could, for example, reverse DP noise mechanisms are basically no better than a nearest neighbor reconstruction um against stingless transform embeddings. Um, that said, our stingless engine product, which is what we sell that lets you create stingless transform for a target model, as you use that engine, you're able to configure the SGT characteristics, and you're able to control that trade-off. Um, so for a stronger obfuscation, stronger protection, you can trade off a little bit with um the model performance. Or you can you can decide where that is at when you create a stingless transform for a particular model. And like, this is maybe a silly question, okay? So, so bear with me. Uh, like if I get this if I get this uh get this string that looks like, you know, I I I definitely agree you can't read that string. I don't I don't know what's going on. Uh, like is there a world where I could just churn it through a bunch of models and like hope one of them understands it? Is that like uh is that is that something that I can do? Or is there a method that means that's not a viable uh way to to try to understand what the what the the garbled message is? Uh, sure. Yeah, so that string, I want to clarify one more time, that string is not what's being sent to the to the model, right? What's being sent are those protected embeddings. And you could try throwing it at the model providers. Um, one of the cool features of our stingless engine, like I said, you can do different configurations and create different stingless transforms, is we're able to create a stingless transform that's tailor trained for a particular deployment of a model. So, if I have a particular model that I'm giving to a compute provider, I can create a stingless transform so that it is kind of coupled with that particular model um on the thing. And so, you're not just going to throw it at um whatever model you pull off of hugging face. So, the genius plan of just shoving it through a bunch of models is not going to is not going to do anything useful. >> Yeah, we we we can counter that one. Yes. >> good thought, but we can counter that one. And so so like okay. So, that's great. I mean it it it you know, we we can make sure it's etc. etc. It's protected, but is it is it still fast? Like is it does latency degrade tremendously? Is there like uh what what what's what's the trade-off in in that regard? >> Yeah, so the the latency is not very big. Um, so it's a very small machine learning model that's running locally. Um, like I said, we were running this one on a spark and it was running fine, but it also runs on commodity hardware. Like we test this regularly on 3090s, 4090s, or faster. Um, what we find is that for most models, um, the transform itself is in the order of milliseconds of additional latency. And I want to say that's a one-time cost. So, when you send a request, you do the transformation on the prompt. Usually that's significantly smaller, like way less than 1% of the actual time for generation. Um, so like if I'm sending my model off to a giant quant model in the cloud, I take the one-time cost of my local stainless transform, my generation might be 10 seconds or plus, but my latency is an additional 10, 20 milliseconds if that. So, basically like we we may see a little bit of latency cost, but because it's for just this initial payload, it's not it's not like every token that gets sent. We're we're adding this this this this time time loss. So, that's >> Yes. Yeah, your throughput should not be affected. Excellent. Excellent. We had a question from Alexsanya Senke, curious about how you approach execution control for autonomous agents. Is there an external control layer? Do you rely on guard rails inside the model? I think this is more about like the agent itself as opposed to specifically the the ProtoPie AI technology, but what how do you guys think about that? Yeah. What do you think about that one, Q? Yeah, I mean, I think that Yeah, this is an area where I think we've been seeing a lot of research being conducted in into um mainly around jailbreaking and things like that. And you know, I I'm I feel like you're probably leaning towards, you know, when you say if something goes wrong um I mean, there's a lot of things that can go wrong and um I think that, you know, first and foremost, we we we have a like a multi-agent strategy, I think, is taking a lot of as getting getting the most traction, basically saying having agents that watch agents. It's a supervisor pattern, basically. And say, okay, um these agents I know are in an isolated secure environment. They can't be tampered with. They're not as susceptible maybe to my main line agents to uh say prompt injection or other kind classes of tampering. And um their sole purpose is to watch for certain conditions that I know or are going to potentially violate, you know, whatever my constraints or requirements are and that they will will stop it. But you know, we we still have to have, if we want to really have the full value of agents, we have to as scary as it is, keep leaning into how do I use use agents to fix problems with agents. And um I think this this is a problem that's recurring not just in like we're seeing this a lot in agent decoding and handling enterprise data, you know, it's one of the areas that we're working on at IBM is that the best way to empower agents to do to be their best actually in many cases looks very different from the way we have handled uh built systems to uh suit humans. And so, we have to think uh like one particular area is complexity. Usually with humans, we go to great lengths to minimize complexity. That's actually not a problem with agents in many cases. So, we can be we can have introduced more complexity to deliver the better results, such as not containing not breaking things. Um so, that's kind of a a long-winded answer. Yeah, I mean listen, it's a long-winded answer because it's too. It's a big problem, right? It's not a Right. I I I fear for the day that we we describe the solution to this problem in in few tokens. So, it's uh you know, it it makes sense. We we we had a stream uh the the other week where we talked about some of this stuff, and it was a it was a 45-minute session all and we and we just started scratching the surface, right? So, uh you know, it it it it's it's interesting. So, this is dope, Andrew. Is this uh something that can exist for a model like NeMo-Tron? Like uh is it is it possible for a Utopia AI to to give these stained glass powers to uh to to to to a model like NeMo-Tron? Yeah. Um so, the I mentioned before that stained glass engine technology. Um that can be used to create something for NeMo-Tron. Um any licensee to our stained glass engine can create stained glass transforms, which you know, could be anyone on this call. They can come and contact me. Um we can we can set something up. Um and any of our professional services providers can also build SGTs for you as well. Um and we have some announcements coming next week at GTC that we're very excited to share in that regard. Love it. Love it. Absolutely love it. Uh at uh Uttara Sawant uh 821 from YouTube asked, "Is there a safety validity expiration time on stained glass transforms?" So, like this is a classic, you know, I got a rotate tokens, I got to you know the the these kinds of things usually come with like like a shelf life. Is this the case for something like stained glass transforms? Yeah, so I keep I feel like I keep harping on this. The stained glass engine lets you create stained glass transforms various characteristics security characteristics. Um we today we don't support this. Um the idea that you can rotate it through. It's something that we've discussed before. It's something that's on our road map. Um it's So, the answer is not yet. Um but it is something that we we hope to support. And like is this is this Another Okay. Guys, this is a Dev Committee live stream with Protobee AI and we're also joined by Q from IBM. Uh I don't know a lot about this technology. So, I'm learning, you know, with you guys. And my question is like uh Is this something that like we would be able to say, "Okay, they've just they've discovered our our system. They've they've they you know they have access to our to our to our protected embeddings." You know what I mean? Like they they they cracked the code or or did the analog. Uh is it something like a simple as we can retrain a new stained glass transform? Is it something like we're kind of we're we're up a creek now? Like what what does it look like if you know, people are very clever about getting around systems. Let's say enough time has passed that the very clever nefarious ne'er-do-wells right get get get their way. Is there like a easy way for us to to remedy or or or pivot with something like this? Yeah, so I mean there's a couple different aspects to that. And yes, I agree. People are very clever with very clever ways of getting around things and it's something we we see every day. Um one of the one of the things I would say I mentioned this earlier, but like we have some that research that there's some preprints on archive you can look at about reconstructability, that that sort of thing. Um a lot of those common or lot that those techniques that we kind of talk about that if you all you have is the transform data, um like if you have a leak of my stained glass protected stuff has been sent elsewhere, that reconstructability, like these attacks that work against, for example, DP noise mechanisms, they don't work against stained glass. Um they're no better than nearest neighbor. Um the other half of that is if you have a stained glass transform that's been compromised, you can retrain a stained glass transform, stained glass engine, we can configure something that lets you get a new one, that way you can recouple it and it's not like it's burned forever. So you rotate your token effectively, right? Like it's uh >> Yeah, you can rotate your stained glass. Yeah. That's And that I mean, this is the idea, right? We want to We want to make sure that, you know, this is this is the from the security stream couple couple weeks ago, right? Like we want to have layers and expect these systems can fail because people are are are are very clever and the idea that it's easy to kind of reset, I think, is a really a re- really important. Uh And and so how does this work with something like uh VLLM queue? Like what's the How are we changing kind of how we we do things like inference backends to to help accommodate this this kind of new newish security layer? Yeah, this is a you know, one of the extensions to VLLM uh enable this kind of, you know, pluggability and and the way that uh Protopia is um cuz we're basically modifying uh the input or the the way that data is input to the model to bypass the first layer of the the model um on the server side cuz the original model is still there. We're just We're just computing part of it over here and then routing it in, you you a couple of deep. In addition, um Protopia has technology that picks it up off the backside and you can have a per user encryption key that decides how that gets sent back to you. And that's just regular standard encryption and it's you know, it's it's configured at inference time. So you could literally technically, you know, reconfigure uh the way that that's sent back to you and dynamically. So vLLM, I think, is definitely setting the pace in terms of um model inference servers. Not just from performance. I think a lot of our work with NVIDIA is around uh making vLLM the most performant inference server available. But uh and it's phenomenal in that way. You know, a lot of the optimizations that we have, I think, you know, Dynamo uh NVIDIA Dynamo by default is using vLLM. And um so a lot of great stuff there. I think that um you know, it's this is a partnership between, you know, companies that are like IBM and open source building vLLM with the model providers and companies like Protopia AI that want to extend the way that we handle uh models, whether it's for, you know, maybe exotic caching strategies or exotic uh encryption or, you know, scrambling, you know, or whatever you want to refer to uh you know, this this really uh novel way that Protopia AI approaches handling input data. And I think that actually in a way of all of the vendors with whom I have worked, the Protopia's pushing vLLM, you know, pretty pretty far to the edge in terms of securing the input and output to the model. So and I think we're going to continue to see some really cool innovations in vLLM. Again, with those the partnership with those those communities. Mhm. And I and I also want to say I I've done some contributions to vLLM as an open source developer. Um some of it in relation to these prompt and embed stuff and I've had a bunch of time to work with different contributors at vLLM. It's an amazing piece of software. I'm super thankful for the devs that put in time to that. Some of them IBM employees, some of them not. Um because VLM was the first of the platforms to support these prompt embeds, that kind of started a little bit of a tidal wave to allow other platforms to accept these prompt embeds. Um that make it compatible with the Stained Glass Transform. So, like another one that followed shortly after VLM, partly because of the work that was happening at NVIDIA um in relation to VLM, was NVIDIA NIMs. There are a lot of NVIDIA NIMs now that support prompt embeds and Stained Glass Transform. Um there's a couple other platforms that are built around VLM or inspired by VLM that are now supporting this feature. It's just like I I mean, it's one of those things where, you know, it feels like the constant march of progress, right? Uh for for getting safer, more secure systems is like uh is still ongoing. You know, it's uh we we we've had these technologies for for such a relatively short amount of time. It's nice to see us coming up with novel uh solutions to some of these problems. So, I'm going to do I'm going to pay the slide tax now, guys. So, so actually, I think I might have never actually formally introduced y'all, but uh we're we're joined [laughter] by Q, uh field CTO at IBM, and Andrew, uh AI solution engineer at Protopia AI. Uh I'm Chris as always. Uh we've got some slides to talk about a little bit of the community, I believe, if you want to show that. So, you can reach us at discord.gg/nvidia developer. This is our community Discord. Uh next slide, please, Zach. Uh you can also email Zach. That's the behind-the-scenes legend that you can't see, uh but makes the stream go burr. Uh email him at community@nvidia.com. Next slide, please, Zach. Uh we have GTC coming up. Uh this is I think next week. So, very soon. Uh my understanding is you're going to be able to check out Photopia AI there and in the Inception in the Inception section, which is going to be pretty cool, booth 3520 adjacent sorry to the Inception Pavilion. Very awesome. Next slide, please. Zach. We are also going to have an 8-hour stream on Wednesday. If you can't attend the event live, we want to know what the vibe of GTC is. We're going to be delivering that to you for 8 hours straight on Wednesday, March 18th. Tune in. It'll be fun. If you if you can't make the trip to GTC, at least you get to experience one day. We want to try to showcase what the conference feels like to to people who aren't able to to to make the trek to to the convention center. I believe that's all our slides for this. Last question for for you guys. We've got one from the chat that I'll that I'll that I'll talk about and then I'll ask you one one more question before we let you go, which is I think probably Andrew for you. LinkedIn user asked, "Does it come with a secret key?" This is like your classic pattern. Is this Is this yet another secret key method or is there there's something a little bit different? Yeah, it's distinct from encryption. So, I I hesitate to say it comes with a secret key. Um what the SGL transform is doing is finding an alternative representation that the model understands. So, the model doesn't have to do any decoding. It's model native. It's complementary to encryption. So, like yes, you should be using all of your state-of-the-art encryption over the wire and at rest. Um but at the end of the day in order to get into the language model, the data you send has to be decrypted. And so, the question becomes what is the extra layer of protection I can do in that unencrypted state? So, the model can still get it, but it minimizes my my elsewhere. All about layers in security, right? It's all about layers. With the more layers we have, the less holes we have. And so, uh excellent stuff. ONE LAST QUESTION FOR YOU BOTH, uh just to end it on the community vibes. Uh Andrew, you mentioned, you know, some OSS contributions. Like, what got you started uh walking down the path of uh of working somewhere like Protopia AI and working on uh problems like this. Like, give us a little What what what makes you passionate about uh about this this particular, you know, usage of a particular technology in a particular sub domain of a particular field, right? Like, uh tell us a little bit about it. >> Yeah, the the story I like to tell people, I I used to work in consulting in my last job before this, and I had a bunch of customers that were struggling with this. As we want to run compute in the cloud, but we don't trust the data there in that cloud environment. So, like, how can we do machine learning inference in the cloud, but still keep our data local? And at the time, nobody had an answer. Um Protopia had an answer. Um and it's been exciting since I joined that we've been able to build that answer and make it stronger and stronger. Incredible. I mean, doesn't it doesn't get much cleaner than that, right? Solved problem, found someone working on problem, is now working on problem. Let's go. Uh a classic path. And you, same question to you. Uh you know, uh how did you get into this whole this whole world uh that that is again very specific uh in 2023? >> Yeah, I mean, I've [clears throat] been, you know, in tech professionally for 30 years. I'm a tinkerer, but um man, when when AI, you know, when AI stuff hit, you know, uh with you know, initial job job of ChatGPT, I was like, man, this is this is going to be the biggest wave that has ever hit. Again, I've been professionally in tech for 30 years, right? And I've been through every every, you know, hype cycle thing. Hell, I the internet. I mean, that's how old I am. I went through that hype cycle. And man, nothing compares. Nothing compares to what we we think this is fast. We think that there's been a big shift. We're just getting started. And I I saw that and and, you know, it's we're going to have to re, like I said earlier, we're going to have to re-solve many of the things that we think are solved problems. Such as security. Such as just the basics of how do we use computers as people? How do we work with them? And to me that that there's just that's an incredibly fascinating challenge to be trying to solve and to be trying to help people, help different businesses, large enterprises, and smaller businesses be successful with this technology. And and and actually you have to it starts with having to just sort of like open your mind to rethink and revisit like literally everything. And and and that seems to be very expansionist. It's like, "You, come on. Everything?" Uh it's the truth. I mean, I've seen people I've worked with customers working through like the most fundamentals of their workflows and they're and they're seeing huge gains through a lot of times in those most fundamental things that they're least likely to rethink because they're very comfortable in them. So, tons of things are going to change and I'm just I I I I don't know if everybody's has seen, you know, the movie uh Forrest Gump at one point. There's uh the Lieutenant Dan is like on the top of the ship and he's screaming it's like, "Is this all you got?" Uh that's how I feel every day and, you know, I kind of like it. So, that's me. I mean, we couldn't end on more inspirational words, guys. Uh it is we are just getting started. I think, you know, these kinds of communities are important to keep uh driving us forward and to keep, you know, if if you're like Andrew, you have a problem that you're just passionate about, go find your your your community and and work together on that problem and like like you said, this is the uh this is this is a big shift and so uh you know, there's tons of work left to be done. Don't think it's already solved. Uh thank you so much to both Andrew and Q today. Uh thank you Protopia AI and IBM for showcasing a pretty incredible technology uh to to help us uh keep our data just a little bit safer so we can sleep just a little bit better at night and uh and we will we will see you guys in the next dev community live stream. Uh join us next Wednesday, March 18th for 8 hours straight of of of stream uh and uh and thank you again to our to our experts. Uh have yourselves a great day. Bye, everybody. Thanks.