Meet Concierge

Key Takeaways

folks what I want to do is share with you some research that me and my company have been doing and help you understand that there's a lot of options that you have in the AI space so let's let's dive in so hey everyone my name is Mick Douglas I'm a SS instructor and I love it I've been an instructor for about 15 years and you know one of the things that's really cool is getting to do Show and Tell helping people understand what the realm of possible is with technology and it's it's lovely that one of my hobbies and passions became such a fulfilling vocation and so one of the things that I want to do is share with you a project that my company infosec Innovations is making and will be supporting and oddly with the name like infosec Innovations you have to be kind of forward leaning and so concierge is the tool that we're doing to help address some problems with AI one of the things that um really is quite sobering is that there is a ridiculous AI project failure rate in most organizations and it doesn't have to be that way there's also odd limitations with large language models that might not be readily apparent until you start working with them as well as there are some other options for how you can mitigate some of those issues retrieval augmented generation might be depending on your use case might be something that could be helpful and to be candid concierge is a type of rag and we'll get into that in a little bit but one thing and I was looking in the slack chat while everybody was uh presenting there was a lot of concerns about hey how do we safely put potentially sensitive data up into this cloud is that going to be something that is okay and the answer is I don't know it depends on what your risk model is and so what I've done with um my crew is we're making this tool here that um concierge is our response to these primary concerns now I want to be very transparent concierge would not exist if it weren't for the help of the Sans instructor Community I've received a lot of constructive uh feedback some of it very pointed but I love it in terms of the challenges that they say like hey this tool can be better and for those of you who are watching whether real time or on the reporing I'd love to hear critiques on how we can make this be a tool that might be more optimal or giving you options for addressing some of the concerns and more that you may have and I want to be clear coners is not meant to replace tools like the GPT series by open AI it's not meant to replace Gemini by Google it's meant to be another option so let's talk about some of these problems the AI project failure rate is absolutely horrific and is something that we're not talking about nearly enough not just in the information security space but in general it overall and I don't know if you're aware of this or not but there's approximately an 80% failure rate for AI projects and that's not just some madeup you know BS statistic that's actually grounded in hard science this was actually a uh quote that came from a professor from the Harvard Business School last year on the Harvard Business Review idea cast which is a podcast that I highly highly recommend they were talking about the problems with getting AI stood up running and working as expected and so to that end one of the the core tenants of concierge is that we want to make it to be as low drag for you to get up and running and using this tool a crazy key guiding goal around this is to have a just add data approach and so here let's do a live demo of what that may mean or what this can be so you should be seeing the concierge UI here and the way this works is you can go into collection management and then create a collection to house documents or logs or data of any particular type that uh strikes are fancy so in this case I'm going to do one for let's just call it nist 800 171 now for Speed sake I'm going to just upload a single Le PDF but you could if you wanted have multiple documents so what you then do is you go into the loader and you'll select which collection you want the data to be loaded into you click browse files let me go here demo and we're going to go into concierge so the 171 R2 for those of you who aren't familiar this is uh how to protect non-federal standards and it's it's quite a lengthy PDF it's at least 140 pages in length what just happened in that little progress bar concierge read read air quotes around that it read the PDF a page at a time and then split those pages into small blocks this is I'm going to give credit where it's do this is a technique that I picked up from David Holzer at CDI last year in December M and like inspiration at just the right time because we were playing around with doing stuff like this but this vectorization and splitting it the way that he did works out really well because now what you can do is here let's hold on a second Let me refresh um what you can now do once that data is loaded you can go into the prompter pick the collection that you want to ask questions about and then one of the things that's neat and you'll see here in a second concierge will actually give you references and respond to a particular element so let's try this what is multi Factor authentication so you ask that question and it gives you the references from where in the document it came like where that where that answer is being derived and then it generated this is net new text here this is in response resp to the reference material that was given now the thing that's really cool about this is these are all clickable links so if you click them it literally takes you to the material that is the source for your particular question and the thing that's lovely about this now is you have the ability to very crisply and cleanly see how the model how your AI is responding to your questions which is amazing and some something that we've been enjoying a lot um uh for those who are participating live I just had an issue slack crashed on my machine so I'm not going to be answering or addressing any slack stuff until um after my talk I'm very sorry about that but I want to just set expectations so that's a quick tease into what you can do with coners but there's there's some more interesting subtle things that I want to talk about first we've been talking a lot in the industry as a whole about large language models and they're nothing new they've kind of captured the attention of folks and one of the things that I find a bit odd is that many of the projects that are out there make standing up a llm a lot harder than it needs to be and in some cases there's people actively advocating that you create your own llm your own model and you know there's a time and a place for that but I don't think that that's something that most folks should do and I'm not trying to poo poo that there again there is a time and a place for it but creating an llm is actually a little tricky so the idea that you have some training set that you trust and you can ingest and have the AI cons consume it and then have the transformation process being done to create this highly compressed representation of that data I mean that's neat and all that's a lot of work and for most use cases my contention is it's unneeded so what we do with concierge is you actually pick the model that you want at install time and again one of our guiding principles Beyond con behind concierge is that it should be Fallout the chair easy to set up and use and here's a screen capture from the install when you run when you once you download the coner project and run the install as of right now there's seven questions we we'll probably have a few more before we hit the 1.0 release but one of the questions that we ask is which language model do you want to have and we're using the mistol 7B which is a fairly small model it only takes about uh 4.1 gig of space and the memory requirements are quite modest which is nice because this allows concierge to run locally which is something I'll get into in just a little bit so let's stick a pin there and I'll come back now one thing that's neat about L the way they work and why they've caught our attention I really like what Dave earlier said is that we can in human language talk to these tools and through um natural language process they're able to come up with answers that are very good to us and and very appealing the problem is this model itself here let me get my my pen here the model itself might not be trained on data that you care about right like this model might have dated information it might have incorrect information there's plenty of reasons that we might not trust the model and there's other issues too I feel that not enough folks talk about hallucinations and part of the issue when it comes to hallucinations is that we just don't know what's going on with our with our data you know sometimes the AI by the way the reason we call these hallucinations is the AI isn't ever right or wrong we'll talk about in terms of acceptable responses but the AI it has no idea what it's saying what's happening is it receives this going back a slide couple slides this prompt encoding is where things can go wrong frequently because it doesn't understand the input it doesn't understand the output it's doing pattern recognition and pattern response so if any of these bits go wrong you can wind up with h garbage as a response but what's even more Insidious is when you get a response that is subtly wrong it appears to be right but it's not at all so one thing that you need to remember is that every AI out there no matter how sophisticated no matter how well financed they're like this dog doing science it's doing stuff nobody knows how that dog got there nobody knows what the dog's doing including that dog and that's what our AIS are so one of the reasons that AI research has Advanced greatly recently is that we're having a new critique of these tools and we're moving to things like retrieval augmented generation or rag this too Is Not A New Concept this has been around for a while so the idea behind a rag is instead of just having the interaction with the model there's some referential data you know maybe it's from a search engine maybe it's from a trusted data set that's used in formulating the response it's not too complicated but once you have that additional context from whatever tool you'll get the response and then it's uh usually uh things like language models are used to kind of smooth the answer and make it be um approachable and human readable and this works pretty okay but different rag types are out there and some of them have different issues so API or search engine back this can be a problem if the API is off offline you know we've seen outages in the cloud search engines will sometimes have bad data or you know search engine optimization attacks we'll make it so that you get you know non-desirable results like you know Google and a lot of search engines aren't giving us the outputs that we would like also some Rags will use external predefined data sets those can be expensive and difficult to set up and there's frequently some pretty serious copyright issues with these when what my favorite types of Rags are and this is what concierge is it's a self-trained AI it's a self-trained rag on data that you provide and that's what you just saw I gave it that n document and it buil a vector database and it was using that Vector database to answer your problems so these clouds might not be trustworthy you can get bad search responses and the data input absolutely can be challenging these are things that I feel that concierge does handle and solves quite nicely for most users there is still despite our best efforts though there's still one issue that concierge has it still can hallucinate it's a little more hallucination resilient but it's not perfect you always have to uh trust but verify so little recap con is a rag more correctly it's a self-referencing modular offline rag here's a visual representation of it right now coner will consume PDFs and web pages we will very soon in the next month or so be able to support Microsoft Office documents you put those in the collections and then you can ask for information about those Collections and here we'll do another quick demo so let's close that let's go to prompter and let's ask a question Sans 497 is the ENT class so you could ask something like what is opsac and in a moment you get the references and then it explains what opsac is which this is amazing amazing it allows you to very quickly find data that's of interest in large data sets and it's all offline so you can create as many collections as you need this allows you to organize data to your liking um The Collection manager system is something that you can use to create other collections you can see how much content is in them and in later revisions we'll have the ability to move data from collection to collection we don't have that yet but we we will concierge is self-referencing you've already seen that where we have these references on the search results but with what's even cooler is if you go to prompter you can ask concierge to only give you information without an AI response so if I click search I can say show me about opsc and you get the references without any AI opinion and you can click that and then after a moment you will load up oh no demo fail after a moment what should happen is you will get the content loading up that was used for this particular bit of reference so it's awesome so that's search mode coner just constrained by the data you feed it this is part of the reason that coner is very hallucina hallucination resilient because it won't go out and get untrusted data it's only trained on your data you can also use it to do things like checking facts and one of the things that I like is so you go fact check you can do a search like we no longer need to practice opsac and after a moment we get this answer we no longer need to practice obsc is not true based on the context provided for those of you who are in media definitely reach out and come in contact with me because this whole fact check function is to help fight disinfo online I have been using it to great effect quietly I'd love to get the word out more on this function now the thing that really makes concierge special in my opinion is that it works 100% offline here's a block of the code from concierge and you don't need to be a you know a code ninja to uh key in on this key thing all the data we're doing this all locally also because the is local we can do some things that cloud providers cannot do a moment ago we were talking about tokens concierge has no hard token limits now depending on your model that you'll use there may be some token limits like a mistl 7B has a 128,000 theoretical token limit as you appro approximate that it becomes very very slow but what's amazing is compare that to some of the GPT models where your token limit is 4,000 tokens this is a huge huge Improvement one other thing that's neat about coner is that we've flipped how system prompts work on their ear now system prompts uh for those of you who don't know when you make a request of a one of these generative AI tools you type your user prompt but there's usually a wrapper where additional context additional elements of a question are added and for most most users this is way beyond their technical ability and is frankly something that they shouldn't have to do with concierge we have broken system prompts down into a modular unit that we call tasks and you've already seen how this works in the prompter tasks are predefined system prompts that allow you to generate content based off of what particular use case you're after the prompts then we don't have time to get into this but you can make your own prompts and have them entirely locally nobody on the coner team will ever see what your prompts are unless you choose to submit them back to the project so this is a really really cool uh opportunity for you to extend and play and have your own local AI your data is local your prompts are local and even the logs are all local which is great because some my clients even if they're working with uh public data they might be worried about some of the liabilities that may be asked just from the prompts they're giving maybe they're asking spicy questions that if you don't have the proper context around that question it may seem like they're up to no good but you have to ask those questions with coner you can now safely do that knowing that your prompt will never leave your system unless you let it go so let's I want to show you another thing Counterpoint this is is an indicator of how far you can go with uh this sort of capability so uh Counterpoint is a prompt where you make a statement and coner will give you a series of rebuttal to help sharpen your argument so you could say this for some reason I'm still thinking about matters of obsc so um let's just go with a fairly General one um opsc does not need to be done early in an ENT investigation let's let's see what this comes up with so it gave some references and now it gave rebuttal and let's see here we got the retroactive opsc efforts can leave critical information exposed that's pretty good early implement of opset practices ures a stronger security posture throughout that's pretty good the lack of initial opset considerations can lead to unintentionally revealing that's also good so let's go to four proper opsc from the start reduces the attack surface for adversaries minimizing Man four for four so far let's go to five oh my gosh five five five five or five so this allows you to sharpen your arguments review documentation critically slice through pallets of documentation they already have a few clients that are using coners to great effect including those that are in highly highly regulated Industries where they have a lot of documentation a lot of regulatory requirements being able to find what's required of them and what is uh needed is is a godson for these folks so let's talk a little bit about where we go from here because concierge as I demoed it right now now that is the 0.1 version of concierge we have a robust road map and we're actively pursuing it and frankly we would love some help on on this journey so we're here's some of the high Lev things that we're planning on doing we're actively working on the install working on Mac OS there's a poll request that came in from one of my staff from a little bit ago we're doing some final QA tests on it and then we'll be adding better logging and we're we're looking to have a system that will be very very um safe for most organizations to use if they're worried about regulatory or risk issues now here's something that I haven't talked about yet because of the modeling the way that we're doing things coners does not require a GPU now for transparency sake the machine that I was using for for this demo I'm VPN into a a system back home that machine is using an Nvidia 4070 not the highest end not the lowest end but I did that for sake of speed if you're not using a GPU most of the queries that we did in this demo will take about a minute minute and a half the rebuttal one the argumentative one that might have taken maybe 2 minutes so coner will run on commodity Hardware any modern system also a couple things to note about concierge is it's going to be free as in Liberty and free as in beer there um concierge is buy that you saw in the framework that exists is all Apache 2 license and all of the components that we're using in coner are 100% OSS so there's not going to be any odd license surprises or issues that you run into and though we will have monetization when we hit the 1.0 version there'll be some like Consulting things and stuff we're not there yet but when we are there there will be some Support options like for organizations that require support contracts we will absolutely help you with that but we're not going to be pay Walling the the thing we're we will have conator always that will always always be a free tier for it so hopefully this wed your appetite and you want to get started to get started go to data concierge. and here this is the concierge. site here let's put that up here oh dear it's very small what you're going to want to do is click this get it on GitHub that will take you to the GitHub repo where concierge is out and you can start pulling down all the data and get it all set up we have a nice walkr on how to get it started and if you're running into any issues in uh playing around with it definitely let us know we want it to be a very easy and low drag setup for you but this is how you get it going in just a few moments and if you're a Dev here's some help that we would love to have also remember coners we're still Alpha so check the work and one important note we are going to be changing the database system and the database schema so if you start using this tool in a production environment you will need to reload data we will absolutely not do data transforms for moving from the 1.0 to um o or I'm sorry .1 to o.2 so let's have some fun thanks for attending and go get datac concierge. or concierge as we call it and I'll be loading slack back up and answering your questions starting now thank you all