Large Language Models Bootcamp Information Session

Key Takeaways

We'll go ahead and get started here. So, welcome to the information session everyone. My name is Rajbal. I'm one of the lead instructors and um chief data scientist at data science dojo. I'm going to be talking about um the large language models boot camp that we have and also the agentic AI boot camp uh that we recently launched and I will explain what is the difference between the two. Um I've been doing uh machine learning, data science, AI, analytics, give it any name that you like. I've been doing this for quite some time. um and um you know have been building models, building machine learning models, building uh uh agentic AI uh lately very recently we have products that we have built I mean we have a training side of it we are not purely a training company we also uh provide services and we have a product uh being used by uh big enterprises uh globally um and that's uh and that is a practical experience when we teach uh that practical experience actually comes in in our uh in our curriculum in the in how we teach um any topic. um one of the oldest companies uh on the planet when it comes to machine learning, AI, analytics, anything under the sun in that uh space. Uh more than 11,000 people who have graduated from our programs and I'm talking about not self-paced programs even though we have do have some self-paced learning but primarily um these 11,000 graduates. These are our face-to-face synchronous learning graduates. uh lots of companies and pretty much any country in the planet has been represented in one of our trainings. So why did we decide to start this boot camp? We are possibly we are the first boot camp in industry even now there's no 40-hour boot camp. I mean there's like 2hour boot camps out there 3hour 4hour boot camps but really boot camp is should be a boot camp like uh military boot camp. Uh so why did we start this boot camp and we are the first one in industry and we happen to be you know the highest quality boot camp out there but I mean why did we start this boot camp? It turns out um that uh many companies uh enterprises when they start building uh LLM applications, they realize that uh building an agent or building uh maybe some kind of custom GPT model in uh in chat GPT. Well, it seems quite straightforward, but when you start to build an application and try to deploy it and operationalize for a customer in healthcare, in uh cloud computing, in finance, in uh retail, there is a lot of challenges that you would run into. The first and for foremost challenge challenge is that uh prompts are going to be brittle. the same model. Even the shiniest models when you uh deploy them uh um a slightly different variation of a prompt can actually result in substantially different results. Um then there are uh you know your your application may be uh an application that requires low latency versus higher latency. Then uh hallucination if you are pro here in this information session uh likely you are familiar with what hallucination means uh you know LLMs can have a mind of their own they might actually give you a response that may not be uh factually correct or even if it is factually correct it may not actually be relevant to what you are uh what you are um the information that they were given at hand. Um then uh the cost aspect of it right. So um it may look like chat GPT make may make us uh make it look like that uh these systems are uh inexpensive. um you know Gemini is there you know anthropic is there there's a lot of applications that are out there but in reality when you start to uh adopt these applications in your enterprise well costs are uh costs go out of hand very very quickly um after all of this uh you know let's say cost is cost is okay I mean you have the budgets and you have everything under control the application is working very well. But what about the data governance? Um um what if you index the wrong information or what what if you uh what if you index the information without uh without proper data governance and access and authorization controls in place. Um there is a lot of challenges when you start building an application like this. You know the scaling becomes scaling aspect becomes a challenge. you know regulatory u challenges are there. Guardrails become a problem. Uh you know anyone can make these uh these applications say whatever they want them to say. Um you know initially uh there are some guardrails that are there in place but you have to have some organizational guardrails in place. Um so many different things are possible. Um we realized that going uh starting with a P is very easy. Many companies that came to us they they had call them you know I call it the graveyard of PC's. So many companies that started building these applications they realized um that uh well it's not as easy as as it may sound. You start with a very uh very cool application, but when you try to operationalize it, when you try to productionize it, start to give it to your your customers, when you start to give it to your uh you know your employees, uh there are challenges, there are a lot of barriers to adoption and that led us to really well launched this boot camp. So we be we became a product company. We have been a training company. we started building Agentic AI uh products and then we said yeah I mean why don't we add one more training to our portfolio because I mean we have enough uh experience building these systems that we can actually teach other people how to build these systems from scratch. So the the overall the topics that we cover are um um in in in depth. We go in all the topics in depth. I will show you in our learning platform what kind of content we have. But we start with embeddings and transformers and attention mechanism. We go dive deep practical exercises vector databases. You will uh understand uh all the ideas behind vector vector databases. I will actually show some of these uh in our learning platform. So you you can get a better sense uh langchain and agentic workflows. How do you um how do you connect your LLM application to a Dropbox or SharePoint or a Salesforce uh um uh or you know really any of the data sources? How do you uh chunk your documents? How do you use u Gemini model? How do you use OpenAI models or versus GPD4 GPD um 40 or 4.1 or how do you use Llama model? How do you uh connect your application to a search engine? Um how do you uh build an agent? How do you how how do you uh build that reasoning uh in within your uh LLM application? So we talk about all of that. We talk about observability and monitoring in detail. Um guardrails, how do you implement guardrails? um you know ethical guardrails, topical guardrails, uh you know any kind of uh legal issues u um harassment, bias, how do you actually prevent your models to actually act in a manner uh that can get you in trouble? Uh um I mean lately we have started seeing uh things that are h in news uh like Air Canada getting sued by a passenger because the chatbot uh told the told the passenger uh certain things that were not factually correct and the court ruled in favor of a customer. And recently actually last week uh I was looking at this news article that now insurance companies they have started offering insurance against uh uh any chatbot uh lawsuit right so you have a chatbot on your website and the if you get sued then the then the insurance uh your insurance uh policy will kick in. We uh talk in detail about evaluation just like for those of you coming with a from a from a traditional machine learning background. Uh you know how do you evaluate your model? How do you evaluate uh the correctness of the model? How do you uh evaluate the uh correctness of the answer uh relevance of the answer to your question? How do you um how do you make sure that it does not have anything uh inappropriate? Uh how do you set up that benchmark um uh deployment or ops? We talk in detail fine-tuning uh you know you take a model uh a model has um model has been trained to work well on a um on the data set that they were trained on. But what if you have a very specific data set uh that was not used for training the model? How do you fine-tune the model on your own enterprise data set? We have practical uh we have the theory and practice of all of them. And then how do you build um what are the challenges that you run into while building up these LLM applications and what are the product management challenges that you will run into. Um so we have two distinct products. Um uh the LLM boot camp is a it is a 5-day 40hour intensive program. Uh it is uh it is offered both in class and instructorled online. It is as it can be you can attend it synchronously from wherever you are or you can attend uh this uh boot camp in person in Seattle. So um it is uh primarily um geared towards absolute absolute beginners. So if you're an absolute beginner, you don't know what transformers are. Uh you're you don't know what attention mechanism encoder decoder uh is and you you you want to start from scratch, right? So, so this is uh this is the boot camp you want to attend. Um, and I will show you once again, I will go to the curriculum. I will show you a few things uh in in a bit. uh actual exercises, how does the learning platform and the course material look like? The agentic AI boot camp uh this boot camp is more of a uh call it a slightly um advanced version of the LLM boot camp. And so we do cover some topics that are covered in the LLM boot camp, but some of the fundamentals they are not covered in class. Um we expect uh the attendees to actually know some of the foundations. I mean if they they should already understand how an LLM works or what is an embedding and what is a transformer etc. But for anyone who does not have the right background if they want to come in and then we can give them self-paced we give them self-paced material and they can actually um go through that remedial material before they actually come to the training. Um but we take a deeper dive into the into building AI agents. We do talk about AI agents in the LLM boot camp but we take a more coding intensive deeper dive in building those agentic uh architectures. Um we have um different kind of agentic architectures that we will uh dive into uh you know how do you do state management in a uh in an agentic framework. um you know there are um there are different kind of design patterns that we will be covering in uh in uh this boot camp and I will explain that uh in a bit. Um so when you look at an application uh I will go through this very quickly uh because I have to go and uh talk about the the course material. I have to show you the actual course material. Um so um at the core of an LLM application is your um your LLM whether you are accessing it through directly on prem it could be on your laptop it could be on your desktop it could be in the somewhere in a VM in your cloud it can also be um an API that you're using and now um you know this is how it is deployed but it could be a co open source or closed source model uh example of Open sour uh open source is Llama and um I know Llama 2 model for instance right Mr. call lama 2 um or llama 3 4 those models llama series models and the examples of closed source would be uh open AI anthropic right so we do touch upon mainly we focus on llama and open AI because we have to focus one open source and one closed source uh but fundamentals remain the same once you know it and on one platform you know it on other platforms um most of the LLM applications in enterprise very likely Um I would say in majority of the cases you're going to end up building a rag application which is a retrieval augmented generation application. So when you build a retrieval augmented generation application, a vector database is going to be at the core of it or call it uh the context the context is going to be at the core of it and whether you're receiving that context from using uh from a Salesforce from a dynamic CRM or from a web search or from a SQL server or your MySQL right so or a Postgress right so uh the context is think of that as your data that augments your LLM. LLMs can actually act pretty dumb. I mean alone if you talk to an LLM, LLM can be actually quite dumb. Uh an example would be is if you asked uh a GPT40 model, GPT 4.1, you know, state-of-the-art model, right? So if you go and ask them who's the president of United States at the moment they uh you you might get a response like hey uh my knowledge is limited to um October of 2023 or October of 2024 and as of that date um Joe Biden was the president of United States some something along those lines. uh so and there's this common confusion right so is CH is no no I mean when I go to chat GBT chat GPT knows everything chat GBT is not your LLM chart GPT is an application built on LLM uh on a foundation LLM so what we will teach you is how do you be build a robust application like chat GPT on top of an um an open source or a closed source model so how do you add that context so uh to add that context a vector database is going to be at the heart of it right so you need a vector database at the heart of it right so and we talk about all the fundamentals and I keep saying this I will show you some exercises that will actually clarify I mean how do you actually uh um how how are we going to teach this um then a vector database can store uh you know it can build a keyword index we'll talk about those keyword indexes how do you build keyword indexes and you know your classic BM25 TF IDF type indexes and then also your semantic search index uh where you need an embedding model. We'll talk all about embeddings uh in detail um different how do you create embeddings what does embeddings even mean? Um then you need frameworks like langchain or llama index. We'll talk about uh lang chain in a lot of detail like in uh in in great depth actually we spend about 8 to 10 hours just on lang chain uh out of 40 hours in this LLM boot camp. So um then there is observability uh we actually have this whole discussion or observability we have discussion around guardrails uh deployment all of that so we cover pretty much the entire ecosystem. So uh uh after 40 hours when uh our attendees our learners when they leave they are actually able to build an end toend LLM application. Um I will show you what uh this entire course might look like for you. So when we uh when um when we sign up uh when our learners sign up uh they are given access to this learning platform. Once again, this is not a self-paced course, but to make our interactive in uh you that uh instructorled course more efficient, we have resources that are already set up, right? So, these are broadly speaking the modules. We that we have separated them out as courses. So, we start with a highle introduction, right? What does it mean to build an LL LLM application? Well, why does an LLM, you know, how how does an how is an LLM built just the core foundation model? How is it uh how's it built? Uh what is prompt engineering? What what are the risks and you know and um challenges when you're building an LLM application in enterprise? We talk about all of that. uh we uh we talk about the end to-end architecture just to give we set that breadth of the entire ecosystem in about 3 hours on the first day. The first three hours are setting setting you up with a broad uh understanding of the breadth of topics. Then we get start taking a deep dive into each of the components that we have understood in bread. So we start with this transformers and attention mechanism. we go deep into uh this topic. Uh we have Luis Luis Rano. He is the he is the instructor for this uh this particular module. Uh we talk about you know start with an introduction to very quickly AI deep learning um neural networks. I mean how how does this feed forward and all of those layers how do they work? We talk about uh um the attention mechanism, encoder decoder architecture and so on. Uh we talk about all of that. Then we get go into uh do a quick exercise in prompt engineering um at a very high level uh because prompt engineering is something that we will continue to uh go back to as we actually uh keep learning. Um so I will uh double click on this one of the modules. Let's say a practical introduction to vector databases. So when we go inside um so a a typical course may look like this. Uh um you know we have all the setup guides. How do you set up a vector database server? Um how do you set up your remote desktop and you know we have uh all the guidelines all the everything is set up here. Then we go uh and uh let me show you what we have here. So for instance where we start the way we start is um so we spent about four to five five to six hour five five and a half hours on uh on vector databases. Um so we start with the foundations we spend about two and a half hours half of the time uh we spend this time on explaining the theory part of it. Well, why do you need a vector database? And what is a vector database? How is it different from a SQL database for instance or a NoSQL database for instance? Um, and then we start building our momentum and uh if you look at this, this is how an embedding is. An embedding is in a vector space. How do you do a vector search? So, we keep talking about these topics. You know, what does a vector search uh look like? What kind of searches can you perform in a vector database? You can do a vector search. You can have a pure vector search, you can have a pure keyword search or you can have a hybrid search that actually does a vector search and also does a keyword search and does some sort of applies some sort of reranking on the final results and then then gives it uh gives a more I would say more robust uh results as opposed to only purely focusing on uh either keyword or uh semantic search. uh then there is uh you know your your where and having and other clauses that you have in your SQL database. Then you have this faceting and filtering also possible. We talk about all of those topics. We also talk about um how what what is it what what data structures and what uh techniques are used uh that make these uh vector databases capable of returning um returning um in a maybe 10 to 20 30 40 millisecond uh millisecond um out of you know you have hundreds of millions of embeddings and now you're searching for something and then within a few milliseconds you find the right uh piece of text or right uh document. How do they do it? So we talk about all of it. Um and of course I cannot go through the entire thing but you can see that we are explaining step by step piece by piece we are explaining everything here and once you understand all of this we go to practical exercises and uh we will go to for instance um we will explain we have a vector search practical exercise um then we uh we uh we have a similarity search exercise we have a hybrid search exercise and so on. And what we have done here is uh we we have preloaded these exercises in the uh in our um sandboxes and the way this works is if you look at this I've clicked on this here and now when I click on this that uh uh code is autopop populated in your web browser and if you notice in the URL this is my dedicated space. So just like I mean all learners they will have their own dedicated space. Uh so if you make any changes to the code they are persistent. You can actually copy and paste make changes to the code maybe do something else uh you know all of that. So this is included in uh when you sign up you have access to these sandboxes for one year after the boot camp. Uh so this is an example of hybrid search. we uh go through this um we show you how to create u the V8 URL. These keys won't work. Uh these have been disabled. So just in case anyone is uh drooling here. So uh you know so we give you all these keys uh um and we enable you we give you your own vector database. You you go and create um you know a vector database in the cloud. We uh we have a partnership with VVA. will give you all the resources that you need. Um then we'll give you open API uh open AI API keys. Um so and then you will set these up. Um explain run run uh you know okay what is this? Well this is a table equivalent of uh in a SQL world we call it a collection here. So what am I doing here? I'm setting this up with a text embedding three large. Uh how do I use a small embeddings model? How do you use an ADA embedding model? or how do you use a llama embedding model? Well, um we talk about all of that. Um well, in this case, uh I'm vectorizing uh some properties and not vectorizing other properties. What does that mean? I'm importing my data. And then if you look at this, I'm doing a hybrid search. Um you know, uh if alpha is zero, then it makes it purely a keyword search. If alpha is one, it makes it a pure vector search. If we set it to 0.5 then it's partly uh partly uh vector and partly um partly um semantic search. I see Jad you have a question. I don't think we can enable audio but I mean feel free to type your question in chat and it will be uh in chat or in Q&A and it will be routed to me or I I can see it actually. So Jad please feel free to type uh your question and I'm going to actually look at it. Okay. And then while the question comes up I am going to continue. So now um um there is this um you know in this case I'm so we changed different variations. For instance uh you know this is let's say I'm searching for cobra but cobra was never mentioned in any of the documents. This keyword did not exist. Well a rattlesnake was there. So semantically um if I do a pure semantic search well cobra would be a match to the rattlesnake document. But if I do a pure keyword search well this word was never mentioned and you will see an empty uh result that came back came back. If you do a hybrid search it's partly uh partly uh vector partly or partly semantic or part and partly um keyword. So uh we get the boh best of best of both worlds. Um yeah so uh we go through this just like I'm explaining. Of course this it's not going to be as rushed of an introduction as you see right now but uh you know how do you filter how do you facet your data uh and so on. Okay, so Crispen your question is how does gra uh graph rag fit into the mix mix. So uh we actually do have a session on graph rag as well and uh so the question is how does graph rag come into picture. So and I'm assuming uh Chrisen you are familiar with knowledge graphs and that's why you're asking this question. So, graph rag actually um um LLMs are uh actually um I mean as I said earlier I mean many of us already know about what what does hallucination mean in the context of LLMs. Um uh if you have a knowledge graph, right? So if you have knowledge graph of all the relationships between entities, chances are um uh chances are that uh uh your model uh even if it is misled by some text actually uh being in close proximity of each other. I'm trying to come up with a good example, you know. Um, you know, it could be Space Needle. Well, uh, an LLM is not going to mess up on a Space Needle, but a Space Needle is in Seattle, right? So, maybe I have a, uh, um, I have a, uh, for some reason my model actually got confused because of a lot of text being there. Uh, it cons confused Space Needle that is a building, a monument in Seattle. uh it confused it with uh well a needle in space. I'm totally made up example. I know it's not nonsensical but uh you know well that's how it is or a better example actually sleepless in Seattle right so sleepless in Seattle right so well sleepless in Seattle is a movie and sleepless in Seattle um you know I have a knowledge graph it is sleepless in Seattle is a movie and I don't know who the director was but Tom Hanks was there but when uh somehow uh you know my model hallucinated uh because sleepless in Seattle somehow it is confusing it with sleeplessness. Now what knowledge graphs do is they actually step in and guide your LLM generation in the right direction. So this entity relationships they make sure that if a model is uh getting confused uh they ensure that uh uh knowledge graph actually uh or graph rag it actually ensures that your uh your generation actually stays on track and we talk about these topics actually. So in the boot camp we have a session uh by Adam Adam Adam Cowi from Neoforj he actually talks about graph rag so we do cover that too I hope I did answer your question um if not please let me know uh okay so if you look at this um once we have finished our hybrid search we'll go to generative search and generative search call it poor person's uh rag. I mean this is the very very basic rag where we show or retrieval augmented generation where we show that we have a bunch of uh bunch of um you know documents in our vector database and then we are generating um you know this example is based on some documents in vector database we are generating Facebook ads. So we first retrieve and then generate right. Um so uh and that's it right. So if you look at this uh just a very basic example we are building this uh building this momentum slowly uh and gradually and then um the and building things as we uh you know as we understand one topic. Now what do we do? Now what we do for instance right after all of this we go vector compression. Why would you need vector compression? Um, in vector compression, uh, think about this. Uh, let's say I mean I'm I'm Door Dash or I am Uber Eats or I am um, you know, a company that is um, somewhat you know at working in scale. I may potentially have hundreds of millions and possibly maybe billions of embeddings. And once you have these embeddings that u the billions of embeddings so compute and storage and compute and u and uh retrieval it becomes uh inaccurate and also time consuming. Okay. Let me see there's a Q&A. Okay. Okay. I I will answer there's a question about do you get into MCP I will address that question in just a moment. Um so we talk about vector compression um you know well why do you need vector compression and how do you do vector compression right so this uh using PQ product quantization so all of this once again you can see I mean it's not some some you know yet we'll do it we have it right so you can see that these are some uh vector compression exercises uh and you can see we are performing a vector search and now enable PQ by updating the collection config right so basically what I'm saying is uh we'll talk about it how do you do vector compression why do you need vector compression and so on so we'll do it and semantic caching uh sometimes you um uh in if if especially when it's a support-like scenario when the question has always the same answer there is no personalization the question is not situational it doesn't change based on context u for instance uh um what is the capital of France or where is Space Needle located? So these kind of questions um or maybe how do I install um SharePoint on a Windows uh or on a Windows server. It's some questions that have a very clear concrete answers we don't have to generate again. We can go uh get by with semantic caching. So those kind of things we have uh we have this idea of semantic caching. We save on token cost. We save on GPU costs and these are some architectural optimizations that we go for. Um uh so before I continue and talk about more of these modules, I'm going to actually address a question. Do we get into MCP? Uh yes, we will talk about MCP. But when you say get into MCP to what level of detail? We get uh into MCP in a greater detail in our agentic AI boot camp, but we definitely give enough uh context in our LLM boot camp. So once again, as I said, MC LLM boot camp is more for it's a bit more beginner friendly. Uh even if you're not a coder, you will do fine. But um the more of practical exercises and more of the detail and some new some of the nuanced aspects of it uh um they are covered in the agentic AI boot camp. I want to be very clear people have attended our large language models boot camp. they came in as beginners when they went back they were able to actually build things right but uh some of the more nuanced some of the more intricate design patterns and uh more advanced topics uh you know you are now you learn you want to learn advanced chops in uh aentic AI or aentici applications for that we have aentic AI boot camp but otherwise if you're a beginner LLM boot camp will actually get you there So you will be fine. Um, lang chain lang chain is uh think of this as one of the most important tools that you will need in building an end toend application. Well, does it have to be lang chain? No. I mean you can use llama index. The llama index is another one. Okay, Lalita, you cannot see the screen share for some reason. Can others see my screen? Uh can someone just one person can confirm if my screen is visible? Okay, thank you so much Cassie. Um so um so um Lalita can if you can uh double check I mean maybe it's an issue on your end. So lang chain actually helps you build uh end toend LLM applications. So um you know we start with really why do you need lang chain? How do you hook up different kind of language models? How do you um connect to different kind of chat models? How do you set up prompt templates? And of course I cannot show you every single um exercise here. I will click on things here and there randomly. How do you set up prompt templates? So we talk about why do you need prompt templates? Well, you want your prompts to be dynamic, right? So you cannot have your prompts that are very very fixed. So we go and talk about uh you know um you know prompt templates and how do you set up a prompt template. Uh well uh if you look at this I imported this lang chain library called prompt template. Uh started with this we'll give you the open AI API key. You will enter the open API open AI API key. And then if you look at this location is uh we are slowly you know we'll slowly but surely we are we'll get there. So start with the ba very basic prompt template. uh so location and then the whenever the user whatever the user enters on screen um it just goes and plugs it in right so that's uh what langchain does and then uh I have this you know this dynamic behavior of my llm application uh then we go to example selectors how do you give it few short examples uh spyros the code where can we find uh uh this is on our learning platform and a lot of code is actually also available in open source. So a lot of I mean uh code you can find anywhere frankly I mean having these toy examples they are out there uh you know um go check out lang chain uh you know examples go check out llama index examples. So I think the code you can find anywhere. So the question was where can we find the code but this particular code of course it is behind uh you know our learning platform it is behind a login um and this of course we cannot make it publicly available uh because there's a whenever someone runs it uh there's a cost on our end uh we pay for the compute behind the scenes um retrieval how do you connect it to different kind of retrievalss uh so how do you connect your LLM application to Google search or Google maps or or Yelp or uh Salesforce or um you know, think any data source really. Uh um how do you split your text? I mean, how do you chunk your data? And then uh how do you chunk a document? You have a 100page PDF. How do you break it down? Okay. And um uh how do you break um break down? How do you split a markdown? How do you split in HTML? So, we actually do a few of these exercises. Let me let me see. I will just click on any of these for instance, right? So, let me click on this. This is a text splitter example. So, we talk about this and in this case I will very high level I will say this we are showing a recursive text splitter. Is that the only kind of text splitter? No, that's not the only kind of text. you you can see that there are other types of HTML, text splitter and latex. Uh we have this autosuggest installed. So we show you one example and then after that you can be on your own. Uh you change play around with this and then you know bring in any data do whatever you want to do and this is on your I mean you can just go and uh keep using it uh even after the boot camp. Um if you look at this uh so we take this toy example there's a text file that we are chunking. You can see that uh um uh each chunk is thousand tokens. There's a chunk overlap. We discuss in detail. People ask hey why not a 100? Why not a 50? We have that reasoning and discussion. So we we make sure that um I mean for those of you who will end up at the boot camp you will uh see me uh or hear me saying this a lot. We are not going to give you uh fish. We are going to teach you uh how to fish. So we will set up the foundation, we'll teach you, enable you and after that you can be on your own. Uh because we cannot teach everything that is out there, right? So we we are not actually one of those companies that will tell you hey you will become an outstanding you know top-notch AI engineer right after the boot camp because a lot of this is your own undertaking. We are we definitely have the most comprehensive curriculum on the planet. I mean I'm very confident about it right I'm an educator I'm an engineer I can tell you we have the uh I mean I don't think anyone even comes close to this comprehensiveness of the comprehensiveness of the curriculum that we have but at the same time we are also being an educator being a practitioner we know that a lot more is needed that you will actually do on your uh own uh you will have to take your own personal responsibility as well um we talk about chains, we talk about memory, how do you add context to your um context to your application? Then we talk about agents and um and this is I'm still in LLM boot camp. Uh so uh we talk about agents. Um something as simple as how do you actually configure a web search? Um how do you connect Google search or Bing search or duck go search to your uh your LLM application? So uh if you can see we are slowly gradually making our applications more and more uh complicated right so in this case if I'm so if you look at this uh and we explain all the code you don't have to worry about uh you know not being um uh not being well versed in Python all you have to do is run run run run run run run run run run run run run run run run run runrun run run run the code is tested completely all dependencies all libraries everything is installed uh But even if you're not planning to go back and code, you're a product manager, you're a technical product manager, maybe you're a people's manager, uh you eventually you think that you will be hiring a vendor. Still, it is worth going through because uh you know there are many things that you will know as a result of the boot camp that you would otherwise not know. So this is a very very practical experience and those of you who are uh hardcore coders uh well go make make modifications bring in other li other uh leverage other libraries do uh make changes to the code and and see uh what can be done and you have everything installed here uh you're more than welcome to so in this case if you look at this planning and executor and etc. So you do it now uh multi-agent collaboration and then rag agent with langraph and then we talk about langraph lang graph think of that as uh you know one of the frameworks that is out there for building multi-agent applications I mean there is stuff uh going on in Azure foundry Azure foundry offers it AWS Amazon Bedrock they also offer it uh Google has their own uh thing going crewai llama index I think all of them many frameworks are out there. Okay. Um so there is let me see let me handle this question. I think the next question is uh um the next question is uh u when will be the next uh cohort after June? The next cohort after June is going to be in August. uh the dates are not confirmed but somewhere uh in August we will have it. Um so this is an example of land graph in land graph. uh think of this as multiple LLMs actually think of this as a as a as a graph where you have and one LLM here one LLM here and they are collaborating on solving a problem right so and then you're doing the state management there is nodes and edges you're doing state management you're trying to make sure that uh hey I gave it to this one and I'm waiting and and that LLM comes back uh and says uh you know I mean let me let me give a concrete example right so there is a uh there's a pattern called reflection reflection reflection pattern um where um um I asked my LLM application my agent to write um a proposal and u the generator actually generates and there's a reflection agent that actually takes the output from the generated uh or the generator it takes that output from the generator and sends back the treat to the generator and the generator actually fixes uh the fixes based on the feedback. It's almost like a peer review type uh framework. So we talk about all of those things. We go through this in detail. How do you do the state management and all of that? We explain all of this code, what is going on and so on. Uh I will I I think we are out of time here. So I'm going to actually uh rush it through uh now now that by day 2 and a half or maybe about 20 24 hours into the boot camp now we understand what is a uh what is a uh what is a retrieval augmented generation application. We get uh deeper into what kind of challenges you should expect when you're building a rag application in an enterprise. Then we will be talking about um you know LLM observability and guardrails. How do you set up observability? How do you how do you find out how many tokens were used? Which uh LLM was called? Uh how long uh did one call take and so on and how do you put uh enable guardrails? We talk about fine-tuning. Uh in fine-tuning for instance, right? Maybe I can double click here now. So when you um in the fine-tuning module um I mean I I cannot double click on all of them at a very high level. I will show you uh here. Um so we first go through uh the fine-tuning. We set the foundation. Well, what is fine-tuning? What is transfer learning? Uh what is um you know what is uh distillation? What is knowledge distillation? We talk about all of that. Um I hope all of you can see it. We explain that uh you know more of the nuances. How are they different? Uh um then we talk about some challenges in fine-tuning. We talk about optimization right. So we explain u you know this concept of low rank adaptation and um and uh quantization and low rank adaptation. uh we talk about this if you look at this and we give really I mean uh what does low rank actually mean so we we make sure that uh you know learning is uh learning has to be enjoyable right so you know and sometimes when we talking about math ideas I mean they are they should be intuitive in intuitive but we I mean somehow along the way we make them fairly complicated right so we make sure that even those who don't like math or have not had good experience with math, they are actually able to uh they are able to understand the topics that we're talking about. Um so we talk about quantization. What does it mean? Well, it means that we are representing the coefficients of our uh we are representing the coefficients uh uh we are um um the coefficients in a in a low precision representation. So we talk about explain all of this really so you exactly understand what does low rank adaptation in fine-tuning or uh or quantization for that matter Q Laura which is quantized low rank adaptation what does that mean you can we talk about all of this explain in quite a bit of detail and we talk about challenges and once we do it we have this exercise where we actually give you a GPU cluster we give you credit for this GPU cluster. You go and create your own GPU cluster. We give you a piece of code um where you go and uh uh upload the code um and we learn how to fine-tune a llama to 7 billion parameter and 4-bit quantized model. uh we take the unquantized model without quantization the original model uh or I'm sorry uh we take the uh model without fine-tuning uh we start with the uh non-finetuned model uh or the original base model and we also take then the the fine-tuned model and we compare how their performances differing. So all hands-on and you have these very interesting revelations that what uh you know what does fine-tuning entail and when you fine-tune what are certain things that you need to be worried about once again no coding background needed it's again in just like I was showing run run you keep going going going until you basically you know uh you understand what does fine-tuning entail uh we spend a lot of time on evaluation as well. Maybe uh this is probably the last module I will double click on. Um rest assured that all of these modules they have similar content. Uh it is just in the interest of time I cannot actually give you every all the possible details that are out there. So in evaluation we set the context of evaluation. Why do you need evaluation? uh and what are some of the things that we need to worry about? Why do you need uh you know what kind of challenges you will run into uh you know why uh so for evaluation you need evaluation data sets and evaluation metrics and for different types of task I mean you can have a language understanding task you can have a text generation task you can have a Q&A task so depending upon which task you're using which metric you are going to use so we go very deep into you know uh what kind of data sets are available for eval valuation we go through them um and you know really a very in-depth understanding of how do you evaluate an LLM and then once that is done um the same thing uh every module has this area where we actually talk about all of these I will show you one lab here okay so when I click on this you can see now I have um the same thing you get the API key from us and we are going to load and split the document update metadata you know create embeddings and set up a vector store create a prompt template set up a rack chain and extracting questions and ground truth and then after that uh evaluating the model performance right so if you look at this you know um we in this case we are doing faithfulness relevancy precision recall of your metrics And pretty much it. That's that's pretty much it. Um and then uh we uh on the on the last day of the uh of the boot camp uh the LLM 5day boot camp, we have about we allocate about four hours, we give everyone some boilerplate code in a in a VM. We give you a VM. Uh not everyone is a web developer. So we give you a pre-built application and we give you exercises. Let me show you. So and we uh so we help you set up um uh an application and once you have everyone gets their own application and after that if you look at this uh once uh once the basic stuff uh is done you deploy it you push it to GitHub from GitHub you push it to the cloud you have your own uh application with your own URL running and then we say okay why don't you connect a Wikipedia agent uh and what else is there uh why don't you um do a Python ripple uh langchain agent right why don't you build an archive langchain agent and then you you see that we have these exercises how do you add a memory uh conversation memory in a basic chatbot so these kind of things um we do it and once again when you leave uh you understand um uh really the endto-end uh life cycle of an LLM application you understand it in in its entirety and u and yeah I mean you can you're a better if you're a good engineer you get you're a good engineer who can now build LLM products if you're a good product manager uh you know you actually now uh you're still the same person uh whoever you are uh but you are a uh you're now uh you're a technical product manager in building large language model application you're a you're a good PM program manager project manager in building LLM applications. Um, and uh, if you're a dev, then yeah, I mean, you can you can go and get started with building LLM applications. Um, and I think that is it as far as uh, my presentation goes. I will very quickly I think I have taken more time than I should have taken today. Um, this is the technology stack. We cover a lot of things and like different uh, pieces in the ecosystem. We actually spent quite a bit of time. Uh I have gone through the curriculum. I I'm not going to actually spend more time here and I'm going to Yeah. And just bring your laptop by the way. Right. So I mean so we have sometimes hey do I need to install anything? Do I need to have subscriptions? No. I mean we have set it up in a manner that you hit the ground running. I mean you as long as you have a computer that has a web browser that is it because we don't want to run into it issues. my Mac and Python and Windows and Linux type issues. Everything else we give it to you uh in browser enabled labs. Um you will be getting a a verified certificate um from the University of New Mexico. And as a result of this uh many companies actually uh since we are partnering with an accredited university uh many companies they actually cover um the cost of our boot camp as part of their tuition benefits. um last boot camp or last one or two boot camps we have had people from Boeing uh and Apple and uh and Salesforce uh and uh you know of course many companies they pay for their employees and I'm talking about uh um many companies have uh this uh part as part of the benefits program just like your health insurance your dental medical vision uh there's tuition component of it check with your HR uh and we may be actually you may eligible for attending it for free because well we cover uh we are we are offering it in partnership with an accredited university. uh we have a great line of lineup of speakers. Not everyone is there every time. Uh but uh you know these people have been there um and um you know depending upon availability I mean we have people who come in. I can confirm that next time we'll have uh I'm there. I'm I teach some of the boot camp. Luis is going to be there. Jerry has been there a few a few times but I'm not sure next time. Adam is going to be there. Next time John is going to be there. Sebastian is going to be there. So you know it Sam and Sage they will be there. So it it depends I mean who's available and based on that availability uh because these people are I mean they have their own things going and u you know it the sessions have to fit in there uh in their schedule. Uh what else? Uh yeah these are some testimonials from our partners. I mean we have had people from any company that matters on the planet. I mean with our past boot camps and now I mean we have a lot of uh lot of presence globally. Um so our next large language models boot camp is happening from June 9th through 13th 5 days straight 9:00 a.m. to 5:00 p.m. if you want to have that you know that waterfall right. So you know if uh you want to really just be done with it in one week this is your um this is your training the agentic boot camp is because many people asked hey I cannot come to Seattle and I cannot take 5 days off uh it is not possible for me so that's where we introduced this uh you know more of an 8week version 3 hours a week live class and then with some homework uh and then focus is more on uh agenda Authentic AI. So if you want a more spread out version of the large language models boot camp with some more intense and deeper uh discussion on agents than agentic AI boot camp. If you're looking for more uh you know I want to be done with it once so I can just go and build things on my own then large language models boot camp is the one for you. Okay. So Chad I know you have a question. Uh is the foundations course available online? Sorry, not sure if it was this was covered. I'm not sure Jad when you said foundations course. Uh so in the LLM boot camp we cover everything including the foundations. Um but uh in the Aentic AI boot camp if someone is uh is does not have the background, we do make the foundations uh some of the foundational uh work available that is needed to be able to successfully complete the H&TKI boot camp. uh but if you're not sure feel free to set up a time with us and then we will actually guide you which boot camp is the right boot camp for you. Okay. Uh with that I think if there are any other questions I am happy to take those. If not I will see hopefully I will see some of you at the boot camp. Um sounds good then. Thanks everyone for joining today and I'm looking forward to seeing some of