Large Language Models Bootcamp - Information Session

Key Takeaways

Okay. So, we will go ahead and get started. Um, thanks everyone for joining. Uh, my name is Rajakbal. I'm the chief data scientist at data science dojo. and we are going to go over the curriculum for um the light language models boot camp and also answer any questions that any of you has uh about the curriculum and um yeah perhaps it may be a good idea uh if you can utilize the the chat whether you are on Zoom on Zoom call or LinkedIn or Twitter or any of the social media uh any of the channels uh uh if you you know just go ahead where you're you're joining from and um you know anything around I mean what kind of use cases you're working on in large language models or if this is your post introduction I will go ahead and get started uh uh so uh my own background I've been doing this uh this whole thing broadly speaking called AI um call it data science call it analytics machine learning um predictive modeling for a long long time now. Uh back in grad school um and then followed by uh my work at Microsoft uh and then followed by my work at data science dojo. Uh we are one of the oldest players in this space in the upscaling space. We have been around uh any any significant company that matters on the planet uh globally uh someone from uh you know more than 3,000 perhaps close to 4,000 companies uh people have attended uh our training globally um 11,000 plus graduates and I'm not talking about uh you know online self-paced learning type graduates I'm talking about this kind of interaction that we are having either in person on site in classroom or remotely uh through a zoom call. Um so uh let's get uh uh dive right in. Right. So um uh one of the fun things is uh um you know a bit of context about the boot camp. Um we are um uh we are a platform uh also I mean we have a training side of uh the uh our business but the we also have a product side of our business. So over the last two years we have built an large language models uh platform as well. And uh while building that platform we realized uh that building a platform for um for large language models uh or LLM applications uh or deploying agents as we call it you know the agentic so-called agentic uh behavior or agentic frameworks deploying that is uh it is non-trivial and uh what I find actually fascinating is that uh you know uh when you're interviewing people and you talk to someone and then someone would say um yeah I mean uh so why did it take you like 30 30 plus engineers and uh 2 years to build this? I built this in in 1 hour, right? So, and uh the problem is building um u an LLM chatbot on a single page or maybe a few pages of PDF on your local machine versus building it in a manner that an enterprise with u uh tens of thousands of c customers maybe hundreds of thousands of customers, millions of customers or perhaps hundred or thousand or tens of thousands of employees. it is non-trivial. When you start building those applications, you run into challenges that you did not foresee otherwise. Um I think everyone knows um what hallucinations are. Um or at least you have heard of it. Um but um uh you know not getting the right right answer you know if the answers are not grounded this is a common problem and u you know when you talk about this kind of problem hey what's the big deal I mean it is a big deal because uh you may be under some kind of regulatory radar if you or maybe you may be uh out of compliance legally if your chatbot or your AI assistant does not give uh a correct answer then um not all chat bots are going to be you know some chat in some kind of assistant let's not call them chat bots anymore chat is only one of the applications but now we are moving toward a world where um um you're moving toward a world where where you have uh uh you have agents actually collaborating with each other almost think like uh you know this idea of that your IT team become will become your uh your IT team is going to become your um HR team. You know, you're you're hiring this contingent workforce that you bring in uh you bring in these knowledge workers who are able to you deploy these knowledge workers and um uh and uh really get get the knowledge work done. And when you I see some of you have posted your introduction, thanks for that. Let me see. Uh we have uh uh Juan from Ecuador and uh and Sami from Toronto and uh who else is there? Let me see. I see Rajay, you are my first name and last initial. Raja I right. So Rajay, you are from Jordan and then I see uh DeArd from uh UK. Um welcome uh to the information session everyone. Um so going back to it when you deploy an application like this you may have other considerations. Is your data safe or not? Uh um why can't I just go and use chat GPT because I have uh some regulatory or legal obligations that I cannot put my customer data in a third party cloud. um then you're talking about you know the cost of operating this um uh chat JPT may may make us look like that these applications are going to be free but I don't know how many of us know that chat GPT clo loses you know millions of dollars um you know every single day I mean close to a million dollar a day I think that's their burn rate and they lose this much money while serving these models right um then you have scaling challenges um you build it uh this kind of application on a single document uh maybe 10 documents but what about a million documents? How are you going to build an application like this when you have your knowledge base is uh so huge? Um then you uh some of the more recent challenges uh like uh does your agent have a longerterm memory or your agentic application does it have longerterm memory or not? um does your application uh can it reason or not? Can it reason and reflect or not? Can it plan or not? Um and um all of these all of these challenges they make uh building and deploying uh these applications and maintaining these applications incredibly hard. And this is what we teach in this boot camp. And uh that's what the boot camp exists. uh and it's quite intense uh but like most uh non-trivial endeavors uh in life anything that is non-trivial right so uh this is going to be a pursuit that uh will take effort and uh and we take pride in the fact that it is actually very intense curriculum but uh um as long as uh as long as uh our learners they know very basic fundamental working knowledge of programing programming language and they understand in general you know they are in the data uh space uh they have done some data related work I mean that's all we need and rest we can take care of during the boot camp so during the boot camp we start with the the the foundations the the whole idea of uh embeddings and transformers and attention mechanism uh in detail so I will show you the curriculum in a bit I mean I will show you access give you access to the learning platform as well or not access but I will show you the learning platform. We take a very deep dive into vector databases. We get into uh a lot of details of lang chain and agentic workflows. Uh how do you create a graph uh for a uh that can plan and reason and also reflect um uh different kind of what we commonly now um that's commonly used or abused word agentic right? So we talk about agentic workflows. Uh we get into the details of observability and monitoring in a lot of detail. We talk about guardrails. Um how do you how do you get your company or your enterprise out keep your company or enterprise out of trouble, right? So you put these guardrails. Uh don't give medical advice or don't uh don't uh quote a price. uh you know u case in point uh Air Canada got sued by a customer uh because uh the bot actually offered some discount uh for bereavement fair and court ruled in favor of the customer. So those kind of things we talk about that of course prompt engineering will be there. Evaluation is the key. If anyone is coming from your classic machine learning background evaluation is actually uh the lifeblood of any kind of not just LLM application or geni application really for uh any machine learning model or for that matter any effort in life evaluation is the key. I mean how do you define how do you evaluate um I mean think about KPIs right so if you do not have a a way to measure the correctness or um you know the groundedness uh of your model and faithfulness and all sorts of metrics how correct is the answer how relevant is the answer and so on um well you don't know whether you built a good application or not and then we talk about deployment and ops uh um and fine-tuning and general challenges in building applications like this. Uh I am one of the lead instructors along with a few other people. I think we have uh the list of instructors somewhere later in the slide deck. Um and then um I spend close to look you know almost my entire day in building these systems. So um and the other instructors are like that. U the what differentiates us from any other company out there is the people who are teaching uh you this is who exactly know how these things are used right so it's almost like uh um you know you can um uh you can learn from someone who has just read a few papers or watched a few YouTube tutorials or you know attended a few talks and then you can learn from someone who has actually built these things and of course the perspective is going to vary. Um it is an in-class uh option. There is an in-class option and there is also an instructorled remote option. Both happen at the same time. U so you can actually uh be in Seattle. The in-person uh session actually happens in Seattle. You can also actually attend the um it remotely. the same uh program 5 days, 40 hours, 8 hours a day. You can be in Seattle, you can attend it remotely. And uh what we have done is I will I will show you in a bit. What we have done is we have created this uh very comprehensive curriculum and the tools and the way the everything is organized. Um it is set up in a manner that uh you know you don't have to worry about installing anything on your computer. We have online uh notebooks uh for everything. If you're doing a project, we will give you a virtual machine. Uh everyone gets their own virtual machine where everything is set up. So you don't have to worry about you just log in as long as you have a web browser. You uh that's all you need. Uh you don't have to worry about API keys uh for from OpenAI or GPU clusters etc. Uh it is everything is actually included in your registration fee. Um and then um this is a typical architecture um and it has uh perhaps I think it is time for us to actually um update this a little bit. I mean this slide is a bit dated but I will actually do a voice over and explain things to you. uh um uh the the core of it is uh well uh the core of an LLM application is uh what we call a large language model. And a large language model can be uh it can be uh it can be hosted somewhere. Um it can be closed source or it can be open open source. uh open source uh you know the notably llama uh and mistrol uh and deepseek and other models which are open source models then uh these models can also be closed source and closed source u most notable one that all of us know are uh open AI GPD uh series models um and uh you can use any of them and then you can also it could be it could be open source and closed source it could be in um in the cloud and or onrem right so um uh so when you have an LLM application you will have at the core of it this LLM model large language model but uh that's not it um for building these applications you also need a vector database what is a vector database well uh you know when you have these documents you take the the semantic representation or for that matter the lexical representation of these uh documents ments or the document pieces or fragments or shards of these documents, you what we commonly call chunks, you put them in a uh in um in a vector database or your traditional search uh index um and then basically store it and you retrieve them when the time comes when you need it. Then you have this thing what we call an embedding model. What is an embedding model? embedding uh models actually take your um uh take your documents or the the chunks of documents as we call them and then convert them into vectors uh and the vector representation of a bunch of text we call it an embedding. You store these embeddings in a vector database and then where does the where does this uh document comes from? These come from your data pipelines. Now in addition to all of this um uh we also need some kind of reasoning on top of it and not even not just reasoning you need to be able to connect to uh connect to different data sources. Uh you have your data in SharePoint or uh or um Dropbox or it could be an Amazon S3 bucket. uh you um you may have some data in um let's say zoom info or um you know some kind of API access it could be in a SQL server in it could be an excel sheet um how do you get all of that data and start processing it you know when we go to chat GPT I mean once again it makes it look incredibly easy but behind the scen scenes there's a lot of query processing that happens you you take the query uh possibly uh you know convert the query into a different uh you know maybe create versions of queries and on top of that maybe hitting the web search and and so on. So all of that all of that plumbing uh it is non-trivial there's a lot of work that happens during that plumbing and then there are tools like lang chain and llama index that actually help you most notable ones there's crew AI autogen is there there is um um there is Microsoft has this called semantic kernel there is a lot of lot of tools in this ecosystem um and uh we essentially we cover the entire entire ecosystem in quite a lot of detail. Um you know when you talk about logging and monitoring um uh you know uh logging and monitoring and uh um observability as we call it right. So uh can we uh can we go and um you know just see how uh you know how much time it took for um you know convertering the query, how much time um how much time uh the inference took, how much time uh reflection took and all of that. You have to be able to know this, right? Um uh but I mean thanks to open source I mean this may sound like uh overwhelming uh but the fact of the matter is if you understand all of this you don't have to build these things from scratch. I see a you have a question there is a lot of components and interactions within this architecture. This is quite a large tech stack. Can you please share the prerequisites that will make us get the most out when doing the boot camp? Um uh so Amit uh yes I mean um uh you would actually be surprised uh I don't know what your background is AIT if you want to type in uh that would be helpful but if you have written code in the past and if you uh you're in general um uh in if you're in general um familiar with the problems that you want to solve uh in this in this space I'm very confident, right? So, I'm very confident that you should be able to actually finish this because we are not going to teach you how to build these tools from scratch. We will teach you. It's almost like we are not going to uh teach you how to build things uh bare bones writing the first line of observability code. We'll tell you what is the best uh open-source tool or what are the best open-source tools that uh you should get for um um uh for uh for implementing observability. We partner with Arise uh and I will show you later. We partner with Arise. Arise is the uh in observability. arises the top uh top uh uh framework uh for observability. Uh then uh when we talk about uh uh you know validation we use mainly we talk about raggas once again undisputed the best uh you know player in that space. Uh when we talk about lang chain is a partner for this uh for this uh boot camp. Uh we cover lang chain extensively. When we talk about vector database, uh we primarily we use VB8 uh which is once again the top the leading vector database in the space openai we use open AI embeddings even though we touch upon llama as well in one of the exercises that I will show you. So we touch upon each of these. So we are not going to drill down and teach you every single thing in a uh in every block. will teach you one thing uh it give you uh a high level idea how things work and then uh and you will be able to um uh you will be able to take it from there. So Amit your background is in applied robotics and AI. I would be shocked you know I would uh I mean you should be able to I mean I'm I'm not even questioning this because we have had and I will show you once I show you the learning platform you will actually see why I'm so convinced. Uh Manoji your question is which cloud platform is used by you during the boot camp? Uh a bunch. Uh so for uh we uh for our fine-tuning uh exercise we use uh runpod u and we deploy things in the cloud. Let me actually I think I can switch. So this is our large language models boot camp and these are all the modules I can get into. Um maybe I can get I cannot really uh given the time that we have I don't think I can go through every single topic here but um what I'm going to do is uh I'm going to click here and there and I will show you and happy to if you want to double click on any of them I'm happy to show you in detail but so this is our module on attention mechanism and transformer architecture I can show you for instance you can see that each module is going to have the same format. I will have um the theory followed by practical exercises and followed by resources. And if you look at this uh this session is taught by Luis Luis Serrano. Some of you may already know him. you know it's uh so um know I would say he's one of the best educators out there when it comes to anything related to machine learning and especially I personally enjoy his session every single time he talks about you know building that momentum even if you don't know anything about genai even if you don't know anything about neural networks he's going to actually get started I mean talk about you know the entire thing and slowly build that momentum and start talking about the attention mechanism, you know, the, you know, multi-headed attention, multi-headed attention and, you know, all of that. So, uh, and then the the beauty of this is that we talk about the intuition side of it as much as we talk about the math side of it. So, intuition is actually very important because not all of us understand the complex math behind all that is happening. Uh, we talk about the intuition a lot and of course explain the math. Um then uh once you know what the how embeddings are created and we have done the practical exercise we'll go to uh vector databases and now um uh once again when we go inside vector databases u um this is um so I can I can tell you right so let me go back uh so on day one we spend time on understanding the overall idea of large language models the end toend architecture uh you know what is rag what fine-tuning, what is pre-training, why token limit, what is context window, um did I say finetuning? Maybe I I think I did mention fine-tuning, what is a vector database? What are embeddings? Uh you know, what are the risks and challenges and so on. Uh then we also on the first day we talk about uh the transformers and attention mechanism because that is the core of how LLMs work. On the second day uh and we do do a little bit of prompt engineering on the second day we get into uh vector databases and when we get into vector databases we are not just there hey there's something called a vector database and let's you know let's go use it we spend good two two and a half hour on the theory side of it uh uh you know we start with a lay person understanding okay so you know uh traditional vector traditional uh uh relational database Maybe you know a graph database. Maybe you need know a time series database. We talk about all of that and then we um draw that parallel. Uh okay. So how is a vector database different? Well vector database is you know the store for taking embeddings and storing them in a uh in a manner that you can retrieve any document uh represented as a vector. uh out of you know hundreds of millions and potentially billions of those documents that are there you are able to retrieve them in less than you know 50 or less than 100 millisecond right so that's that's the beauty of vector database how does this happen we explain this idea of vector searches and keyword searches and uh and something that combines a semantic and the keyword searches uh all of that um uh then um we explain this uh in detail and not just at a very high level we get into the algorithms and the data structures uh data structures that actually are used to make this retrieval efficient right so if you go look up uh we get into HNSW and all of that uh and then of course I cannot go through the entire session uh we don't have time uh but we go through this in a lot of detail and once we have done this that is not it we go into the practical exercises right after this. So if there is you know there's an exercise on vector search and there's an exercise of similarity search and there's an exercise on hybrid search and uh uh you don't need to worry about hey I which laptop or what GPU do I need uh my it would not allow me to install uh anything on my laptop I have a company laptop don't worry about it bring in a laptop that has a browser web browser and that's it right so because we have all these in our online uh cloud computing labs. Let me actually click on uh this generative search lab. So when I click on this, you can see that it is the same Jupyter notebooks that uh you would otherwise get or otherwise use locally as well. And when you look at this, I'm going to go through this uh run uh step by step. Uh we'll uh give you access to a vector database server. You will create your own vector database. You will create your API key and uh we'll give you the you um you will you will create your own opin API u uh API key for the vector database. will give you the API keys for uh for uh inference uh open uh open AI API keys and then we walk through this process explain this code hey what do you think is happening here you are creating a collection of vector embeddings uh what does this mean you're using the ADA model how do I use uh GPD4 well um GP4 now it's not an embedding model but let's say you're using the small uh embedding model or large embedding models by uh model by open AI or you want to use go here how do you modify this uh you see that this is a generative model that we are using I'm using GP4 mini uh we go through this step by step every single thing we explain this right so it's not it is not an exercise on only just giving you the theory side of it we actually go through this uh step by step and uh it builds slowly this then this then this and we take this approach approach slowly we guide you through the process. Um all of this done. Uh and then uh you can see um most of the topics are there u that are relevant and uh here we go. Right. So we spent like a two two and a half hours on the theory side of it. Another two and a half hours on the lab side of it. And by this time you actually really really uh understand what does it mean to have embeddings and what does it mean to have uh uh you store the embeddings in a vector database. Okay. Um then after that uh once that is done uh we dive into uh uh lang chain and lang chain is u if you uh if you look at it u it is one of the most popular uh frameworks out there uh when it comes to um when it comes to large language models and generative AI lot of development happening and we uh take uh this approach which uh once we actually explain to you what are the different problems that we are handling in um um that lang chain can actually help in you know connecting to different data sources you know reading different data data types you know your data could be in a in a cloud storage it could be local it could be in a SQL server uh your data could be an excel file a PDF you know CSV how do you handle all of that right so we talk about all of that each lab um you know how do you create prompt templates you know you click on it we walk through the lab we spend about 8 hours just on lchain totally uh in in all uh we spend about 8 hours we talk about chains how do you build memory how do you uh how do you build uh factor in short-term memory and now there's a concept of long-term memory that is emerging as well so how do you build this short-term and longerterm memory into uh into your agentic applications And then we also talk about you know these state graphs you know how do you build a land graph. Um think about this that uh modern LLM applications uh for the lack of a better word I mean this this has actually emerged within the last one year uh or one and a half year right so it is now mainstream that uh you are no longer just actually doing a single LLM call uh LLM's actually first uh um let's say I ask u my model to ask my application to write a write a proposal for me, right? So, hey, here is the RF an RFP, create a proposal for me. But then you actually also have a reflection node. So, you you write it and there is another a peer review uh that reviews it. It goes back uh with feedback. Feedback is corrected and then reflection. So those kind of planning, reasoning, reflection type behaviors. Uh land graph uh we talk about this once again the labs are there. You can take a look at this. But so we go through how do you create a land graph? How do you set up a retriever there and how do you add nodes to graph uh nodes to this graph? You can see that there is this agent state. How do you create nodes and edges? You can see that uh we are explaining every single thing here. Uh and once again if you can read Python uh uh if you if you can uh uh read Python uh that's all what is uh uh that's all what is uh needed and of course some basic understanding of uh at least an appreciation of what uh you know the general uh understanding of the problem you understand you know you have been working in data space and that's that's important. Um let me take some questions here. So I see Ahmed you have again this question. You can address the following question toward the end of your session. Okay. So I will probably maybe I can handle it now since I'm here. Um can you please share some suggestion of how should we continue learning and practicing post boot camp? We emphasize a lot right. uh emphasize a lot on this and one one thing is uh we have been 10 years in business uh but we uh uh never um we have never actually we are not in the business of selling snake oil here right so a lot of companies would tell you that hey become a data science uh data scientist and always I mean I've been in this space for almost half of my life now right so uh I I I find this uh preposterous I find this actually ridiculous is I can we are not actually in the business of turning some like uh some metamorphosis happening where we took someone and converted them to data scientist or AI engineers that's not our job our job is to give you the right um context inspire you most importantly right inspire you uh in a manner that you find these problems actually exciting right so our boot camps tend to be I mean go check out our testimonials I mean we have hundreds possibly thousands of people on like no John Do and Jane Doe's actually real people with real LinkedIn. Go and ask anyone. We have maybe more than thousand people on our website listed as our uh alumni. Um and basically our job is to actually enable you and um and then give you enough that you at least a bigger the biggest challenge that happens in this space is I don't know what I don't know. Um so where should I start? Uh how should I how should I structure? Should I learn this first or go to that first and what is the minimum amount of uh you know learning that is needed from in here before I move on to the move on to the next topic. Um so if you look at this right so these are the these are not new barriers in learning. What we have done is think of this as a recipe that we have done after this boot camp. We are very confident that you uh you will be able to actually go and figure out um you know what is what else is needed because 40 hours is simply not enough to become an expert in anything right even uh I mean even if if we were a cooking class I mean we 40 hours I mean what can you do I mean perfection happens when we uh go and uh do things on our on our own uh to that end uh yes we have a lot resources. You have access to our learning platform. Uh we have more uh more sessions, more talks. Uh if there's a new invited speaker to our boot camp, everyone who has attended in the past, they get a zoom invite. They join us uh there. Uh we have a lot more uh you know learning uh we have sandboxes, more exercises because uh we don't finish all the exercises. But more importantly, I think uh this is for any any of you who actually attends the boot camp, you will hear me saying our job is not to give you the fish. Our job is to teach you how to fish. And uh and this approach has I mean I've personally taught like more than 8,000 people in uh in uh in our data science boot camp and now a few hundred people in this large language models boot camp. And uh we uh emphasize on um you know uh having that ability to understand uh and ask the right questions. We cannot give you all the answers but we enable you uh how you can ask the right questions. And uh the follow-up question on this in the same flow was does data science dojo provide infrastructure dev environment for post-arning uh for learning post boot camp? Yes. So you will have access to all of these labs even after the boot camp ends for one year and one year is I'm bound to tell you for legal reasons. I mean we we have never cut anyone off but uh you know contractually you know it is one year but people I mean people have had access to this for uh several years and we don't cut them off because well it is we have a fixed cost of this infrastructure and uh um but I mean you have uh any updates that we do if something brand new shiny happens um we try to update within a few weeks of this we are playing catchup actually so anything new happens you can always come back and hey new topics go and click the lab and then uh you can actually run the lab uh and and of course this is running in our infrastructure and we'll support you on that um let me go back and try to finish I think it is already 11:37 on my end um so where we lang chain I've talked about it then we actually talk about uh the challenges in building a retrieval augmented generation application right so we talk about all sorts of challenges from ingestion to scalability um uh to scalability to you know how do you chunk I mean bigger chunks smaller chunks model context windows um you know how do you set up the architecture um uh how do you set up guardrails on it um which embedding model do you use um in terms do you do query alteration or not uh I can keep going I mean it's it's fascate fasinating. I mean I absolutely enjoy uh teaching the boot camp as well because I learn from people as well. people coming from different industries, different backgrounds and and very often people have uh people have other things. Uh you know uh we actually a few people um uh who we get are the ones who have already started building this internally but they realize that I mean there are things that are missing. So they come in with their holes you know the gaps in their knowledge and then um they actually keep us honest and keep us humble right so I mean hey Raja we tried this this does not work so it's a very uh I would like to think that you know uh we get very interesting business perspective legal and compliance perspective and so on so we get uh a very uh very good balance of uh viewpoints actually from all directions both in person and online Right. So online attendees are actually as involved as in person. So we have this classroom where you know we have this uh you know 360 cameras in the center so everyone can see everyone else uh both online and in person. Uh um then the next thing is uh uh I will go back to the curriculum. We talk about the observability and guardrails. Uh we talk about uh you know how do you monitor your application? How do you make sure that your application uh you know you are logging uh uh the events and happenings uh correctly? The time it took, how many tokens were consumed, in what order uh things were called, when did retrieval happen, when did inference happen, all of that uh you know we talk about that again with practical exercises. Um we uh get into fine-tuning as well. In fine-tuning um I can maybe double click on this. In fine-tuning when you get inside finetuning we talk about uh we talk about uh um you know overall let me see which is where is our finetuning yeah this one is our main module. So we talk about fine-tuning uh as well and we uh we start with transfer learning um um we talk about transfer learning we talk about uh you know the difference between transfer learning and fine-tuning what are base models you know what are um you know what are you know instruction uh fine-tune models and all of that so we talk about all of those things uh then I will fast forward we talk about you know full fine-tuning optimization strategies, you know, Laura and quantization, low rank adaptation and uh and u quantization uh strategies you're compressing the model so you don't need as much of a as much of a uh compute that is needed. So we talk about all of that. We explain all the ideas, right? So you know we just don't we simply it's not I mean it's uh it takes it sucks fun out of it if you say hey is there such some such such a thing called quantization and uh and you compress your um from a floating point to a I don't know 8 bit integer right that's not fun we actually explain it and then once you're done with this we have these exercises and now we have a separate exercise where we actually teach people how to fine-tune a 4bit it uh quantized 7 billion parameter um a 7 billion parameter llama 2 model. We are still at llama too but we go through it. Uh we give you that notebook. We give you the f the fine-tuning data set and then we also uh go to runpod uh runpod is a think of that as a GPU uh cloud on demand. Uh we give you access to runpod um we show give you the instructions. I think instructions should be somewhere over here. So if you look at this uh you know you get u we use let me see finetuning LLMs uh these are not resources uh let me see this is the lab okay so uh if you look at this uh we have this exercise we teach you uh and we give you credit by the way uh you get uh credit on your uh in that GPU cloud we show you the steps uh then we go to downloadable resources we give you the notebook uh you know um this is a notebook uh that uh you will go and use. I think it got downloaded. Let me actually open file and notepad. So I don't use this computer for my coding. So but you can see okay this is a ip yb is a uh u it's a python notebook but we give you the code with all dependencies and everything uh and then um we give you the data set and then you go and fine-tune the model we discuss uh we ask the same question go to the uh the model that was before fine-tuning and before and after we compare the answers and we develop that intuition why fine-tuning I guess more fine-tuning actually is uh um when it will work and what would be the challenges when you're building fine uh a fine-tuned model. Um then we uh go into evaluation. Uh similarly, let me double click here in evaluation. Um it is not just uh you know a very high level uh you know superficial treatment of evaluation. We actually get into a very detailed understanding of why evaluation is hard. We start with that uh develop that intuition. Why is evaluation difficult? Then we talk about what do you need to evaluate? You need well evaluation is is task specific. Are you evaluating on a natural language understanding task or text generation task or you know what kind of task it is? And then we introduce you to all the data sets that are there for evaluation. You can see what why would when would you use which data set and then we uh talk about uh some benchmarks. We uh explain some of the traditional language translation benchmarks. And once again I I uh I do not want this to appear as hey I mean this is too much stuff I cannot get it. uh you know we have had people who did not even come from a software engineering background even they were able to actually uh able to actually um finish the and understand the material successfully because I think uh a lot of these things are not as difficult as they appear to be it really it's generally um uh it is how we approach them right so what problem are they solving if you understand the problem uh the the uh the these topics will start making sense a whole lot more. So we talk about these traditional machine learning uh machine translation type metrics. We start with them. we slowly go to more semantic um uh semantic approaches to evaluation and then we go all the way to this thing called raggas and we explain that in a raggas you have multiple ways of evaluation that you're not only evaluating how well your model is performing you al also evaluating how well your retrieval is performing how well you are uh how good your um retrieval performance how how well you are recalling uh all the data and then after that uh we get we go and do the same thing right so uh click on this notebook um uh and then um I can high level tell you uh raggas is for um evaluating a retrieval augmented generation application um then rouge is for more machine learn machine translation type blue roach and mteor uh they are for keyword or lexical similarity uh evaluation metrics. GVAL is when you have one model evaluating another one LLM evaluating another LLM. Uh bird score is once again you know it's uh um it's a semantic measure but we'll spend more time on ragger space metrics because that is more uh more relevant uh in our context when you're building a rag uh pipeline. But if you look at this uh we talk about loading and texting you know uh updating document and metadata creating embeddings we go through this run this cell okay this happened go do the next cell this happened this happened and then we uh actually go and um explain this right so right here uh you know we are uh showing you exactly how things are done okay and on the last day uh on um uh toward the last 3 to four hours uh of the the boot camp. We actually uh give everyone this um you know uh a project and the project is uh once again you have clear instructions. We have set things up uh because we are you know time is uh limited. So we give you um a dev environment set up everyone gets a virtual machine in the cloud. we go and um um give you login. You log into the work that VM you don't have to be a coder. We have set things up um and uh everyone actually goes and builds their own uh and I can call it a chatbot. I mean maybe a maybe a glorified chatbot uh that also can access and reason that can access perhaps uh uh search index. It can hit Wikipedia. uh it can uh also do a Google or Bing search and so on. So those kind of things we do it uh in the pro and by the end of this I mean you have uh you understand uh it end to end. So uh as I said earlier fairly detailed um treatment uh I don't think anyone uh else I mean we are the first boot camp in industry uh as far as you know being the first and also no one is actually going to that level of detail. um you know we are uh I'm very confident that if you want to build these applications uh you know uh you will be able to actually uh get started with building these applications um uh after attending the boot camp um you can be a product manager you can be a people manager we have had dev managers attend this we have also had people who just said I mean I'm I'm curious and uh uh and still you can see that even if you're not a coder uh running these labs it's not very hard for you so you can still get a lot out of it people who did not know coding I don't want to oversell this but people who did not know coding I mean this being able to run this that's not a problem right so you know you can run it and make sense out of it okay this is the step and we talk about during the boot camp we explain all the cells so I mean there are there have been people who are not particularly coders but they they did just fine uh Um so if anyone if you're a product manager a technical product manager who uh wants to guide products uh in this case um if you are a technical product manager who wants to v venture into uh the LLM uh or gen space this is the boot camp for you because uh you cannot guide a product that is safe and compliant and is scalable uh and meets customer needs uh unless you know what are the challenges and we go through those challenges in a lot lot of detail. Uh with that I'm going to actually go back and see if there are any more questions. Right. Um so um Amit you are a PhD student at Purdue. Uh are there any discounts? Amit why don't you drop us a note uh and I will uh I would like to make it work right. So just uh just drop a note and would love to have you. Um then we have uh I think there was a question when you are unable to attend the boot camp on April 7th some who was it one one you able to unable to attend the boot camp on April 7th. So I was told by our staff that uh uh the next one is happening in June right so uh so after that we have another boot camp from June 9th to 13th. I don't know if it is up on the website uh yet but uh you can actually uh um but we will reach out to you right so let's see and there was another question so Leon uh Leon from uh Twitter uh asked this question I joined late how much is the boot camp so the boot camp uh I do not know what the price is the price changes usually uh the the full price of the boot camp is $5,000 that includes everything you know all the infrastructure cost access to the learning resources for one year uh the GPU cloud during the boot camp the open AI keys during the boot camp I will emphasize I mean uh to make sure that it is not a GPU cloud not for the entire year but all the learning resources online the notebooks and etc you have access to this uh for one year um then uh openi keys pretty much everything uh is covered. Uh if you are in person, you're in Seattle, then you know your meals and you know your breakfast, lunch and other uh things that are covered in this uh in the fee. Um the uh the discounts actually vary. uh uh you may have uh um I think we go as far as 30% when we start and as the seats start filling up the discounts actually uh move and right now I think we are at 20% discount so the price should be around $4,000 uh at the moment. Okay, are there any other Okay, sounds good. Any other uh questions? uh anyone I'm happy to show anything else uh that we have this is I know it's a lot to absorb in an information session uh but uh you know uh I I think what we take pride in is we are not here I mean you can this is not your typical information session where you know we'll tell you what we will do and then you don't get to see it you are actually seeing what we have right so I mean this is uh I think this is as transparent as it can get. I think I did not mention that uh you know the people who teach uh at the boot camp they also happen to be the top people. Let me see where is our list of instructors. So this is the technology stack uh that we will be using. You see um some of the technologies that you can expect to be uh um seeing. Let me actually see so I can confirm. Let me go very quickly. Where is the instructor's? Uh oh, I forgot to mention actually. Oh, I this is an important point. So, uh um um we we have partnered with the University of New Mexico uh for uh and you get an verified certificate from the University of New Mexico as a result of this boot camp. And if you work uh for many companies if you are uh getting a certificate um you know yes we are offering the certificate at data science dojo but it it is offered in association with uh the University of New Mexico continuing education. As a result of this, you may want to check with your uh benefits and you should uh very likely that your benefits actually cover the boot camp. Uh for many companies, notably uh in the last boot camp um um for many uh companies um I think last last boot camp we had uh someone from Apple um fully covered by their benefits, right? So it's like getting health insurance from your company. So you don't have to get get approval. Every every company has this central you know benefits type L &D budget. So if you are not sure reach out to us we can actually help you guide it. Go check it out. Uh very likely your company will actually reimburse you for uh the training just uh because of the fact that it is coming from an accredited university in the United States. Okay. uh uh let me see and debra let me come back to your question. So these are the companies that we know for sure. Uh but there are many more uh we know there are many more who actually cover the tution. Um as is I mean if you attend the training you should be able to attend the training uh for sure. So the next boot camp I will be there. I'm one of the lead instructors. Luis will be there. Uh John is going to be there. Sebastian is going to be there. uh sage is going to be there. I don't see caric. Okay. But uh so these are this is all the instructors that we have. Uh uh and then they keep rotating because of the fact that uh you know we are well uh not all of them are available. Finding instructors for this boot camp is not very easy because our condition is pretty strict here. Uh anyone who teaches uh at our boot camp they must be practitioners. They are not some someone who has just who just happens to know the theory part of it. They are actual practitioners right. So um our alumni network I mean so so many people hear from our partners uh I mean what they they say about us I mean the past companies a lot of companies have attended our training uh more than 3,500 uh people uh have attended from our training and this is our next boot camp and I think that should be it and uh one of the question Brad uh your question is uh is the is it the same fee structure for both online and campus versions. Yes, that is the case. Uh um uh it is the same fee actually uh because of you of we have our costs and overhead on of course the support and other sides of it. Um but uh please feel free to reach out and uh you know we'll help you. Not sure what company you work for. Uh but perhaps you know your company might be actually um that boot camp may be covered by your uh tution. There is another question. Okay. I think this is the same question. Okay. Um are there any other questions? Uh anyone? Okay, that is great. Uh thank you so much everyone for attending and I look forward to seeing some of you um at the boot camp. Thank you so much. Thanks Amit and Dea probably. No. face.