Large Language Models Bootcamp - Information Session

Key Takeaways

rajal I am the chief data scientist here at data science Dojo um so um what we are going to do in today's session is we'll go over um in general the curriculum of the boot camp what is the boot camp about and what this is not about um who should attend um who uh do we have as our Target uh audience or uh potential Learners uh we'll talk about that and uh and we'll go over the logistics we'll talk about uh you know the the learning platform that we have the labs the content uh the curriculum I think I already said curriculum and then also options for reimbursement uh many companies actually do reimburse uh uh for this uh uh if you attend the boot camp um most employers actually uh since we are um offering um a certificate from accredited institution uh I will go over how to actually get reimbursed and use your HR L&D budget for um getting reimbursed for the training um so with that I'm going to go ahead and get started um we have been around for quite some time actually so one of the oldest players in data science uh and analytics and machine learning upscaling and now we are also happen to be the first boot camp uh on the planet um when it comes to larar L language models and generative AI uh tons of graduates tons of companies who have uh upscaled by either um either training um uh training um their employees in bulk um uh through our trainings or by sending their uh their uh Workforce to one of our public offerings um so how did actually this boot camp start uh we are as the name says we are data science Dojo but how did the llm boot camp start uh when we when we look at um the chat gpts and bars and claws or Bing chats of the world and the Geminis of the world um we uh we see some of the use cases of building u a large language model application or um you know a chatbot as we call it often uh even though not all the cases of large language models are chatbots but let's let's call them chatbots for now um there is uh you know you you have this illusion uh when you use it hey the these things work very well and um and we'll just adopt we'll just bring them in you know buy the technology or bring it on board um and uh life is going to be good the problem is uh that when you decide to use these uh uh what we call um you know the large language models in an Enterprise scenario uh things change there are actually uh first and foremost uh a lot of challenges challenges around data in AI governance there are challenges around um regulatory issues um so your data has to be um you know it cannot leave your premise so some of the some of the classic you know traditional analytics and then classic predictive modeling uh and data science challenges um that are there um around uh where does the data go where does the data set um so proprietary data regulatory uh challenges but then there are other challenges as well what let's call it AI governance um you know our un regulated industry if you if you want to use an llm um what happens if your if your medical llm hallucinates what happens if your legal bot actually does not give a good answer so there are a lot of issues around um broadly speaking the data in governance category and then there are some technical challenges um well you have a lot of data what if your model's context PR is big or small uh how much uh how much are you spending on uh uh how much are you spending on uh inference uh how much are you spending on uh building or rather fine-tuning these models um uh how do you evaluate these models uh this is not your true false fraud or nonf fraud or customer churned or non- churned uh type predictions these are um these are um natural language um answers that you have to evaluate how do you actually evaluate um your uh llm after a change in the prompt or after you upgraded the model after you uploaded new new documents in your rag pipeline um so you know how do you ensure um that your model actually continues to work um or your application I should not call it a model your llm application it continues to work uh how do you handle the brittleness of prompts how do you keep the knowledge updated how do you mitigate hallucination how do you detect mitigate um detect measure and mitigate mitigate uh hallucination um whether you use open source or closed Source models there are a lot of issues uh that applications like chat GPT and Bard and um you know CLA they hide from you uh and you know when we are using them as a consumer we don't realize and when we bring this llms in to our Enterprise suddenly we have this realization it is hard it is hard to build llm applications what do you do in that case so this 40 hour boot camp it is a 5day 40-hour boot camp is fundamentally designed to get you from uh you know as a very beginner or zero let's call it zero uh you start with coming in not knowing anything about llms not knowing anything about generative Ai and then um over the next 5 days starting from Monday uh all the way to Friday 5:00 p.m we walk you through um the different components um uh and I will go over the curriculum in a moment we walk you through this comprehensive curriculum uh designed by practitioners uh so everyone uh so I teach uh good 30 40% of the boot camp myself um so it is designed in a manner that you become a practitioner and we have uh ex I mean I will show you there are uh people who have attended uh we are um we have done trainings for big uh entities uh globally uh in llms and we are actually we have some trainings lined up for uh big uh corporations who actually are relying on us for upscaling in large language models and generative AI um so we will be covering most of the mainstream tools and tools and libraries and packages um the the most fun part that I actually personally also enjoy enjoy is the Hands-On component of it we always uh we are known for these trainings um not just the llm boot camp any training that that we do for practitioners uh it is a good mix of theory practice um and then we have uh to toward the end of the boot camp we have a comprehensive project where we actually support you to build an llm application um we are uh um a lot of employers actually um within their HR budget uh this is only some of them uh but many employers uh the fact that we are an accredit uh we offer a certificate through the University of New Mexico um we are um because of our relationship with this uh accredited institution uh a lot of companies you may not be aware but uh you you need to check out or reach out to us you may not be aware but your company uh has some Central L&D budget which uh this boot camp very likely will be eligible for and so you know I can drop some names I me so some of the names that you see most notably Apple we have someone from Apple attending the train our next training funded by uh the company uh we have someone from Boeing attending the training and uh you know Bank of America and so on so we have people actually attending the training from uh uh these different uh companies um so go check it out and if you have any if you need help our help I mean we'll help you with the uh with the process because we actually are working with many companies in that regard um and regarding the training all you need to do is bring your laptop that's it um um so many times people say um you know what software do you need do I need to install when we when I come in um nothing really I mean as long as you have browser enabled computer everything else uh your GPU uh clusters will give you access to those as long as you can you have a web browser you can access um we have jupyter notebooks uh um with python code uh and hundreds of code samples uh we do like 40 to 50 uh exercise 40 to 50 practical exercises across different areas and a Hands-On project so we provide you all the resources including the the cost of using open AI uh or any other uh cloud computing resources any other large language models we actually uh are responsible for uh that cost uh our partners uh we uh we have partnered with some of the leading companies in uh this space uh to bring the boot camp to you um so Lang chain vv8 neo4j and arise and security AI all of these uh are actually our partners just a moment I actually have uh cold so I need to just out coffe and come back sorry about that okay um so uh we have all of these uh uh Partners in the past we have partnered with many other companies so depending upon you know this is for our Seattle uh roster of Partners uh but uh so Partners depending upon their availability and all of that you know they are in and out of the boot camp but this is our roster for the next Food Camp um and then we also try to bring in uh people from industry uh you know as you mature as a technologist as you mature as an engineer you realize that you know building a real world enter uh an Enterprise application it is much more than just knowing the technology part of it you need to know uh real case studies you need really need to know the uh actual practical imp applications uh you also uh need to understand you know what are the challenges what are the regulatory uh concerns that you would have you and and that cannot come in uh without bringing in some practitioners so the people who are coming in uh it is you will not be actually getting people who are um you're not going to be getting people who are merely you know slide presenters uh when um when someone comes in and talks about to say embeddings I mean uh you will um or Transformers you will actually see that they they are known for what they are teaching right so it's not just some uh uh someone who just happens to know and read a paper and attended a conference here or a talk there they actually know things inside out right so uh I am a practitioner myself uh we have built uh uh a platform uh that is being used by companies I mean north of um 00 billion dollar companies that are using our platform so um so you will actually get all the insights from um from me as in uh um I mean I can call myself uh uh the the main product manager the main architect of the application I and then uh and other people who will be presenting they're also they know exactly what they're talking about so what I'm going to do right now is I'm going to go over the curriculum I will show you what we have uh in the boot camp uh I will do my best to actually cover as much as I can um and based on the questions that we have I can go and uh uh I can go and um you know drill down um and Kevin maybe I can answer a question right now since I see it is hugging face a partner no hugging face is not a partner yet and um but that's a good idea I think we should reach out to them uh what the our partnership ecosystem the way we have built it is you know we we have partners that that are dedicated for instance I mean give I give you an idea for Vector databases um uh vv8 is a partner and why vb8 because they are one of the uh leading Vector databases uh in industry and one of the the I I uh they are the one of the few uh AI first or AI native Vector databases because other everyone is building a vector database even you know um uh you know mongodb as now a vector functionality postgress has Vector functionality uh SQL servers has Vector functionality traditional classic search like Lucine and solar they also have Vector functionality so everyone is trying to hop into onto that Vector database bandwagon but we have picked the best and the you know the best uh best Vector database then um uh I will show you in a moment um uh knowledge graphs are actually have shown um have um I mean uh there is evidence that knowledge crafts actually improve the performance of your rag uh pipeline so we have partnered with NE 4J and neo4j actually is our partner for knowledge cfts so we have someone from neo4j they come in we do Hands-On exercises and so on um then we have Lang chain and Lang chain is probably Lang chain and llama index primarily I mean these are the two uh uh foremost uh Frameworks that are used uh in that realm uh for for compute we are partnered with runpod once again uh a well-known name for observability we are partnered with arise um we have there are other players in the space but I hope this answers the question uh we we do not partner partner for the sake of partnership the Partnerships are actually quite meaningful and uh the our partners are actually involved in uh in delivering the boot Gap uh so Zach Martin on Twitter asks if there is informational link I can send to some of the mentees I work with that are looking for structured course workor around this they could not attend yes uh Zach uh we will be actually giving you um we will be providing you uh the link and the team will actually um share the link at the end of the boot camp and I think they may already have shared a link uh to the recording of this session um so I will go over the curriculum now so 5 days 40 hours we start at 9:00 a.m. um uh on a Monday uh and then once you come in on 9:00 a.m. uh people arrive earlier uh those who are uh so be you can attend in person in Seattle uh the next boot camp that is happening on February 3rd from February 3rd to uh 7th um so you can attend the boot camp in person in Seattle or you can attend uh it online some people they decide they want to attend it from the comfort of their home for for any reason so the Boot Camp starts uh at 9:00 a.m. the instruction starts at 9:00 a.m. uh but people arrive early I mean having breakfast and all of that at 9:00 a.m. we start with the overall idea of uh we spent a very good idea to get grain and understanding of the breadth of the overall ecosystem so we start with uh um well um you know how do how do machines actually learn and uh a very generic generic basic idea of uh you know how does machine learning work then you go on and understand you know your um what are you know what does really embeddings mean uh we go and talk about Vector databases we talk about Rag and retrieval augmented generation we talk about finetuning uh we talk about context windows and you know context length uh we talk about you know those those token limits and all of that we talk about the risks and U you know what kind of challenges that you will run into when you're building an llm application uh we talk about the any kind of security and other uh challenges so by lunchtime on day one you are very very comfortable with the the general appreciation you appreciate the entire um you know the bigger picture of what is going on and then post lunch on day one that is where we start actually to dive deep into the respective components so think about this uh the breadth first so you go uh in breadth you first uh understand the generic idea generic idea and then once you have understood the breadth you go and start to talk about uh um the more finer details of the respective components let me just a moment okay so the first session uh in this uh let's call it the the first uh in-depth session uh post lunch on day one usually our lunch breaks are 35 to 40 minutes so Luis Luis Sano he actually teaches the first session and we start with a very basic idea we don't expect you to have u a background in machine learning we don't expect you to have uh very deep understanding of neural networks or you know uh RNN CNN we start with a very basic idea really uh getting you up to speed and all the way building our building this momentum building this uh um solid understanding uh and then suddenly you know things start coming together uh you know we talk about uh you know how do neural networks work and how does a perceptron uh work and then uh you know leading and uh culminating in um talking about attention mechanism talking about uh the Transformer architecture talking about the encoder decoder architecture and how does an llm infer and and so on um these slides are taking some time to actually load up so let me see I hope uh this makes sense how you generate language and you can see that we slowly get into the Transformers business we talk about this very detailed session um uh really Lis is uh I mean I look up to him as a I mean I'm an educator at heart but I mean I cannot you know Teach as well as Lis Luis is amazing so he teaches it really well uh um and uh you know this is one of the most enjoyable sessions uh if I may um once this is done now we come back and talk about uh okay now and and we have Hands-On exercises but then we talk about okay now that you have embeddings what do you do with these embeddings well we put these embeddings uh and we store these embeddings on into a vector we load these embeddings in a vector database right so well what is a vector database so we then we spend 6 hours roughly 5 to 6 hours on Vector databases and the way this works is we start with a theoretical discussion very much very in-depth theoretical discussion starting from something as simple as how is a vector database different from a your traditional SQL or no SQL database so we start to talk about that uh go over it you know we step by step we are building this generic idea uh okay so what does a vector datab do what kind of searches like it offers a vector search a keyword search what about if I want both uh hybrid search uh we talk about you I want a bit of SQL as well a bit of SQL like functionality where I can filter things as well so in that case um uh you know how do you filter things and uh you can see that very slowly this ISS about 169 slides you know it's we don't cover every every single slide but you know it's a 2 and a half to three hours uh session just on the theory side of it um then uh we actually talk about um we actually talk about um you know how is how does indexing happen in uh Vector databases um and then you can see that slowly gradually we are going to build things I mean these are some of the foundational Concepts and we we take our time to actually explain uh the foundational ideas foundational Concepts in Vector databases so you can see that I will uh you know I will of course not go over every single slide here I'm going to uh you know show you what else do we do now once you are done with this um you know well you understand the theory what do you do uh in practice so uh if you look at this you get uh thanks to our partner vv8 uh so you will um you will get uh access to a um uh VB server you will go and deploy your own Vector database and then you will set up uh uh you will have these exercises on Vector search similarity search and hybrid search let me click on hybrid search for instance uh so there are six to seven exercise sizes Depending on time sometimes we do six sometimes seven uh you know this is a lot of practical work um so we do this you can see that you have this uh you know connecting to vat you have these keys and now uh creating a collection think of collection as a table in your uh table in your uh SQL database so we create a collection uh and then we perform some searches and you can see over here importing the data I'm importing some data uh and we talk through these exercises uh run run run stop maybe change from uh change this from a mammoth to an elephant uh change it to a fox or I know chameleon or a gecko just uh change it and see how uh what documents are retrieve from your vector databases and you can see uh and then try to adjust and tweak uh this Alpha parameter that uh actually decides how much of how much of keyword versus how much of sematic search do you want uh and you can see that uh you know it's a uh it's one question that I get is hey I'm not a programmer uh that's okay right because all you have to do is you know you are going to run run run and uh you just have to be able to you know read python code and that should be okay and uh and after that even if you are uh I would say if you want to be a technical product manager or founder in um in um or maybe an um an engineer u in uh while building an llm product um you know you should be able to actually very very easily follow this content um but by no stretch of imagination this is easy content right so it's tough uh of course uh we do our best but uh it's intense and but the intense in a good way that by the end of the uh boot camp you should be able to build your own applications uh uh so you can see that we are talking about a lot of different ideas this is for filtering uh Etc and then now you can see that we have an example of gen generative search and generative search think of this as a um as a very very basic example of retrieval augmented Generation Um then we talk about multi-tenancy Vector compression why do you need Vector compression we talk about that in Theory now you're seeing Vector compression in practice Vector compression is needed well then you have uh too many vectors uh a typical uh rag um a vector database set up with a in a rag pipeline I mean um I mean tens of millions is definitely a possibility and then for bigger companies you're looking at hundreds of millions or billions of embedding so how do you compress this Vector so the searches are as efficient as possible uh sematic caching um so you do not want to uh generate tokens or generate uh responses every time uh how do you cache things uh once we have done all of this then we go to um the knowledge graphs business and then we talk about knowledge grafts So Adam CI from neo4j he actually comes and so this session uh the theory part uh Sebastian uh from vv8 he does it uh and then depending upon you know him or myself uh we cover the Practical exercises uh and then uh for knowledge graphs we'll have Adam CI from uh neo4j so he teaches it and will'll talk about how do you blend in your um how do you blend in a Knowledge Graph into your rag pipeline um once that is done now that you understand what rag is then we talk about you know some of the um some of the uh nuances in um of Lang chain uh so what is Lang chain uh if you are here in this uh the informational session likely you have heard of Lang chain but uh I will still mention think of Lang chain is a framework as a framework that allows you to uh the uh that allows you to build llm applications and many of the the com most common tasks are actually already um U available as a as a library as a as a function called so we talk about let's say when I go to model iio um I will give some of the examples of course the curriculum has a lot of moving parts so I will do my best to actually touch upon as many areas as possible uh so we talk about prompt templates so if you don't want your prompts to be fixed you want prompts to be adaptive uh prompts changing uh let me click this uh click it here so you can see how easy it is that uh you know as long as you have computer with a web browser you click on this the lab po pops up uh we give you the API key you um you know I will run run run here and uh I will plug in my uh openi key and then after that there here is an example of um uh how to create a prompt template then we go to you know how do you um give it you know this few short examples so this is another exercise and a few short examples so if you look at this and then we discuss right so it's not just run run run uh we pause on cells someone asks a question we discuss uh Hey what if I change this from a tiger to a lion or or an elephant or a veale or a pirate uh so all of these examples uh um really internalizing what is all of this about um then we me just close this up uh then we go to retrieval in retrieval uh if you can see there are exercises how do you Lo load documents from different data sources um your SharePoint your um your Dropbox uh your Google Drive how do you actually go and load uh documents then we talk about uh you know different kind of chunking approaches how do you chunk data uh using uh L chain we talk about connecting to different types of vector stores Etc um and then uh chains uh how do you break down your llm function calls instead of calling it in a s one go how do you actually call them as a connected um uh how do you call call an llm as a um and not just as single function call but as a con to connected function calls and I think when this lab pops up I should be able to show you this practical exercises um so you know for in in this case I will start with this exercise uh very often I teach this session so I would talk about okay what is this Library we go and check up check okay what what is this doing uh llm chain uh this is a new one uh so chat openi by now you have already covered it but chat open but when you reach this exercise llm chain is the new one prom template you already know by now uh so we we talk about this explain all of this and in this case you can see that uh um the this template is uh we have broken down this prompt into three uh successive prompts and then it ex we ex execute then we talk about other kinds of chain chains like router chain and map reduce chain and so on uh we talk about memory um uh and memory is the think of this as the easiest easiest analogy would be is um when I go to chat GPT when I start typing uh I ask a qu uh I ask a question chat GP response but when I ask a next question um you all already have this ability uh to um the the your application should have this ability to interpret that your next question is in the context of the previous question so we talk about different kinds of possibilities of how you can create uh different kinds of memories in your uh uh in your llm application okay um then we talk about um agents right so it's a it is all the buzz you know uh it is all the uh all the hype is around agentic behavior and uh we actually start with very simp simple search agent simple very simple very basic Wikipedia agent and then we slowly build um more of a multi-agent system um and culminating in um uh multi-agent collaboration uh using langra um and once again if you have heard of Lang chain possibly you have heard of Lang graph um but the idea is that when you allocate um when you allocate uh or when you assign uh a prompt to uh when you assign a prompt to your llm application your llm application actually plans or plans how you are how are you going to ask these questions and then it breaks it down um you know it plans how it is going to be um answering the question first I will go retrieve this from a vector database then I will go to my search engine and retrieve this information then I will go call this API and retrieve this information and then it combines all of those things um so you can see that we have uh these uh we we talk about you know uh how do you define different tools that you have in your uh in your uh graph and how do you create the graph you're defining the tools you can Define the agent nodes and uh defining the edge logic and so on um you can see um some interesting examples here um very detailed uh and very comprehensive of session we spend about almost a day 6 to 8 hours um um 6 to eight hours on just L chain because L chain um is very crucial for those of you coming from uh a a classic machine learning background think of Lang chain as more like uh it's I know it is not a very exact analogy but think of this as psyched learn of llm applications right so because I mean it is so comprehensive if you want to read a file well there's a um there is a library for this if you want to transform uh you know join uh you know um if you want to uh do some feature engineering if you want to call this this uh model or that model uh very similar to that a very comprehensive uh framework that allows you to do stuff at uh lightning speed um so uh after L chain is done uh we actually also talk about fine-tuning uh we get into a lot of detail for fine tuning um the same drill I will speed up now I think uh I will leave some room for questions at the end as well let me see I think some questions did pop up in between okay where is the boot camp mmed Kahan so mmed uh okay let me go back uh all the way there are more questions so so where's the boot camp the next boot camp has happening in Seattle we have done these boot camps in Seattle and online so the same boot camp is happening at the same time in Seattle and online some people join remotely some people are actually uh in the room in Seattle uh but um um uh we have done these boot camps in other cities as well but at least the next boot camp is happening definitely in Seattle [Music] umu which company is this yesu uh I can guarantee that people who are uh um uh um who are the instructors they have very very um I would say they have extensive experience of what they are uh talking about um so um I'm one of the lead instructors for this uh training and I spent for the last two years I have spent 60% of my time and my 60% of time my time think about like maybe eight hours a day uh I've spent U most of my waking time uh in actually building an application uh from product management to technical discussion to architectures and re architectures uh not just from in perspective but also from the software engineering and large large scale distributed system design um so um and uh and the application that we have built right so anyone can build an application right as a matter of fact I mean you can build something in an hour if you'd like to but I'm talking about an application where um our most recent client is an 0 plus billion dollar company and their entire uh tech support stack uh or their technical support sef3 and sef4 um um support uh queries they are going to be resolved by what we have built so I'm I'm I can safely say that uh you know we are the the Rost of instructors is actually um some of the leading people in Industry uh at this point um let me see which companies the instructors have worked for unfortunately I worked only I mean if you subtract my experience as an intern I've worked for only two companies myself uh you know Microsoft and data science Dojo right so yeah it's not a very extensive experience but I mean uh on that was on a lighter note uh you can rest assured that I mean uh whether they have worked for one two or five companies they are practitioners so I can assure you that part uh let me see [Music] what other questions are there okay Charlie your question so multi-agent system is just a collection of functionality that has not been organized to accomplish specific tasks and can be built by ever increasing layer of complexity and functionality so really just a system development approach that allows inclusion of llm and gen capabilities Charlie the uh yeah I will have to be very careful in answering this question so my quick response is yes multi-agent system is um essentially you're you're looking at um um basically uh you have a bunch of different uh what we call tools right so um you have uh a web search and you have an API Call to um something and you have uh uh you have a rag application you have Vector database you have a SQL database um uh multi-agent system uh at the a very basic uh simple application of a multi-agent system would be is that uh you have these uh a bunch of different data sources bunch of different uh uh sources of information and you are um you have a planner that actually coordinates uh your actions um across different uh different kinds of tasks uh I I should do this task and followed by this task and this task so it has this reasoning ability uh and I I I'm oversimplifying it of course uh it's it's quite quite difficult to the build these systems in the beginning of course uh and then um so there's something called assistants and my definition of assistants versus uh agents is assistants are more they are less autonomous you have to keep interacting with them agents are that uh you know they can still work when you're uh behind the scenes right so um and you can you can um you can connect you can assign them tasks uh and or they can take actions on their behalf and Charlie feel free to shoot another question if this wasn't satisfactory because I have to also wrap up the presentation happy to answer if you have more questions um I will go back uh now so uh we we do fine tuning in fine tuning we'll uh first go to um we'll go to uh we will go on to explain um um the general idea of fine tuning why do you need fine tuning what is pre-training why not pre-train all the time why fine tuning um and then uh what is transfer learning how are they related uh challenges and fine tuning we talk about you know uh replaying the data again we talk about catastrophic forgetting uh you know the some kind of CH the the challenges that happen when you're fine tuning then we um talk about uh uh optimization when you're doing fine tuning quantization low rank adaptation uh Cur and then this is uh about 2ish hours hour and a half to two hours on the theory of fine tuning then we um then we uh go ahead and spend another two hours on fine-tuning uh L 2 7 billion uh model uh 4 bit quantized model to be precise uh and then we compare the model that was not fine tune versus a fine tune model uh we give you uh all the resources the GPU cluster so the cost we bear the cost of fine tuning uh and all you do is just go and you know log into the relevant place that we uh ask you to um then we talk about um uh observability uh and monitoring uh so imagine imagine that uh you are uh you have uh you um it's almost like I mean uh wouldn't you want to log uh you know logging is important uh uh so uh what happened right so how how did you break down the prompt I mean uh what were the altered queries whether uh whether we hit R uh rag uh whether we hit a vector database what chunks were returned from Vector database um and then um so really being able to tell what is going on uh do you have any prompts that are toxic or not um so we have this session arise which is once again a leading um framework and they also have a SAS uh uh option so um uh we talk about uh observability in depth we have Hands-On exercises for those as well um we uh go with that uh then we we spend quite a bit of time on evaluation evaluation is uh interestingly enough I mean I've been doing machine learning for a long long time um uh one of the things is uh I uh that I always found fascinating is people do not know how to evaluate these systems and I I know I mean yeah I can calculate accuracy I can calculate Precision recall really understanding the spirit of evaluation not many people understand it right so what we do here is we talk about uh perhaps I can show you a sample of the lecture slides if that makes sense so it it is not right so many of many evaluation modules they might look like hey we are uh you know uh just a moment okay okay let me just allow me I think this light deck is a bit out ofd and you can see that nothing is scripted here these are actually real this is real stuff that hence this uh okay I think I found the correct slide deck here um um okay this one yes so I will bring up this actual slide deck now now take a look at this right so uh so we are practitioner so this this slide deck started like a year and a half ago it looked very different and now after uh close to Six Enterprise clients uh and uh uh a lot of feedback we have come down to this that we are talking about you know um so evaluation it is going to be in a lot of detail then we uh when you look at this uh now we are talking about well evaluation data sets and evaluation metrics uh we start with what are some of the evaluation tasks right so if you look at this um are we talking about uh translation are we talking about understanding are we talking about text generation are we talking about dialogues are we talking about evaluation for bias so we talk about all of that we are not actually uh anything that we know by now we are going to teach right so of course I mean this uh space is evolving very very uh fast so anything that is out there uh we try to update um and it is hard to keep up but you can see now that we talk about different types of different data sets that are out there what kind of uh benchmarks are there we talk about start with the standard machine uh machine translation uh evaluation metrics and we go all the way to you know rou is also your traditional metric for machine translation we but B score is a you know somewhat semantic um evaluation uh approach and then we go all the way to uh you know ragas and ragas is uh when you evaluate not just your response but your entire um uh entire rag pipeline or rag setup so you can see that uh you know as I as I said um this is a practitioner's view of things and then when you get a practitioner VI of things U um we have included anything that we thought uh um so uh so um these some of these slides are I wish I knew so when I started uh building a year and a half ago uh we would have built something much more robust if you knew and in some cases if these things have existed so ragas was not around I believe or at least I did not know that dragas was there but we incorporated dragas in our own product um only like 6 months ago so we have evolved quite a bit and then once you understand the theory side of it now you can see that we have exercises around all of these uh all the um the words or the the keywords that we had dropped U moments ago we have it uh those keywords here uh then we um also have uh this exercises around uh deploying a fine tuning uh uh model um and finally we have uh uh we have this uh project and uh you uh we give you boilerplate code um and you start with that boilerplate code um and we give all the resources right so I I wish we had the time to actually go into the detail but we give you all the resources uh we explain how things are working and uh we ask you to push your code into a GitHub repository then you we uh you deploy uh your application in streamlet we ask you to hey can you add this chain um can you uh add a search agent can you add uh can you do this can you do that so we slowly gradually actually um you know everything that we have learned we put it together um what else let me go back and see if I have anything that is left I'm not going to go into this uh okay I I should go into this right so this is roughly the technology stack that we use uh we will be um primarily we use openi or for the work that we do you get openi keys we touch upon L 2 um you know hugging face we use it but not the on the model side but hugging face does get uh uh leveraged Azure search vv8 uh you know uh uh Lang chain so you can see that uh uh we will be touching upon a lot of things in this ecosystem so this I've already talked about this uh the general idea of uh you know this is the curriculum in more in a uh in a slide fashion but I have already showed you this I don't need to go over these bullets again and again um this is is our set of instructors I can confirm for the next one I'm going to be there for sure Lis is confirmed Adam is confirmed John is confirmed Sebastian is confirmed uh Sophie is going to be there Sage is going to be there and so you know it's it's a good mix of um uh it's it is a good mix of uh uh instructors and really will all of them be there at the same time I wish we could um the problem is that uh you know these are busy people they're not just I mean they have their other um high-profile startups they are you know busy they have their day jobs as well so it is going to be actually hard to have all of them at the same time but I mean the ones that I said I I mean those people are going to be there and this is our partners I mean they love us the partners who are collaborating with us and this and in addition to that customers uh this is only some of the companies that have attended our training uh we have had people from traveling all the way from Australia as far as Australia uh I think on the planet it is as far as it can get um so we have people traveling from Australia uh the Middle East um you know people attending from Europe South America uh and so on um and our next boot camp is happening in Seattle from February 3D third to 7th uh so same boot camp is going to be some people will be attending online some people will be attending in person um happy to answer any other questions let me see uh the questions are let me very quickly see okay uh what background do I need um uh what background do I need uh uh to attend the boot camp um so um really uh you don't need to be so typically the qu question is um even if you don't know um uh machine learning uh your uh you you know the classic machine learning I don't think you need that uh even if you are not a programmer if you're not a coder uh not a developer that's okay um we have tutorials uh that can ramp you up on python you don't need a whole lot of python if you can read python code that's pretty much all you need uh to to be able to attend um what else right so a general interest in uh in uh building llm applications should be uh fine okay um let me see I'm going through the comments here in case there are any uh any questions usually there is one more question that gets asked I don't see it here uh you know proportion of theory versus practice so um it's a good mix uh so we don't uh overemphasize on one thing because some courses they can be very very theoretical right so um really if even if you understood uh the entire Transformer architecture even if you understood the attention is all you need paper with all the math in it um you cannot build a good application because uh building a solid uh llm application it is much more than um just knowing the AI component as a matter of fact I mean I can safely make the statement the 95% of building an Lam application is actually good solid software engineering maybe there's 5% AI you know so you still need to be a great software engineer or a great software product manager or great software project manager yes there is AI component to it but under the hood um you actually use the AI compon component as a blackbox so it is really uh AI is mostly for the most part uh AI uh is being used as a blackbox and then you are a good architect you are a good plumber uh you know glorified plumber who's actually bringing things together and building things you have to have a very good understanding of business side of it you have to have a very good understanding of the uh risk side of it um and uh the security implications the regulatory implications you have to be able to I mean you have to have a good sense of ux design so uh we talk about all of that uh from our experience we share our best practices uh we have sessions where we actually talk about uh I did not mention those uh slides because they don't fall under you know the standard category but uh uh you know there is a session called challenges in building rag applications and then over there we talk about the challenges that we faced and we think that uh anyone who's build uh setting out to build an llm application should be familiar okay and I will just take a look in Cas there is another question and uh I think this is it I do not see any other questions uh I will just give it a few more seconds if there are no more questions okay uh thank you so much everyone for attending today uh okay what is the total cost uh so uh the total cost uh is as we speak uh on the website it is $3,500 after a 30% discount um the so we what we do is um the uh the prices actually change based on um you know the first few seats uh go on 30% discount then the next few go on 25% discount so it's more of that uh how early you register how many people have already already en enrolled Etc so uh and um I've not checked the price myself uh in the last few days but I the last I checked it was at 30% discount uh and $3,500 if you work in the United States um uh it is very likely that your employer uh already covers this boot camp but you'll have to reach out if that is the case um and we can help you really sort this out okay and uh sounds good thank you so much everyone I will end the call okay once again there any boot camp in Dallas interestingly Dallas is the only place where actually a friend of mine he's insisting that you need to come here and then we may have a boot camp in Dallas so so linga ready if you are interested please drop us a note uh it's just an interesting uh coincidence uh uh we may have a boot camp in Dallas uh but I cannot confirm I mean for now Seattle is the place uh um sounds good um and then uh um my company has some programs to cover the cost how can I get a brochure I plan to attend virtually uh is it possible that you reach out since see you are on the zoom call so you can click on one of the uh links that have been posted in the webinar chat uh go ahead and click and reach out to us and we'll uh we'll take it from there we will be more than happy to help you out okay and uh yeah last minute questions are coming in I think that should be it um thank you uh everyone and I'm looking forward to seeing at least some of you at the boot camp have a great rest of day