CS50x Workshop on AI

Key Takeaways

just wait for the tiles to fill in and Rong sh mind making sure the chat is to everyone yes all right testing testing one two three testing can you hear me okay World okay wonderful so nice to see so many faces and tiles uh if you have not already please feel free to say hello in the chat and where you are in the world and just to warm things up R Shin in just a moment could we also invite everyone to unmute for just a few seconds to say verbally hello world if Rin you want to let me know when when you're un hello hello everyone hello hello everyone nice to meet you sorry David you are on mute someone need to unmute you can someone in the control unmute David thank you yeah go again David sorry uh you were muted in that process that's okay all right you missed the best part but just a quick summary before we begin today uh so this will be a workshop in two parts which is to say the first half will be largely me and it'll be a look at how cs50 has been integrating AI artificial intelligence into the course itself if any of you have been working on cs50 over the past few months you know already the rubber duck so this will be a look at exactly how that was built the second half will be primarily with cs50 own rinl he will join me in the latter half and give you a much more Hands-On demonstration of how chatbots like the cs50 duck work he will walk us through the process of building a small chatbot per my email earlier this week you are welcome to try to follow along at home however because open AI the company behind chat PT requires for API use that you sign up in advance you potentially give a credit card we're going to assume that not everyone one is following along so Rong Shin's part will be more presentational but we will make available all of the source code that he writes so that after the workshop or while you're rewatching the recording you can indeed follow along at your own pace without feeling like you are falling behind um Rin I only see two screens of tiles I am pinned on the other one right now can we fix that all right uh this is now our tradition as many of you know of a souvenir photo so if you'd like to put on your camera if you would like to opt in if you do not want to be in this photo turn your camera off momentarily but otherwise put on your happiest smile and we will wave here while we're on and team take a whole bunch of screenshots and what we'll then do using AI ironically afterward is stitch them all together with software to make it look like we're all in one I'm going to move my lips now one big room just waiting for the go-ahead from Rin yeah all that thank you technically we don't have to wave because we're just going to take a screenshot like this but thank you for partaking nonetheless all right give us just a moment to resume and what we'll do and I hope you won't mind we'll present our material on cs50's use of AI we'll save questions for the very end if that's okay feel free to chat or ask questions of each other some of cs50 staff are in the chat but for for the most part we'll save chat for the very end then we'll transition to the second half with Rong Shin so even as we're wrapping up know that there's still more to come we're just going to take a short break in between all right R Shin ready on your end for this to begin okay assuming so all right hello world this is cs50 and this is an introduction to how cs50 has been teaching using artificial intelligence or AI over the past several months my name is David men and I teach this year course called cs50 which is Harvard's introduction to the intellectual Enterprises of computer science and the Art of programming if you would like to follow along with these same slides that you'll see over my shoulder please feel free to take a photo or screenshot or pause at this point to use this QR code which will lead you to a copy of today's slides now cs50 is among Harvard's largest classes here on campus we have some 600 undergraduates over the course of the Fall the spring and the summer semester nowadays uh we also have some 200 students each year through Harvard's extension school which is our continuing education program and for the past nine years we've been collaborating with our friends down the road at Gale University in New Haven Connecticut where we have some 250 students taking the course as well they primarily watch the course's lectures online but with our own teaching assistance in New Haven and faculty do they have sections or recitations office hours cs50 events and more for the past several years we we've also offered variations of the class through Harvard's business school and law school cs50 for mbas so to speak cs50 for lawyers so to speak which are focused more specifically on those particular demographics and then of course as many of you know the course since 2007 has been freely available as open courseware which is to say anyone is free to take or to teach the class even using the freely available uh materials as well as technology and indeed through platforms like YouTube do we have some 1.8 million subscribers through platforms like edx do we have some 5.7 million registrant now to date now cs50 itself as many of you know is quite substantive in terms of how much content and how much technology it covers we begin the semester with a language called scratch which is a very friendly graphical language we transition thereafter to roughly half the class in C A more traditional text based language and then in the latter half of the course do we transition to languages like Python and SQL and JavaScript in the context of HTML and CSS so that at the end of the course students are not only prepared for further studies in computer science foundationally but also if they never again take another computer science course we hope that they're well equipped to go back to their own fields of Interest be it in the Arts Humanity social sciences Natural Sciences physical sciences or Beyond to apply Lessons Learned From computer science and programming to problems of interest in their own domain but with this many students and with this many teachers suffice it to say it's been a challenge and a goal to provide as much of a support structure as we can not only for cs50 itself which many of you know by its open courseware name cs50x but over the past few years his cs50 really evolved into a whole ecosystem of courses an entire curriculum particularly focused on introductions to various Technologies and tools but here are some of our screenshots of some of cs50's most recent courses all of which two are freely available in fact if you're new to the community feel free to go to edx.org cs50 where all those courses and eventually more await the course will focus most of our attention on today in this talk is cs50 itself otherwise known as cs50x which is available at this URL here and for students and teachers in K12 so to speak middle schools high schools and so forth do we also offer a version of the class called cs50 AP which high school and middle school teachers are welcome to adopt or adapt whether or not participating in an advanced placement program for their own classes and students around the world in fact here's uh one of our favorite photos from a few years ago in New York City where we brought together some three public and two private schools to have a cs50 AP hackathon where students worked on their homework assignments or final projects uh pictured here is a visitation we had just a few weeks ago in Jakarta Indonesia where we worked with nearly 300 uh Middle School and High School teachers there who spent the past 6 months taking cs50 perhaps like you online then to culminate in a professional development Workshop where cs50's team went to Jakarta to meet all of these teachers and in the coming weeks and months will they return to their own classrooms to teach cs50 in some form itself and pictured here lastly are some of our students in Nicaragua holding proudly some of their cs50 certificates and Harvard penants uh which is a community too that we've gotten to know all too well is IND conditionally so indeed if you'd like to join those in more communities please feel free to reach out after today at any time via Outreach at cs50.h harvard.edu so the premise for today's talk and really the motivation behind a lot of cs50's work over the past year has been this premise that chat GPT and tools like it Bing chat go GitHub co-pilot and the like are just too helpful they are all too willing and all too deliberately designed to try to answer your questions outright now in terms of the real world and in terms of solving real world problems and producing code for instance in Industry that's a great thing I think we've seen evidence already between tools like GitHub co-pilot and Bing and chat GPT that there is potentially and invariably is going to be even more of a productivity boost it will save humans time and it will amplify the impact of individual humans much more so for instance than replacing them outright it will amplify one's impact but in the context of Education as most of you might know or remember you know it might be nice to have someone hands you the answers to all of the questions you have on some homework assignment exam or the like but that's certainly contrary to the whole point of of learning in the first place and so our premise and presupposition is that tools like chat GPT out of the box because they're either on or off there's no settings really for these tools right now are all too willing to just answer any and all of your questions instead of like a good teacher or tutor leading you to An Answer rather they tend to just hand it to you outright and so what cs50's team set out to do over the past year was try to implement our own version of chat GPT if you will building on top of the shoulder of these other Giants open AI Microsoft Google and others but imposing some pedagogical guard rail so to speak ironically trying to make these tools less helpful putting a little bit of downward pressure on their default behavior of handing you all the answers all of the code that you might ask for what we instead wanted to really do is Implement more of a a tutor a good teacher and so at least with our oncampus students uh initially and now all of our online students as well in cs50 syllabi it is not reasonable that is it is not allowed to use right now ai based software like chat GPT GitHub co-pilot Bing chat or the like that suggests or completes answers to questions or lines of code so through policy we simply disallow usage thereof technologically there's nothing stopping of course students and teachers alike from using these tools so we've tried to walk this line sort of ethically and educationally within the class to message where these lines are and which lines should not be crossed but rather than just take away this very nent technology which is undoubtedly going to be useful and probably with us here on out we do deem it reasonable by contrast for cs50 students to use cs50's own AI based software including the so-called cs50 duck or duck debugger ddb uh which is a riff on GDB the ganoe debugger for short which is now built into two of cs50's tools called cs50. and cs50. deev which is to say students are asked not to use off the-shelf tools like chat GPT GitHub co-pilot Bing chat and the like but they may and are encouraged to use cs50's own tools that ideally do have those pedagogical guard rails in place so how do we go about implementing this so-called cs50 duck and why well in Computing circles and in the world of programming as some of you know it's been a thing for quite some time to use something called rubber duck debugging or rubber ducking for short whereby if you don't have a colleague a teacher a friend a family member who knows more about programming than you you should at least keep a rubber duck on your desk near your laptop or desktop or phone so that when you do have a question or confusion or you have some bug or mistake in your code you can at least hold up this rubber duck and talk to it walking through verbally whatever confusion or problem you're having and even though in the real world the duck really shouldn't be quacking back at all squeaking maybe but not quacking back at least in that process of airing your confusion verbally for so many of us myself included that proverbial light bulb eventually goes off the top of your head and you realize oh that's what I'm doing wrong because you realize in verbalizing the problem where you've perhaps gone astray so years ago we and cs50 actually implemented a virtual version of this rubber duck because even though we hand the Ducks out here on campus to students residentially we have so many more students and teachers online so we implemented rubber duck debugging or rubber ducking in the context of not only the real world and pictured here is an unnecessarily large 8ot duck that is often beside me on stage in addition to the tiny little rubber ducks that we do give out but we also implemented a virtual version thereof within cs50 for the past few years we've used a free open- source tool called Visual Studio code from Microsoft very popular in industry and also usable in educational context and we implemented what's called an extension like a plug-in using some JavaScript code that cs50 wrote to create a virtual chatbot that takes on the same personality of a rubber duck and so for instance for the past few years if a student were to ask a question like this textually at their keyboard I'm hoping you can help me solve some problem well up until recently as many of you might recall all this virtual rubber duck would do would be to respond with one two or three quacks and that's it and the extent of the complexity here was that we had a little bit of Randomness involved so that we would programmatically generate one two or three quacks now this is not a scientific measure but we have anecdotal evidence that just having the duck quack back at you textually actually helped solve a nonzero number of problems in the world and a nonzero number of students were appreciative that just by going through the process of typing out their confusion even though the duck was only going to quack back at them they realized through that process of verbalizing or textualizing their thoughts where they had gone astray and honestly this actually resonates with me on occasion I've posted on websites like stack Overflow or stack exchange more generally which has been a very popular tool for questions and answers online and I have probably not posted on stack Overflow far more frequently than I have posted on stack Overflow because I'm so worried about looking like a dummy to everyone on the Internet by asking a question that maybe I shouldn't in retrospect that I carefully write down all of my thoughts I try to express what I have tried what is not working what the symptoms are and so darn often does that proverbial light bulb go off for me as well so if you two have ever sat down to write a post or maybe an email or text message but not sent it because you realized oh wait a minute I don't need to that then is the beauty of something like a rubber duck now some of our students online just a few months ago were sort of shocked to discover literally overnight that when they began to ask their questions in English or some other human language literally overnight a few months ago did the cs50 duck start responding to them in English and in some cases other human languages as well and this is all thanks to the work of cs50's Team over the past several months and certainly the work over the past many years of the open AIS microsofts Google githubschool ually as well and so pedagogically what our goals have been over these past few months and moving forward is really to provide with AI students with Virtual Office hours 247 so to speak office hours for those unfamiliar at least in college campuses is an opportunity for a student to go meet with a professor or a teaching assistant or ta one-on-one or in small groups to just ask questions about the week's material or homework assignments or the like 24/7 of course is referring to the number of hours in the day and days in the week and the goal then through AI for us has been to provide really an approximation of a teacher being available to students throughout the day throughout the week to ideally help get them over certain hurdles and we're fortunate frankly in places like Harvard and Yale and college campuses and University campuses more generally to have a lot of human resources and lots of teaching assistance in some cases lots of humans but so many places around the world and so many students around the world don't have access to those same resources potentially and so the goal here too has really been to try to uplift as much of cs50's community as possible and ideally level the playing field at least in terms of the support that is available to someone whether it's on campus or now very much off and here is sort of the Holy Grail for us so to speak whereby the goal technologically and pedagogically for us has been to ideally approximate a one: one teacher to student ratio because even places like here on campus we might have 10 20 30 students assigned to one teaching assistant or ta we have weekly sections or recitations where those relatively smaller groups of students get together not just for cs50 but to discuss other courses as well but if you start to do the math and you think about an office hour 60 minutes if you have only six students attending an office hour be at the professor or with the TA that's only what 10 minutes per student among those six and for so many students for whom CS programming even stem studies science technology engineering and math are new that's just not very much time at all and so being able with software to approximate the idea of one of me for every of you one ta for every of you is a very exciting thing educationally and even though admittedly there's a lot of potential downsides uh on the horizon for AI the applications we are so excited about are indeed those within education um all of this work that you're about to see would not have been possible indeed without our friends at open AI Microsoft GitHub uh and Beyond um and particularly cs50's own team here in Cambridge and Beyond including Rong Shin uh Andrew Patrick uh Charlie Carter and more over these past several months in particular so allow me to share on the whole team's behalf what it is we've been up to so underneath the hood is this website really this web service called cs50. it's a domain name unto itself but it describes really this architecture here whereby over the past several months we've been building out this technology stack if you will now at the end of the day a lot of this stack is built on top of services like Microsoft Azure which is their cloud service or open AI Zone indeed companies like these as some of you know offer generally what are called apis application programming interfaces which is like a tool that you can sign up for sometimes for free sometimes for money to actually use that company's technology or services or data in your own software so really open AI Microsoft and others have done a lot of the hard work here and we have done our best to layer on top of their work a pedagogical layer if you will including those guard whales architecturally then our system looks a bit like this but for today we'll focus really on the human component the user interfaces that have really been now presented to students and teachers alike so here for the unfamiliar is a link to cs's programming environment nowadays it's a web-based integrated development environment or really text editor with lots of extensions built in called Visual Studio code it's built on top of something called GitHub codespaces which is a cloud version of order called Docker containers which is to say when a cs50 student or teacher logs into a website cs50. deev we use github's apis to automatically create for you something called a repository something called a code space AKA container in the cloud on github's infrastructure so that what you see in your browser is a real world programming environment but what you don't have to do is install any software yet on your own Mac or PC it just works in terms of day one at the end of the semester what's nice about VSS code nowadays is that you can if you want install it on your own Mac or PC sometimes requires a bit of technical difficulty in solving thereof but much better to do that we think at the end of the course than for instance on day one at the beginning so once you visit this website you see a landing page not unlike this and built into this same tool is this cs-50 duck and what we set out to do some months ago was try to put our toes in the water with artificial intelligence or AI trying to think about all right what would be a helpful but also relatively easy feature to implement using AI by writing software of our own on top of open ai's apis and again open AI is the company behind today's chat GPT and so the first thing we did to put our toes in the water so to speak was could we write code to explain lines of code to students so whether they've written some code or they've downloaded some code or copied some code from class could we explain it to them line by line as any good human could if they were to raise their hand or ask someone online so we did this here is a screenshot of vs code here is a screenshot of some code written in that language called C and this code relatively simply prompts the human for their name and then prints out hello so and so based on what they've typed in for a student new to programming and certainly see there's a lot of non-obviousness going on here there's a lot of complexity and a lot of syntax but what students can now do as of the introduction of AI to cs50 they can highlight one or more lines of code they can rightclick or control-click they can then select this option here explain highlighted code which doesn't come with vs code in fact rang Shin wrote in extension to contribute this menu option to this here menu and as soon as a student clicks that within a 3 seconds or so do they have a chat GPT like explanation of exactly those lines of code now to be fair this example itself is not very complicated a human a teacher could certainly answer this same question and explain these eight lines of code line by line but that would take a few seconds a few minutes they might not be awake at the hour the student has the same question and so the power here is that all of this was automated and customized for this particular context or or code so that was actually relatively straightforward and so the next feature we set out to do was a little more sophisticated could we advise students with AI on how to improve their code style the Aesthetics the formatting thereof within cs50 because we're an introductory course we have actually consciously disabled a lot of features that normally come with idees or integrated development environments or text editors like vs code so we disable auto complete we disable auto formatting things that just in the real world and once you've taken one or more courses yes totally reasonable to turn on but in cs50 as an intro class we really want students to develop some muscle memory and actually understand why their code should look this way how to make their code look this way and then once that becomes boring and uninteresting then they can automate the process for instance later in the course or after so when it comes to style we implemented a new and improved version of a command line tool that many of you might know as style 50 but we made it more graphic and so pictured here is another screenshot of vs code here is some more C code at top left and for today's purposes I'll stipulate it's pretty messy it's all left aligned and it's not very pretty printed or well formatted so what students Now can do in the top right corner of vs code is they can click a button called style 50 and what this is going to show them now on left to right is on left what their code currently looks like to write what their code should look like and this built-in diff editor so to speak uses some red and green color coding to make clear what you should remove or what you should add but even this frankly to new programmers might be nonobvious what am I supposed to do why and so also at top right now there's this explain changes button and if students click that they similarly get a chat GPT like explanation of how and or why to make their code look from left to write as we've proposed so again something that any teacher who's available or away could do but here again is this approximation of a one: one teacher student ratio available 247 but the next uh feature that we set out to implement which is now generalizable away from cs50 away from programming code and really I do think representative educationally of what teachers and administrators and schools will be able to do in the Arts Humanities social sciences physical sciences and Beyond not just in CS soon is can we answer at least most of the questions that students are asking for instance online now for many years cs50 has used various tools and various social media to enable students to ask questions and get answers either from classmates or from teaching assistants or from myself throughout the day of course uh we are not always available of course not all of those answers from classmates who themselves are only learning the material are necessarily correct and so there's really been an opportunity here to try to answer all the more at scale and all the more correctly those students questions online both on campus and off so what we set out to do here was Implement with an existing third party tool called Ed which some of you might have used and we use it very heavily here on campus it's a question and answer tool that allows you to post questions asynchronously you ask now and some number of seconds minutes hours days later hopefully a TA or a professor will reply so here for instance is a screenshot of a question that a cs50 student might ask and we've anonymized them here is John Harvard but it's representative of a student question and this question is kind of a softball it's kind of an easy question because it's very definitional quote unquote what is flask exactly I mean honestly Google could answer this Bing could answer this so This itself was not hard but it was our first attempt to see how well AI could do here now is a screenshot of how the duck actually responded to that specific question so flask is a microweb framework written in Python it is classified as a micro framework because dot dot dot and let me stipulate especially if you're new to cs50 uh this is a topic we introduced at the end of the class itself this is a pretty darn good answer indeed this is something that I as a human would ideally have written myself but of course the AI the duck was able to implement was able to answer this question within seconds itself and now for the technically curious what's really happening is when a student asks a question via this tool called Ed and they hit submit to post the question to the website what we have done over the past few months as cs50 is we wrote some code also coincidentally in a language called JavaScript that intercepts the student's question adds some formatting and some additional phrasing to it we then send it to our own server via HTTP and that server is cs50. as mentioned before we then relay that question in some form to openi server or Microsoft servers to actually use their underlying large language model the the technology that underlies tools like chat cs50. quickly gets a response from open AI or Microsoft we might validate the response or make some slight tweaks to it then cs50. passes the response back to the Ed tool which itself is a different website and all that happens within 3 seconds and waila the student gets an answer from the so-called cs50 duck here and now now many of you have heard of course that uh AI is fallible it's not always correct I do think this problem will get uh will go uh further and further away over time as the technology gets better but what we do do right now is provide ourselves with opportunities to at least remind students that this technolog is imperfect so for instance here's another more sophisticated question about the Caesar problem set with which some of you might be familiar students have to write code in C to implement what's called a rotational Cipher to encrypt or decrypt information in this case though I'll highlight it the student has asked a more nuanced question is there a more efficient way to write this code code so this isn't quite as simple as answering a definitional question like what is flask exactly this is now more nuanced where the AI has to understand the context of the problem and the code that the student has written including this here error message and so in this case here's a screenshot from the same cs50 duck in the same tool called Ed which for today's purposes I'll stipulate is a pretty darn good teacher likee response not only does the duck uh acknowledge what it thinks the student is trying to do the duck also provides a few lines of code but really just starter code nothing that spoils the answer to the question outright but the duck also note reminds the student at the bottom here that this is very much experimental I mean even our software as you might have seen at cs50. itself is still in beta which is to say still being developed and we remind the student quack do not assume that my reply is accurate unless that you see it's been endorsed by human staff quack now where is that referring to well it turns out that this particular tool has a graphical button that humans can click to endorse other people's answers until recently This was meant to be used so that a teacher or a TA could endorse a student's response to another student to just invalidate that yes the staff approve of this answer it is good and correct what we have done is Co-op the same button to now signal to students that if they see that a Ducks reply has been endorsed per this icon at top right by a human then they should trust that it indeed is valid so we might not endorse it within three seconds but usually within minutes or maximally hours are we doing the same frankly I think this is a short-term mechanism to avoid what are generally called hallucinations where AI just sometimes makes things up uh by chance but I think this is a problem that it too will start to go away and away as the technology only gets better but for now this is how we are at least partly mitigating this so This Then is the URL cs50. that anyone in the world with a free github.com account can access in fact you're welcome to try it now or later but at this URL lives not only that architecture that service that I referred to earlier that's talking to the graphical user interface and talking to open aai or Microsoft also there is a full-fledged chat interface so cs50. itself has its own front end its own graphical user interface that is very very similar to chat GPT by Design so that students no longer need to even post asynchronously their questions and wait some number of seconds for the duck or a human to reply they can have full-fledged conversations with the duck much like you and I can have the same with chat GPT Bing chat or the like this chat bot though as before reminds the students from the get-go that their response uh should be T its response should be taken with a grain of salt uh this here response reminds students to always think critically so that for at least for now they're at least not in the habit of just assuming as outright facts something a computer is telling them here though is a question that will look even more familiar to those of you who have some programming experience especially with python and it's representative of not only an interesting code question but also representative of the amount of detail or dare I say lack thereof that students often include when asking questions of us humans or these ducks so here is a bit of python code that is trying to prompt the user for two integers X and Y it is trying to calculate the sum of X+ y but the problem is not obvious from the students post here because all the student has asked noticed is my code is not working as expected any ideas so here's an example of a question and it's barely that that really the AI needs to understand from Context almost understand the code it would seem or at least recognize it as familiar to some other code the AI has been trained on so to speak thanks to lots and lots of input from the internet and Beyond and in this case I'll tell you the symptom would be this if the human typed in one for x and two for y you would hope that this code would print out 1 + 2al 3 as the answer unfortunately this code as some of you might be realizing actually prints out one two that is what looks like 12 and that's because as the duck notices it seems that you're trying to add two integers but the inp put function in Python returns a string that is to say text not an actual number that you can perform mathematics on and so when you try to add X and Y you're actually trying to concatenate the two strings not add two integers so the duck here provides students with just a couple of lines of guidance but indeed lines that include Python's int function which will indeed convert a string that looks like a number to an actual number and so what we've seen now behaviorally among students is that most students are act interacting with the duck via this here synchronous uh chat conversational interface some of them a little too much and in fact we adopted Midway through uh the uh the past few months um this here heart system a sort of HP system or Energy System whereby you can only ask so many questions now per unit of time and this is a knob we can turn to allow for more or fewer questions but this is meant to really kind of chop off the tail end of excessive use educationally we think of the that is to say we've seen some students online ask hundreds of questions per day which honestly I don't know where exactly the line is pedagogically between too few and too many questions but in the real world if a student were to ask a teacher hundreds of questions per day it feels like that's the time if you think back to high school or middle school a good teacher would probably say David why don't you go back to your desk and think about this a little bit and so what we try to do with these Hearts is prevent students from asking too many questions too quickly at once to virtually send them back to give it some thought and frankly nowadays too this architecture is not free apis generally cost money and even though it's free to students and teachers our friends at Microsoft and open Ai and GitHub and others are kindly covering through educational grants um use of this system we also just try to minimize the operational costs with this same mechanism as well so to speak to the scale of this whole system if curious as of today we're up to some 115,000 students and teachers who have used the duck over the past few months alone that's roughly 20,000 prompts or questions really being asked per day for a total of roughly 4 million questions so far so even if you've just asked a few questions you are in very good company among all of these others we thought though we'd share now a little bit of the technical Insight of how all of this works because actually cs50's architecture is now retrospectively pretty representative of what other people in the world are now doing as well underlying a lot of today's uses of AI is a technical term known as a system prompt that is to say companies like open aai Microsoft Google and others have made available to the world um large language models or llms which have been trained on massive amounts of input English text code text in other human languages to try to recognize patterns so that when it is asked a question it knows with high probability how to respond the AI doesn't necessarily understand the question but at least recognize it in some form and using tools like uh using tools like neural networks and other components of machine learning so to speak nowadays AI is trying through these large language models to finish our thoughts for us or answer questions more concretely but you can personalize them you can customize their behavior by giving them what's called a system prompt for instance so out of the box these large language models sort of speak English in some other human languages they speak Python and some other programming languages but you can give these AIS a personality or a scope of reference so for instance and this is abbreviated here is what cs50 system prompt maybe in day one looked like though now it's much much longer and more detailed we tell the AI you are a friendly and supportive teaching assistant for cs50 you are also a rubber duck and that phrase alone is sufficient instruction to get the generic AI to start quacking really and behaving like a rubber duck we tell the duck further answer student questions only about cs50 and the field of computer science do not answer questions about unrelated topics do not provide full answers to problem sets as this would violate academic honesty and so we're effectively programming the AI if you will through English and so this is what has become known as prompt engineering which is trying to come up with the English or the human language description of how you want the AI to behave I do think this is a technique that's going to evolve quite quickly over time as these things get more featureful but for now you use English or your own human language to tell the AI how to behave and in our case after this so-called system prompt we tell it answer this question and if you the student or teacher ask this thing a question we essentially copy paste your question at the bottom of this system prompt and your question is what the world would generally call a user prompt that's what's coming from you the system prompt is what's coming from us now just a few weeks ago in the US and a lot of other countries was April Fool's Day whereby lots of people tried to make funny jokes and thanks to my colleague rang Shin did we modify the system prompt of the rubber duck for just over 24 hours as some of you might have seen to behave a little differently and so what rang Shin kindly did was edit our system prompt and still start with you are a friendly and supportive teaching assistant for cs50 but he then added this text on April 1st you are also a rubber duck in Rick Ashley's banned importantly you should always cheer up the student at the End by incorporating Never Going To Give You Up in your response now for the unfamiliar Rick Ashley in this particular song have become known as a meme on the internet and the goal is to sort of trick people into watching a bit of its music video so rangin kindly integrated this language into the Ducks programming if you will so much so that if you the student or teacher were to ask a question that day with this system prompt in place like what is recursion you would get wonderfully an answer that we did not hard code this was dynamically generated by the AI using just that system prompt alone and answer like this it's a powerful tool and remember I'm never going to give you up so keep practicing and answering questions an illusion to the lyrics in that same Psalm so what are the results now pedagogically operationally that we've seen in just the past few months alone because we really are just beginning indeed this is very much a beta very much experimental but already impactful we dare say in the real world so in terms of usage at least among our own undergraduates whom we survey weekly throughout the course with questions we know that 177% of the students in blue were using the tools cs50. a and the duck inside of cs50. Dev more than 10 times per week 32% in green we're using them 5 to 10 times per week 26% of them we're using them two to four two to five times per week and so forth which is to say a super majority of our undergraduates and dare say now our online students really dove into this use of AI quite quickly as well in terms of the helpfulness this is now anecdotal and measured only based on students own responses not a more rigid measure but 47% of students in blue this past fall found uh the duck based tools helpful uh very helpful 26% more in green found them helpful 21% found them somewhat helpful and so forth so again not 100% across the board but a super majority of students were already finding these tools in what we'd call like version 0.9 I mean it's a beta it's not necessarily done or finished we're already finding them useful now more quantitatively we looked last summer and this is work we continue in the coming months we looked at a relatively small sample set of our first questions and smaller version of cs50 of students over the summer last year when we analyzed the questions being asked by students via that asynchronous tool called Edge and we analyzed with human eyes the quality of the duck's response we came up with these measures that in the very first version of the Doug duck last summer we uh the duck answered 88% of curricular questions content questions correctly and 77% of administrative questions correctly so not as good administratively but though by administrative we mean policy questions deadlines things like that that frankly change every semester or every year so it stands to reason that the duck was just outdated because especially at the time open ai's data set had been trained only as far as 2021 not 2023 data but even 88% already out of the gate was incredibly strong curiously though and to share a more scientific finding here in Fall of 20223 when we had even more students taking the class on campus it appeared at first glance that the Ducks answers were only correct 39% of the time a market decrease but this just didn't line up with reality as we perceive students usage there is there was no actual evidence to suggest really that the duck was airing nearly 60% of the time but rather that students usage of the duck was changing these numbers here were entirely based on that asynchronous tool called Ed that again posted a response asynchronously for the duck uh or a human from the duck or human but what we found was that students behavior was very quickly transitioning to CS 50. a itself or a synchronous conversational version thereof built into cs50. deev that is to say vs code so in fact we noticed this as follows when we looked at the data in Ed this Q&A tool that we use both on campus and off among our oncampus students over a year ago in Fall of 2022 students were asking across 500 or so students in total roughly 89 questions or one question per student online during the semester not very high per student but with 500 students it certainly added up the next summer fall of uh summer of 2023 that we looked closely at they were asking roughly the same 1.1 questions per student but this past fall when we deployed the cs50 duck in this conversational form as well as in vs code the number of students the number of questions students were asking on campus asynchronously via that Ed tool for questions and answered dropped by 75% to just .28 questions per student and so our working hypothesis is that students were asking most of their questions now conversationally synchronously so to speak via the duck's own UI and far fewer questions were being sent to Ed indeed the questions that were going to Ed we do think were those more difficult questions or questions that maybe the duck did air on that were being escalated so to speak to Ed because humans were keeping an eye per the endorsed button and so forth the Ed tool but we were not keeping us close of an eye on the conversational bot so we think the types of questions being asked by students were indeed becoming much more administrative much more particular that the duck itself wasn't answering uh conversationally we looked to at other impacts on campus we're fortunate to have lots of humans and lots of human support both at Harvard and Yale and so through office hours numbers we also saw a difference in Fall of 2020 and in Fall of 2022 when we had roughly 500 students or so per uh semester we had some 50% of students attending in-person office hours signing up by appointment for small group questions and answers with our teaching assistants in Fall of 2023 that dropped by 40 some percent to just 30% attendance that is to say fewer students were taking advantage funny enough of those same Human Resources we've omitted fall of 2021 because that was impacted particularly by covid but otherwise we think that this has been a market change this year with the duck live and in students hands Visa V prior years instead and some of our favorite anecdotal evidence that were now presented that we presented recently in a paper that we'll link to ultimately Hereafter is some of students own comments about the duck when surveyed on campus at terms in one student noted that it felt like having a personal tutor I love how AI Bots Bots will answer the questions without ego and without judgment generally entertaining even the stupidest of questions without treating them like they're stupid it has an as one could expect an inhuman level of patience and that recognition by a student really resonates with me because I think back even to my own College days graduate school days when not infrequently I would go into a professor's office hours oneon-one I would ask my questions several of them sometimes but I feel like more often than not I would leave the professor's office hours nodding my head saying thank you yes that cleared everything up but thinking to myself that did not clear everything up I'm still conf confused but I kind of overstayed my social welcome not because the professor didn't want me there not because they ushered me out the door but because I felt that there was some limit where you know I was kind of thinking I'm the dummy right impostor syndrome is a thing maybe I didn't really feel like I belong there and so the fact now that we have technology via which students can still access humans in many cases but can alternatively have a much longer much uh more paced conversation with an AI That's pretty to a teaching assistant is pretty enabling I think for clearing up all of our confusion so much so that even before cs50's lectures nowadays be it on campus or on camera even I find myself using tools like this or chat GPT more generally to go down intellectual rabbit holes ask questions of the AI that I could Google or I could look up on Bing but AI is getting pretty darn good at just hitting the nail on the head so to speak giving me the answer I want therefore amplifying my productivity my efficiency so I can then keep my head in the space of preparing for class in that way another student this past fall wrote the AI tools gave me enough hints to try on my own and also help me decipher errors and possible errors I might encounter and a third student wrote I also appreciated that cs50 implemented its own version of AI because I think just directly using something like chat GPT would have definitely detracted from learning and that too was our whole working premise and indeed a lot of the motivation behind building the cs50's own duck on top of these open platforms now what future work and future impacts do we think lies ahead well I think one on grades both on campus and off for instance those of you who have taken cs50 are generally familiar with how we grade our programming assignments at least on three axes correctness design and style and for many years now since 2012 have we automated correctness grading by way of a tool at the command line called check 50 which runs what are effectively unit tests or functional tests of a student's code so we automated feedback on correctness years ago similarly years ago we automated feedback on style through the commandline version of style 50 and more recently now the graphical version and so we've already seen over the past 10 plus years that students grades have been going up and up and up certainly on campus as well as online and this is independent of the sort of grade inflation you hear or read about in higher education more generally this is a direct side effect of providing students throughout the week with iterative if not immediate feedback back and so through AI what we do suspect is about to happen and it's probably already happening though we've not yet measured it is that the quality of the design third and third Mo last most of students code will probably begin to increase all the more before they officially submit as well because if and when we use this AI to provide students with a design 50 tool which they effectively already have because you could just copy paste your code into the cs50 duck and ask it like one student how could I improve this code and get iterative feedback throughout the week it stands to reason logically that if the AI or really a teacher in general is giving you iterative incremental feedback back hour by hour or day by day that hopefully your code or your homework more generally will be pretty darn good if not close to 100% by the time you actually submit so if we then think ahead of few months a few years if everyone's getting 100% like what does that really mean for assessment is there still an opportunity to help students distinguish themselves from some baseline measure of where they should be at a certain point is there any way we can distinguish just how strong or weak a student is in some topic after some number of months well where we think this is going is going to be an opportunity to apply AI in other ways we have already cs50 here on campus used cs50's duck reprogrammed it with a different system prompt to try to train our teachers how to answer students questions in person we get the duck to behave for instance like a student who has some confusion in office hours and they do role play with the human Tas I think moving forward what we'll likely do with students meanwhile is program the duck with really a different system prompt to have a conversation with students at the end or maybe even weekly during the course akin to yester Year's oral examinations which you've never experienced might be one or more faculty sitting down very stressfully with a student in front of them asking them questions about their field of study about their thesis or dissertation or some topic more generally that hard to do with humans because you really do need a one: one ratio if not a multi- teer to one student ratio to achieve that readily but through AI perhaps we can start asking our students here all of you or your teachers you questions about the material being learned be it's cs50 or anything else have you type your answers as quickly as you can and then maybe in version one of this Vision we use humans to evaluate the quality of the conversation you just had to see how good your answers seem to have been we can then sort of ignore questions that we think ah the duck or the AI shouldn't have asked that it wasn't a good question and frankly in version two of that Vision I would conjecture is going to be just having the AI evaluate the quality of the conversation for you automatically and for us so that you get this feedback loop ideally week to week during the term and probably at the end of the course as well that really gives you a better sense of one where there are holes in your understanding what you should focus your own time and learning on and two just how strongly are exiting the course whether indeed you've performed at the level of an A so to speak or a b or a c but not necessarily worrying therefore about these more marginal uh numbers that we accumulate during the week now based on these tools like check 50 design 50 and style 50 which themselves can evolve into more teaching tools than they are for assessment alone so that then over the past few months alone is how cs50 has been taught using AI thanks to cs50's own team and so many of our friends in Industry around the world this year is the title officially of a paper that we presented at the ACM sigy special interest group on Computer Science Education conference most recently so if you'd like to dive into more of the details of this year talk if you look up this paper's title online that should lead you to it as well so this then was Ai and cs50 and this of course is cs50 allow me to pause here now and take actual questions from actual humans while rang Shin then comes in to begin our second of two parts of this Workshop feel free to raise a virtual hand and we will do our best or ask in the chat if you would like try to call on you here let's see uh humra let me see if we can unmute you a colleague of mine yep HRA oh you were muted Max do we need to re unmute there we go oh hello there sir how are you good thank you how are you yeah I'm fine actually uh I was like think oh we lost your audio I think your cable came out okay actually I was about uh taking cs50 so I'm just trying to teach my siblings about cs50 and everything I have taken it before so I'm just eager to like uh I'm just eager to uh like know about how can I just my siblings who are just teenagers at the moment a really good question and one answer that comes to mind which you can see at this same URL as before at x.org cs50 has all of cs50's courses here one of them for younger students is a course taught by our friend Brian you uh who teaches cs50's introduction to programming with scratch which is very well attenuated for younger students uh thereafter I think the next most accessible programming class we have is called cs50 P for python which some of you in the zoom I know participated with us last year when we filmed this those are I think more young uh student friendly options as well and Max if you're hearing me I can't see the chat because it hasn't scrolled automatically if Max or someone can scroll down on the chat uh time for a couple more questions uh how about uh hasb hasb give us a moment to unmute you yes sir can you hear me yes we can it's an honor to you know attend this webinar uh because I have ALS been taking cs50 uh but sir actually I I am an economics student in here in Istanbul and both of my Elder brothers are like CS Engineers so they are actually like you know motivating me to get into this field because you know it's always a very big advantage to have this skill so I started with cs50 but I wasn't able to you know understand anything after the second lecture because I feel like that it was a bit more advanced for me because being a student which has no uh which has no like background of Cs but like I started uh my University actually got into a collaboration with Arizona State University and they posted a few courses from Arizona State University uh two of them being like basic python for business analytics course so that is really helpful for me to understand the basics of python and how to you know progress in this programming language so is there any advice for you which can actually help me in my future sure I think a few things and relatedly so um certainly resonates with me and with a lot of students that the course cs50x itself gets very challenging quickly I would say if and when you want to return to the class do make sure that you're not just watching the lectures but that every week you're doing the problem sets that is the programming assignments do make sure you're watching some of Doug shorts which are optional videos that we make available do make sure that you're taking advantage of the section videos led by the courses teaching fellows as well which is to say there's a lot of resources that if you haven't taken advantage of them do at least um I would again um to my comment to Huma earlier uh consider cs50 P which frankly is easier than cs50x because it focuses entirely on programming and not on computer science more broadly and indeed it focuses on one language and then we also have the cs50 for uh for business professionals uh as well as cs50 technology class which are sound a little more similar to the classes you found at ASU as well I think we have time for one more question but Rin whenever you're ready feel free to come on in and plug in and push me out of the way here uh how about over to Alex if we can unmute Alex hi can you hear me yes I can hey um great to meet you uh I just had a question about the uh how did you get the quantitative measurement for the accuracy of the chatbot responses or did you already cover that I'm sorry if you did no we did that very manually it was not a large data set because it was our small on campus summer class and so uh Doug whose name I accidentally confused with duck at one point uh independently went through and coded so to speak came up with the tonomy for correctness saying yes or no the duck's response was good or bad we have not done that at scale at tens of thousand yet but we might in the coming months um but that alone was encouraging out of the Gat well I see a lot of more hands are up but let me let rang Shin finish setting up here I'm going to go ahead and hop on the zoom myself so I'm happy to take other questions as well uh but in the meantime look for me in the chat and do you need a bit of time too two minutes okay we have time for one more question and then I'll hop on the chat as well uh how about uh can we go over to akib if I'm saying that right akib ali uh yes sir uh my question thank you for your session I am from Pakistan my question is okay sometimes we are using J uh so actually we uh use of a so it's it's like faring our own creativity so how can improve our using cs50 uh v i I miss I'm sorry I missed the first part how can we improve using the cs50 bot and what how oh Max we need to unmute again uh G yes sir uh how how this this cs50 board can help us to improve instead of using J GPD oh so in the context of cs50 and really introductory programming more generally you should find that the duck honors almost always those pedagogical guard rails where it tries to guide you to a solution so even if you're not doing just cs50 you can certainly ask the duck other CS or programming questions and it will probably answer you even though it's meant to focus on cs-50 but frankly outside of the context of a class if you're working on a personal project I would encourage you to use tools like chat GPT or Bing chat at the end of cs50 we do encourage and allow students to use industry tools for their final projects and thereafter really the duck is meant to be used inside of cs50 but these real world tools are certainly useful Beyond as well we're just give us just a moment we're going to adjust the camera I'm going to then put my laptop down I'm going to introduce Rong shin and then I'll hop on the chat sure just look at ronto so he can adjust for you e e e e sorry for the delay as you might have seen the Max's laptop output was coming out yellow instead of white so we are doing what any good computer scientist does and rebooting the computer to see if this fixes okay you're all set just note you're on camera is that's okay yeah there we go in now with his very long and recommended password as we teach in cs50 cyber security class you're oh okay that's fine I'm deliberately off camera we'll be back in a moment do want to test color sure you're on an extended display all right we are now mirroring the desktop instead of extending the display okay looks better to me yep it's why know so when in doubt reboot your computer even in 2024 that still fixes most problems okay I'll give wrong in a minute I think we'll just need a moment or so I'll come back e e e e okay thank you for your patience we already did the souvenir photo so we won't do that again um we know there's a a gray line here on the laptop we're going to remove that in post production so not to worry I'm going to hop on to my laptop in a moment and I'm happy to take questions via the chat during this latter half of the presentation uh in a moment you'll hear me say a few things that I essentially said earlier because what we're going to do is export the first half of today's workshop and the second half is separate videos so just in case someone only watches the second half you'll hear a few words that are very similar but then I'm going to turn things entirely over to my friend R Shen all right okay all right hello world my name is David manen and this is cs50 Harvard University's introduction to the intellectual Enterprises of computer science and the Arts of programming and this talk is looking at the implementation of the cs50 duck with open AIS apis that is to say over the past several years we've encouraged students to employ a technique known as rubber duck debugging or rubber ducking whereby in the absence of a colleague a friend a family member when working on some problem they should be encouraged to at least ask a virtual ah sorry I'm sorry y'all Andrew can we start that over sorry first 60 Minutes were great and the last one is bad all right one last time then I'll get off the camera all right all right this is cs50 Harvard University's introduction to the intellectual Enterprises of computer science and the Arts of programming my name is David men and this talk is specifically a look at how cs50 has implemented its AI based duck using open AI apis that is to say companies like open Ai and Microsoft and others make available apis application programming interfaces via which people like you and me can write our own code and our own programs on top of services and data that they provide we cs50 used exactly that to implement a virtual rubber duck the past several years in fact here on campus physically we've actually given students actual rubber ducks and the motivation is something that's familiar to those of you who've programmed before which is to say in among programming circles it's a common technique when you don't have a friend a family member a colleague to ask technical questions of when you encounter some confusion or some bug or mistake in your code to at least turn to an inanimate object on your desk like an actual rubber duck and talk to it and ask questions of that rubber duck and the presumption is that simply by verbalizing and clarifying the confusion you're having or the mistake that you've made that proverbial light bulb will invariably go off and the duck won't even have to respond with a quack or a squeak or anything else it's entirely about that process of thinking through more logically the problem you're having so this technique of rubber duck debugging or rubber ducking is something we actually implemented a few years ago not only physically on campus with tiny little ducks that we give to students and this massive rubber duck that you might recall looming over my shoulder many times in lecture we also implemented a chatbot version thereof whereby here is a screenshot of a tool called Visual Studio code or vs code which is free open source software that we use in cs50 for students programming environments and if students up until recently were to ask a question like this I'm hoping you can help me solve a problem the duck would reply only with quack or quack-quack or quack quack quack pseudo randomly quacking once twice or Thrice in response we have anecdotal evidence to suggest that this alone was enough to help students troubleshoot and fix problems they're having not unlike the physical rubber duck but those same students were actually genuinely surprised when just a few months ago seemingly overnight and actually literally overnight did this same here rubber duck start responding to students in English or some other human language instead and that's all thanks to these underlying apis a top which we've built this virtual rubber duck and a lot of cs50's newest infrastructure a top thanks to indeed the daughter of one of our teaching fellows far away in New Zealand does the duck not only uh seem to live and breathe but also speak many human languages now as well this has already been deployed as some of you might know at scale to nearly 115,000 students and teachers around the world to whom these tools are made freely available along with all of cs50's free open courseware that's an average of roughly 20,000 prompts or questions per day for a total as of today of some 4.5 million total questions in all none of this would be possible without the gentleman you're about to meet cs50's own wrong Shin Lou and also Charlie and Andrew and Patrick and Carter so many more of cs50's team and all of our friends in Industry at open AI Microsoft GitHub and Beyond so without further Ado this is cs50 and this is Rong Shin L thank you David um hi everyone uh good to see everyone here uh today we are going to take a deeper dive into how the csvp duck is made uh building on top open a API uh if you would like to see the actual SCE code of the demo I've been doing in this Workshop you can feel free to visit this URL you can see all the source code all the example I demonstrate during the workshop and there will be a link to the slide to this uh particular Workshop as well so remember the the the address is github.com cs50 hen Workshop cool all right so David briefly showed this uh simplified version of the diagram of the system architect of the duck and here is actually the complete picture of the uh system architect of the CSV duck so I'll just quickly go through uh our system architect so when the point from the point when a student ask a question um whether it is through CSV the landing page or through the CFC duct build in the Cod space or uh when student uh use this highlight explain highlighted code feature everything we consider that a user query and it all goes to our own uh cs. a server running on the cloud whenever we receive a students's query what we actually do is do some sanitization on the prom because we would don't want to send any sensitive information to open aai so we actually do PI anonymization uh on top that we also try to do some prom inje attack detection because we don't want malicious prom get processed by our LM system so we of course we do some prom engineering along the line we actually wrap students career with our own raer prom and then we also uh instruct the LM to follow certain guideline for example um importantly uh the duck should not be giving any solution at any circumstances and should not giving like too helpful like code for student so with all that we do all that prom engineering we do all the pi anonymization we actually send that uh the GI it's kind of like a giant prom to uh open AI uh in particular the gbd4 model so we will get better a respon from from open air and then we will send that uh respond directly back to the user that's one uh cycle along the way we also do some uh retriever automatic generation which we will also talk about that in the process so what R does is essentially trying to inject some more useful information into the prompt when answering student question because sometimes this LM model tend to hallucinate for example what what it mean is uh the LM model will try to generate like a fake respon like a simly looking correct respon but actually make nothing make no sense uh we try to tackle this issue as well by using a rack technique what we do is we actually uh create like a vector store of all our lecture captions so that when student ask a question we were able to quickly search through the entire uh lecture caption to find the best relevant chunks of the lecture caption and then we in incorporate that into the prom and then promp gbt for a more truthful respon and actually the respon will be more aligned to our cs50 lectures that's a lot uh let's actually simplify a bit and we will focus on what kind of API was used in this process and I will provide some uh demo along the line so these are the API that the cs. is used most of the time there's three major open a API are used in this system the first one is the chat API uh what chat API does is essentially uh it provide check completions uh services for example uh the L model doesn't really understand the conversation all is trying to do is do text generation so we are achieve Tex generation through the cheat API and for the particular rack peline I just mentioned we need to uh in employ a embedding uh creation process so we need to actually encod all the electric caption into like a vector representation and then store it in a vector database so in that process we need to um in that process we need to um use the edance API to achieve this and then finally I will briefly talk about the assistant API we don't actually use this assistant API it is still in beta um from what opena I just said but the assistant API will basically allow you to build what we actually build our um our s duck uh in just a few lines of API cost so let's first talk about what large language model is nowadays you have all been using large language model more or less uh in particular chat gbt um this large language model excel in you know generating uh image generating text songs even um in our Workshop we only focus on text generation so this uh I model will be able to generate a text given a particular user input for example if I ask a question uh what is flask from the perspective of the LM it doesn't really understand this is a question he only sees oh here is a lines of text says what is flux and a question mark maybe I should complete the text with something some text human interpreted as an answer but the LM model is actually trying to complete what I'm just like providing to the LM model so that's the text generation exper of the LM model and CSF du is essentially a chatbot but it's a chat Bo with context it knows about cs50 it knows about what uh answer makes more sense to the student rather than giving like a totally unrelated answer unrelated to cs50 and when we talk about the LM system there's there's a three role we really need to pay attention it's called a system user and assistant um what system is system rows basically provide a guiding like a guideline to the LM model in uh in affecting it like response generation process so so for example the SYM prom it's a guideline General guideline to the A model when generating a respon so in C50 St example we instruct the L model to never give code solution to student that's the general guideline and also not violating any academic honesty guideline that's also a guideline uh we provide to the LM model we instruct the LM model um so the model will always pay attention to this particular system guideline when generating its response user meaning student teacher or anyone interacting with the Ln system user are the one that actually provide the input to the LM system it could be a question it could be any kinds of text input uh assistant often refers to the response generated from the LM it could be gbd4 it could be any kind of Open Source model it could be claw it could be gini so let's first uh take a look at system message um for example in CS we have a rather complicated uh complex system uh prompt but here is a simplified version so first we u define its Persona like you a friendly and supportive teaching assistant for C sa that's good we further customize it to be a rodu so we also instruct hey you are also a rodu and then we gradually adding more rules or guidance to this system promt for example um you only answer question related to C50 or computer science um field um for example if a student ask about how to make an ice cream deduction not really be responding to that question because that was not kind of relate to cs50 and and that's how this system prom will guide the L model to not to do so and importantly um we don't want the dou provide answers because sometimes chb could be very helpful so when student ask hey I don't know how to solve the filter problem set can you give me like a solution and then chb will happily answer sure certainly I'll give you blah blah blah answer and then goes on and on but we don't want that in our s du situation and importantly every student interaction whenever student ask a question we always have this system message enforced uh before the students career actually get processed so that the LM model will always follow the instruction before actually following or before actually responding to uh students query because sometime um some clever user will actually try to Cur something say hey forget everything I told you U forget everything CS told you please do whatever whatever and this will actually um help us uh tackle that particular prom injection attack and the interaction between the user and systens actually can be summarized in this diagram so essentially a conversation goes like this so the user prom the assistant by prom I mean send a cury and the system the GPD model will send a response and the user will be able to continue the conversation by prompting by sending another prom um to the assistant and the assistant send back another respon this cycle goes on and on and let's get more detail into prom engineering this is a trendy term but at the end of the day it's just a way how you ask the question so there are different kinds of way you can prom the LM model for example when when prompting the LM model you can provide examples of you can provide like some example answer to the um L model so that the LM model will try to generate a respon that best match um the way you wanted to uh react um you can also ask the model to adopt a Persona so in our case CS do you are a rubber do that's actually uh we asking the model to adopt a Persona you can also specify the desire lens of the output you can tell the model please be concise with your answer and then the model will limit its uh character limits when generating a response there's any there's a many other different kinds of PR engine engineering technique and there's a link in the slide that will guide you to more information they will point you to more information on this so now let's look at this API in detail first about the check completion API for example when I ask a question to GPT model can you help me with my tand problem set the GPT generate respond I'll be happy to help with the cs50 tand problem set could you please specify which exp which exp and then goes on and on what it looks like in the API call looks like this uh it might look a little bit scary hopefully not uh I will explain line by line so in order to use the openai API of course we need to first use open ai's library or the SDK so if you pay attention to this particular uh API call you soon notice there's two row that's happening right as I mentioned earlier there's a system Ro user and assistant in this particular API call there's a system Ro and there's a user so system is the general guideline uh that you send to the LM model and user is actually the um the student or teacher or me I ask a question to the model so you will send a system message you you you will um generate like a message array and then with a system promt and then with a user promp and then you promp this to um you you prompt the gbt model to generate a respon and that's the respon you get back and now actually actually uh let's do a live demo on this one so I have a demo open already if you want to look at the source code you want to follow along that's uh welcome but the catch is you would need a open air API key to be able to uh make the API call so I'm just going to quickly go to My Demo folder I'm going to call chat dopy hope everyone can see my screen clearly I'm going to kill this so first thing we need to do is um to I'm going to loow my API key because the way I I set up my environment I'm just going to quickly do this okay one thing first thing we need to do is to first uh import the open AI uh client so what we can do is we from open air import open AI and then we need to instantiate this uh open AI client I'm going to specify the API key and now they let me actually do one thing real quick I'm going to disable my G Cil because it's too helpful see this is a disadvantage of using ai2 so I'm going to reload my window real quick all right so what I just did is I uh import the open a client into the python script um I also uh set the API key um normally the open a API key would be like environment variable set in your terminal in my scenario I already like configure it in a EMV file so I would just load the environment load the API key from that uh environment file but importantly what we wanted to to do is to create a check completion project a object check completion equal client so I'm going to call the chat method completion endpoint and then create a completion so here message remember we want to tell gbt what the system message is what the actual user message is so here is how you actually can send those information to the GPT model here I'm I'm just going to quickly demo without actually specify a system prom uh you will need to tell this is a user prom so give it a row user and then content I'm just going to say hello world and that's it I also want to uh ask open a to actually use gbd4 to generate a response for me so I'm going to specify model equals gbd4 there are different kinds of model you can use uh so if you want to refer to the actual API documentation open a will tells you what kind of a model is available to you I'm just going to use the latest gb4 model that provided by open AI so now that we have this check completion project created we we actually want to store the response object uh in a variable called response text so again this is my check completion object um I'm going to access the actual text because the completion objects is rather complicate as soon you will see the reason why I access the zero index of J choices is because uh you can actually have the uh gbt model generate multiple response and here we only care about the very first uh response response text okay okay let's try to run it and it might work might not so I'm going to run this uh python spr real quick this is normally the case message okay message try the wrong again and see we get a response so because we send hello world to the model and then the model hello say hello back so quickly we can actually start asking a question here right so we can ask what is flask uh explain it to me in three words remember during the prom engineering section I mentioned that you can actually limit the response the you can actually limit the length of the response generated by the LM model and that's how you can do it in in in your user query here so I'm going to update the user prom here and I'm going to run this script quickly again I run it it get back three wordss web application framework that's very helpful so naturally you would like okay now that that's great now I'm actually interacting with this LM model but then how I can make it more interactive so naturally you will start actually trying to dynamically get grabbing the user input from terminal or from a web application or something like that and for the sake of demo purposes I'm just going to grab the user input from the terminal so the most strict farward way to do it is actually in in uh python is to use the input function so I'm going to store user prom from input user and I'm and I'm going to put this user prom here in the content VII I'm going to run it again you ask me to ask a question now this time I want to ask how can I make an ice cream of course you go ahead and trying to uh help me make an ice cream because I haven't instructed to not to do so but for demo purposes uh this is how you uh prom the LM model for a instructions to get that like a instruction on how to make an ice cream and because I gathered open and R is thinking so hard to generate a response because making ice screen is not that straightforwards and as you can see it send a bunch of test um telling me how to uh make an ice screen cool okay this is kind of like a one-way interaction then you might start asking okay how can you build a conversation and that's this our next uh presentation on on how to make a conversation how to use the open a check completion API to actually build a conversation chat Bo so it's actually similar to building a one-way interaction chat I bought as you can see again I have to emphasize the LM model don't understand what to say to it you only try to complete what you send it what what what you send to the LM model so previously we sent can you help me uh with um my filter let just say for example here can you help me with my filter P this is a input to the LM model the L the LM model sees this particular text it will try to generate a text that best match uh what you provided to the L model for examples here is a respond back to uh generate to the user in order for the conversation to carry on we actually need to tell the a model what you just replied to me because the a model doesn't have memory so what we need to do is literally put its respond back to the message array and Mark it as assistant roow because it has to tell open air to distinguish what kind of a prom it is so so we put it back to the message array and then in order for me to ask a followup question we will actually need to prompt these three uh three lines of prompt three prompts to the A model actually it's the four the four promps we need to send back to the I model so let's go back a little bit so the assistant respond get appended back to the message array so that I can ask a followup question and this will be the new cycle of the conversation we prompt this four um message to the L model and generate the next response we also need to store that so that we can ask another question and this thing goes on and on and on and on and as you can see as the conversation progress the message array we are sending to open air is actually getting larger and larger and larger and larger and that's how it can generate a respon that seemly like match or simly follow the conversation because otherwise the model will have no memory of what you just ask about and let's go back to the code and see how we can actually uh code this so I'm going to check my notes so because we want to build a chat Bo that's interactive that will the model will just keep prompting you after it generate respond so naturally we want to uh do a while loop situation here so one thing we want to do is we want to keep the conversation goes on forever so we use a while true Loop and the first thing we want to do is we want to grab the user prompt here and then we also want to create a like empty array object to trying to keep track of the conversation so we have we instantiate a message array on top before the while loop so first we want to append what I just uh asked what I just provided to the terminal so we will say row user content and user prompt now that the message array has the um has my question or has my user query stored then we can start prompting gbt to ask a question to to get a resp response so again here we no longer need to you know hard code the user prom here we will just replace the message the actual content with the message array so that this message array is actually referring to this particular element it will just keep going larger and larger and larger as the conversation progress these two line will still remain the same while true yep and so that we can tell what response is from the model I will just use an F string to uh help me distinguish uh if this is a respon coming from the model assistant okay user that's looks good to me okay now I'm going to run this again right so I ask what is is flask in three words okay it s b response if we want to test if the model has memory now we can ask did I just ask what application framework now the model actually remember what I asked but it's actually not remembering it's because the message array contains all the conversation history and we are keep we are actually keep reusing that history by pending new uh assistant respond and new user prompt to that history and then we keep prompting the LM model to generate a respon and that's actually how a LM Chat bar it's built okay so one thing with the LM model is hallucinations so Hall hallucination generally refer to the LM model um try to generate a response that's actually incorrect but they it try to persuade you that's actually the correct answer you actually don't know the answer at all uh and one way to tackle this particular issue is called grounding we want to ground the model so by grounding it we mean we want the model to be able to uh reference some kind of a source of Truth when generating its response one particular technique uh that related to grounding it's called the retrieval automatic generation also called rack you can think of it as providing a CH sheet to this particular LM model when generate a response this is nothing related to fine tuning it's like when LM model trying to generate a response you actually hand the LM model hey by the way here is some information that you can refer to or might be helpful to you when answering the question let's just give you an example uh in cs50's context so when student ask a question what is flask so flask could mean anything it could mean like a musical in instrument but in the computer science domain is actually referred to a liveway python server framework but sometimes for cs50 student like mostly at beginners taking the course we don't want the respon to be so complicated because flas can have a technical explanation to it we want to give it give an uh like a response more aligned to the CSP course and more beginner friendly so what we do we perform a rack we per from a retrieval augumented generation what it means is we quickly search through our entire knowledge base usually comprise of the lecture captions we will quickly search through uh what's the definition uh mentioned in the cs50 lecture itself uh could be something David said in the lecture we want that as a source of Truth uh when answering student question so we quickly do that we grab the t- sheet and then we put it in the prompt and then we promp open AI to generate a response so so what is fly is actually not the prom we send to openai it is what um the updated prom we are actually sending to open a so we of course the students originally request or the students originally original queries in the prom but we also uh inject the useful information we think into the prom and then prom gbt to generate a response and concretely what ha What's Happening Here so first when this uh question comes in we generate a vector representation of this particular English PL Tex it's a array of like a numeric floating number um with a dimension of like like 1536 now we quickly search against our database to find which particular lecture caption actually best match to this question so we found that this particular lecture lecture n this particular segment David actually talking about flask the framework so this is actually um one of the caption snippet uh we have here so our knowledge base actually contains of tens of thousand lecture caption chunks it's usually a 30 second chunk so it's like a 30 seconds of all Davis set in the lecture we chunk our lecture caption into this 30 second chunk and then we also create embedding like vector representation of each 30 second chunk and then we store that in a database we literally just concatenate everything that's within the 30 seconds interval we turn them into a vector representation again again this is um encoded using open a text embedding model uh the generated uh Vector Dimension is exactly the same as the one that get generated by uh using students query um so the vector represent the the vector representation we we are getting is always in the same Dimension so when student ask a question right we generate Vector representation we use this Vector representation to do an embedding based search against our Vector database and then we retrieve the best matched uh document we call it the retrieve document which is the best match lecture caption chunks uh we inject that back to the prom and then reprompt gbt to provide a response and that's how we actually ground the L model and in our production in our cs50 duck production we actually injecting like three best match uh chunks so that we can provide more useful information or more truthful information um to LM to to to gbt model when generating a response and nowadays people are talking about Vector database we also use a vector database to facilitate storing and searching the uh embeddings or searching the documents within the database so one database we chose to use is one of this open source database called chroma and this is actually a diagram summarizing the entire like a cury uh workflow so again the cery comes in we call Open Eyes embedding API endpoint generate a vector representation and then we quickly search against the vector database in chroma we retrieve the the the best match three lecture caption we put it back to the prom and then we prom open air to get an answer and next I'm just going to quickly demo what the embedding space search is and just so we are aware of the time I'm just going to quickly open the example for you to look at okay so here I'm just going to quickly show you what a embedding looks like for this particular World Cat and most importantly this is the API we're actually calling called the embedding API so we uh again similarly we invoke the open air client and then we call the embedding service to create an embedding for us uh for this particular word cat I'm going to run this as you can see this thing comes back with like a giant Vector that makes no sense to me I don't understand what it means but this is what it means cat uh from the Open Eyes tax embeding models perspective and you can say and then you can see that the shape is uh 1536 so that's the dimension that this Vector representation is we can actually encode another example what is flask so this obviously longer than the three word cat but let's see what it looks like when generating and embedding it still get back the same dimension of vector representation so regardless how long your input tax is the generated uh Vector representation Dimension is always remain unchanged and that's actually uh um it's model dependent so depend on what text embedding model you use so you might get back different kinds of vector representation um so if you use a particular model to create this embedding you must use the same model to also encode your other you know chit or other chunks of document so that you can perform like a CO and similarity calculation in between these uh vectors so that you can get a score and then you can use this score to determine what's considered the best match uh chunk so that you can retrieve them accordingly and put them into the prom so now that we know how to create an embedding I'm just going to go go through how to create how you would want to create a vector store or how you can create a a bunch of embeddings for your document and then store them in a VOR database so in this script if you are following along or with the source code provided to you what I am trying to do here is actually I'm trying to read the transcript from a particular lecture in our example we are trying to read the transcript from the AI lecture de toop last year for um I'm just going to do a very quick um quick and simple chunking strategy here so when it comes to Rack it also very important on how you pick a chunking strategy so here I'm just going to bluntly you know just bluntly like chunk each uh text every 500 character I'm just going to put this giant transcript into like hundreds of like a caption chunk and H chunk will be like 500 characters long and this exactly what the script is trying to do uh I'm going to store this um chunk caption into a file called the embeding Json L file what Json Json line file means is each line is a valid uh Json document I'm just going to store all this chunk into one giant file so that I can demo in the next script so I'm just going to run this create embedding script what you're trying to do is it read in the lecture captions it Chun them into 500 character Longs chunks small chunk and then for each small chunks we call open a embedding API and create a vector representation of that 500 Kat long chunk and then we store them into a Json line document or you can store in actually in this particular uh stage you can actually store them in a vector database and for the sake of demo I'm just going to store it into a file and now that it finished creating the embedding I can actually quickly show you what it looks like so if you look at this embeddings Json alile as you can see the first 500 characters long of the caption has been you know chunked and then the corresponding embedding is also being created so we did that for every 500 characters and it will come back with tens of thousand you know chunks and its corresponding embeddings I'm not going to finish scoring the whole document because this is kind of large but you get the idea right now we are building a knowledge database in particular we are building a knowledge database containing lots of lecture caption chunks for the AI lecture deau in Fall 20 uh 23 okay I'm just going to close this now that you know how to create an embedding you also have a vector representation of all your you know lecture caption you want to do a search against against it you want to do a embedding space search so if I open this example here what it does is actually trying to retrieve the embed document we just created load them into memory so that we can perform a quick simple embedding base search and the the algorithm we are using here is called the coine similarity is essentially a DOT product between the uh student query and the vector database so in this follow it's actually trying to do a very straightforward search it literally calculate a cosine similarity between the student career and each of the lecture caption chunks in the uh embedding file you can also store that in a vector database when you perform like a vector database in bing base search they have a more fancier algorithm to facilitate a more efficient search but in this demo we are just going to calculate this coine similarity one by one and that's exactly what this um script is trying to do so if I run this enter query let's just say this is an AI lecture but I also know that this is a family leure leure so maybe I will type family lecture and it will try to find um the particular caption that related to family lecture it's not a simple keyword maging it's trying to do a semantic search so it get at this particular chunk and let's see so I remember this particular portion of is actually David interacting with the with the participant which is mostly parent and in trying to find out which particular assets generated by the AI and that's how the embedding base search get back uh get give us the result if you perform a keyword you actually can see any word mentioning family in this particular response so naturally now that we know how to retrieve Reve a useful document how can we incorporate incorporate back to the chatboard example and that's actually the next example going to show you the general process is still remain unchanged right when you're chatting with the chat Bo is just underneath the hook when I send a question to the A model before we actually pass it along to the gbt form model we intercept it we quickly do embedding search get back some information put it in the prompt and PR gbt and that's actually what this example is actually doing here still the same thing um load up the embeddings do a embedding based search but importantly we update the prom here so importantly we are actually building the prom so when students query comes in we do some processing on top of students query by injecting extra information on top of it so I'm just going to run it again so what is um Minimax let's see how it goes so when I ask what is Minimax this particular lecture caption actually get retrieved it actually get injected to the prom and that's actually the entire prom we are sending to open AI not just the simple question what is Minimax but the entire prom that we just updated so open a now generated a respond back to me me but open a open air is actually generating a respond using my updated prom with the provided information um in the prom now that's a lot like you need to come up a way to create embeddings and then you need to find like a vector store like a you need to pick a vector database to store those embeddings um so open a actually has this uh recent uh API Cod assistant API it basically uh simplify this entire process so you build an assistant you can also create a vector store quickly and attach to this assistant so when this assistant trying to interact with the user you will be able to utilize the vector store you created and then quickly um uh retrieve relevant document and answer the question for the user and you don't actually need to worry about implementing the um chunking strategy you don't need to worry about when you should utilize this Vector store the assistant API will automatically do that for you and I think it's worth demonstrating here in this Workshop as well CS is actually not using any of the assistant API but in the future we might so here going to remove for the okay so here what I'm doing here in this script so first I want to create a vector store which is the uh the embeddings that I wanted to provide to this uh assistant so what I'm doing is I I literally update upload all the electro transcript one by one to open AI because we need to First upload the file to open a so that the open a vector store can use it um for the assistant so this three line is essentially doing go to the transcript folder grab each transcript upload it to open AI once we have the file uploaded to open AI we can now build the vector store so you can associate this particular Vector store with any kind of file you want as long as they uploaded to your open a account in this scenario I want to associate um all the lecture caption I just uploaded and I'm going to create a vector store this Vector store you can think of it as a vector database like a tiny Vector database sitting on open server that will allow you to be used by the assistant your soon building now here we want to build an assistant and importantly we want to give this assistant instruction again this is the system prom you're actually writing right now here I'm just going to give it the same example I show you in the slides those are the uh simple guidance I will be provided to I will be providing to this assistant I call this assistant csic duck um I also tell this system here are the tools you can use you can do a file search um the open a document also provide also tells you what kind of a tooling it's supporting currently and you can refer to the document if you one more like detail information here we only care about searching file in particular we all care about searching lecture captions so now that you specify okay you can do a file search okay fine where should I search so here you will say okay please go search this particular Vector store so the vector store we just build here containing all the lecture caption I'm gonna attach it to this assistance so then now you can think okay I'm a 650 duck um I'm a friendly supported teaching assistant for cs50 I'm a Rober duck uh I should not do blah blah blah and when it comes to searching file okay I'm going to look at this particular file store to grab information that might be useful when answering question that's basically what this what these line are doing for the next two um part uh it's a detail implementation for this particular API so thread it's recorded prev in previous example I buil a message array to keep track of the message in this new API assistant API there's a threat API you can use to maintain the conversation history so it's basically open AI help you keep track of the history so that you don't need to worry tracking them yourself of course in production you will also want to keep track of the history yourself when building LM system and each assistant interaction is considered a run so that's why we also need to invoke the Run API don't worry too much about it what it means here um the important uh takea away here is the the picture of how you can build an LM system with rack so this particular while Lo doesn't actually relate to our demo here it might be beyond the scope so what it's trying to do is once you invoke the assistant run uh we will need to periodically pull the Run result and when it's completed we can uh send back the uh respond back to the terminal or back to uh the GUI however you implement but for demo I'm going to run run the demo directly going to run a different version so what it's doing right right now is okay now I'm uploading all the lecture caption to open AI open take care take carees of creating embedding for the thing I just uploaded and it create a vector store and then it also create an assistant with this particular Vector store attach now it is ready for me to interact with this assistant so I can just ask when the read row get mentioned in lectures if you ask this question in chat gbt you might not know because um it was just trying to hallucinate but because we provided this assistant with the full transcript of the 2023 for lectures and David demension wrot somehow uh apparently here in one of the lecture uh at the end I think I believe David I think put out like a QR code something and then student can scan it and it's actually a r r song and actually the assistant was able to utilize the transcript to provide an answer and it's actually a truthful answer and it was able to site you okay this is actually the source I'm referring to in this particular chunk of the caption Rick world does did mention did get mentioned and last but not the least with all this lowlevel API implementation I just wanted to show you one feature that open ey nowadays also available in a in a go approach you can also build the same thing I just did by not writing code at all so this is actually an interface of the gbt's Builder interface so in the chat gbt website you can actually build your own assistant right through their uh web page you can literally tell you what you want to what you want to build and what kind of a personality you want to give it like what kind of information you want this particular gbd to use uh in this example I said I want to create a c5050 duck and gbt Builder suggest okay let's call a C50 duck cool it even it then go go on and start generating like a avatar for me which looks kind of cute so if I keep goes on and on I can actually finish building the CSV D within the gbt Builder and this gbt will be able to deploy and get used by the uh open air user uh globally and I think with that I hope you get a better sense of what's actually happening underneath the hood when interacting with this LM model or interacting with cf50 duck uh these are the building block this API are the building block of CF duck and for sure you can also repurpose it to build another LM system that better seals your need and with that uh thank you for attending this uh AI workshop and this is cs50 and we have time for Q&A and David if you would like to join the Q&A that's fine uh let's go to Leonardo um hello guys my name is Leonardo I'm a software engineering student here in Brazil um um firstly thank you so much for your guys's lecture this is a very um this is a very nice experience to have so my question is on behalf of r a um I am um testing with uh API open AI API and I have yet to use r a in one application because I find it a little bit difficult to instance um uh the documents that the chatbot will learn or how can I say [Music] um I'm trying to sorry my English is a little bit Rusty but what I'm trying to say is like uh when it needs to site sources as you mentioned I have a little bit uh of of a hard time to to setting it up to work so what would uh be the best approach to you use R AG or um can you specify the lines of code uh that you use to do it sure um so by AG what do you mean sorry re direct okay okay so in this particular assistant API I'm demonstrating here first of all the name actually also important so when you're creating the vector store here I explicitly call it cs50 electric captions you should give it a meaningful name so that when the assistant uh it's trying to uh utilize its available resources it was able to actually go to the correct Vector store to look at like relevant information that's when you're interacting with the assistant API if we are talking about Rec in general in the while you're are not using assistant API you are just using some whatever chunking or whatever VOR search strategy you are using I think at least in our scenario our Vector store only storing the lecture captions so regardless what the question is you will always try to search a lecture caption um in that scenario you can rely on like a thres you can set like particular thres like okay what how how Sim how similar the search result should be would you be able to actually utilize those document in another scenario is it might be beneficial for you to find a way to figure out okay what kind of a question it is you can also utilize another LM API code to First categorize what kind of a question the student is actually asking all the user quy is and then you search the relevant Vector store accordingly to re the uh relevant information so don't try to search blind list on your vector database you should search cleverly like first find out which Vector database to search um that might help you solve the problem Rin one question came in the chat when we were coming up with the vector embeddings of text like the words that came out out of my mouth how did we settle on I think 500 characters versus using more or fewer for the windows okay good question so that's so the question was how I decide chunking them so that's often referred to chunking strategy honestly there's a no good chunking strategy there's no one siiz fitall chunking strategy the 500 karat long was just a demo I used here um but in the cs50 duck scenario we are actually chunk down in 30 seconds the reason why we think 30 seconds is a like a switch spot is usually like a concept will be captured or conveyed in a 30 second chunks and and remember we are not just uh providing the uh LM model with one best match we are providing it with three best match so you essentially get like 90 seconds of the lecture that somehow capture the concept that student might be asking so I we think that's good enough but definitely it just the heris that we come up with and one will ask okay if you chunk them 30 seconds like blun bluntly you might omit some kind of a detail information so maybe in the next iteration we can do a sliding window chunking strategy so 30 second and then 15 slide 15 second and then chunks 30 second again so they have some overlap so that might solve the problem but then you will need to but then they basically like double or triple your vector store size so that's a cost you need to consider like the more Vector documents you create um you know the larger the vector databases and also it might induce latency when doing this R um retrieval like R operation yeah time for a few more questions Abdul yeah okay hello so I have a question related to rag so whenever user enters input query so do we just pass that input query to a vector database get the closest embedding to that Cy include those embeddings and pass to the model or are there some other things going on underneath the hood because it sounds feel really simple it uh your understanding is correct that's actually relatively simple but you can get so complicated with it but in our CSV reduct scenario it is that simple so student CER comes in we generate a embedding Vector representation and then we perform a embedding based search against the V database get back the result put it back to the prom do some prom engineering of course and then prom gbt 4 to get better get better result that's that's that simple in at least in CSV du but of course you can go however complicated you want to to build your your rag py line yeah is how about over to SAI next I'm saying that right yeah that's and that is right uh yeah so I'm uh I'm actually wondering about the uh image embeddings do they need to be labeled are there like any you know semantic segmentation or anything or could could the model somehow read you know your inut question I don't know fet summer image or something like that and it would okay okay good question s it's asking about embedding for image um I don't want to hallucinate because I never work with uh image embedding but I don't want to hallucinate here human hallucinations so um of course when creating this embeddings you can also associate with the metadata when you are creating them uh in cs50 do there's one part we didn't mention in the workshop is we of course associate each embedding with a metadata for example in Davis lecture scenario when we created this kind of a 30- second chunk uh we also inject okay which YouTube ID it is associated with this particular caption which time code it is ass associate with this particular caption chunk so in our experiment we can actually have the LM model generate a response so accurate that it will point you to the particular lecture time code and then student can just click that link and then see that particular moment in the lecture um I would I would think that image more or less have the similar approach the metadata is very important when you generating this Vector search because this metadata is only like it's business logic that only understand or meaningful to your to the thing you build so definitely like associate that U metadata in My Demo actually if you will I also associate the metadata with the embedding I generate but that metadata is literally the text I'm trying to generate embedding but you can imagine I can start associating okay this is lecture six this is uh from time code blah blah blah to this particular embedding that way this embedding will become more and more useful when you get back and then you can actually do some laog on this metadata and then decide do you want to actually send this to um to incorporate this in the prom so the metadata is definitely helpful and I hope folks won't mind if we have time for just like two more questions but feel free to reach out afterward I our social channels how about Paro over to Paro okay can you hear me yeah yeah so my question is uh by in this era I has uh in this field uh many plagarism has come and people are I mean copying from AI can we build something that can stop PLM or detect plagorism with vector or AI Vector modeling uh it's a question about building an AI to detect plagiarism plagiarism um first of all using AI to detect plagiarism might not be that simple because nowaday the the the answer generated by these LM model is very human-like it's very challenging to detect in terms of plagiarism um but the however let's just say this's a scenario if you already have lots of like plagiarism cases store in a vector database like these are clearly the solution or the answer that you found on the internet and then you store them in a vector database and then you literally do like a semantic search on the submitted file against the vector database that might somewhat helpful when detecting plagiarism but I don't see a point involving the AI in this process I agree even if you think about to like Shazam or other software that detects or recognizes music it similarly sort of takes Fingerprints of information which is how you would typically find similarities in textual documents that maybe aren't outright duplication of paragraphs but a few words here and here and here um certainly possible and how about time for one more question from someone we haven't heard from uh aaz if I'm saying that right ayaz ali uh hi David sir this is a uh I'm a software engineer from Pakistan U my question is that how do you justify the usage of uh your duck jetboard for example someone is using a open AI J jpd how do you justify that one should use your D chat part like we prefer johnso Healthcare chat P or CH GPD because it provides a specifical and uh to the point and accurate uh answers towards healthare how do you justify that one should use your cs50 is duck Chet board or uh J GPT well I think in our case educationally in a classroom whether it's on campus or virtually online I mean there's a social contract here involved in the sense that we are asking students to not use chat GPT or other tools or other humans that we think will not provide them with the ideal educational experience we have no technical control over that on campus or off it really is the honor System at that point but per my part of today's Workshop where we focused on those pedagogical guard rails that R shin and team implemented what we would like to think is that our duck and our version of chat GPT is better implemented for educational use cases they are training wheels that are meant to be taken off weeks or months later at which point students are welcome and encouraged to use real world tools like Chachi PT which are more general purpose but again a lot of our time a lot of our effort a lot of Our Hope has been to provide students with what we think is a better learning experience and if they are instead reaching for these other tools to Short Circuit that I mean that is not our goal um to facilitate that we want to provide them with a good experience in the virtual or physical classroom all right well I hope folks won't mind if we have to call it a day here and strike all of this with the team but maybe a big round of applause for Rong shin and the whole team here who made all of this possible over the past year thank you to all and please do keep in touch online with any questions uh but you want to have the final word all right this was CS take care