Fireside Chat: Alice Oh

Key Takeaways

[Music] yeah hey everyone my name is basa I'm research scientist at coh here for AI and I'm very excited to introduce our guest and today Alys uh Alys is a professor in the school of computing at kais she obtained her PhD in computer science in MIT in 2008 um her major research area is at the intersection of natural language processing and computational social science uh she has contributed significantly to projects uh that explore the ethical implications of AI improve educational outcomes with automated systems and address the challenges of multilingual and culturally aware Computing um during our chat today we will learn more about Ellis career journey and please feel free to drop questions in the Q&A uh through our our convers and I will be sure to give some time at the end uh for those questions um and welcome Alis and I'm really excited to have you here today yeah I'm excited to be here um it's great that kohir is doing this I've uh watched several of the previous uh fire side chats and they're wonderful and I hope um uh the audience will enjoy today's fire chat fire side chat thank you so much and I will start with our classic pickoff question uh which is how did you first get into Ai and NLP research and what was the initial topic you worked on um so I I was interested in language for a long time um I guess the motivation for me was that I had to pick up English as a second language when I was 11 um when we moved to the US from Korea and so um I had I had learned no English at all before then so so it was pretty uh a lasting experience for me um it took me a couple years to really understand what's going on uh you know in school and with the with my friends and then um I initially started to doing like really like not really research but just looking at what research is like as an undergrad at MIT in uh in cognitive science actually how how um kids learn and acquire language so it was it was human learning that uh human language learning that um got me interested in uh language and uh doing research in language and then I thought I would do something like that I went into computational Linguistics for my Master's Degree at CMU um and there so this was more than 20 years ago but we were um working on this project called the communicator and that is basically uh a dialogue system so it was a spoken dialogue system uh but it was task oriented so um we were trying to build a system to replace uh travel reservation agents and so that was my first uh initial project in in research yeah that's really fascinating actually because I actually ended up in NLP but I started from very different place uh to uh machine learning research in general that's why I don't know I think your story is really fascinating compared to mine um also I know that you have worked on diverse set of topics uh in the machine learning and NLP but one aspect that stands out is the uh over ing goal of using llms for social benefit uh I'm particularly impressed by this Focus could you share uh why fostering this connection between AI technology and societal Improvement has become a central interest in your research yeah um well it's not that I uh initially wanted to go in that direction but as I as I did more and more research uh with you know the wonderful students that I have at kaist um I felt like every time we do something that makes the societal impact uh students were happier I was happier and you know it was just more fun to write papers more fun to do the research because at the end we knew that whatever we do uh is going to have a positive impact it's not just you know let's get out one more paper um to put on my CV on the students CV but it's really you know even if the project isn't successful even if our results don't change the world uh in terms of scientific progress at least we're doing something that um Can potentially have a positive impact for the Society and also lead other researchers into into that direction as well yeah I totally agree to that I think yeah that's that's really great Direction to take um I'm very curious about one thing now you said you moved to us when you were at 11 uh and after obtaining your Bachelor master and PhD uh in US uh you moved back to Korea so what was the main motivation to get back to Korea um so uh looking back we weren't so my my husband and I we we were already married we already had had a child and um we weren't planning on staying in Korea for this long we thought that we would maybe go back to the US at some point but uh for our initial um job after both of us completing the PHD we got uh academic positions uh tenure track positions at the same school in Korea and and kaist which is a Science and Technology University in Korea is one of the most active research universities in Korea so we felt that that was a really good opportunity and um once we came to Korea you know our parents both sides of our parents live in Korea we were both you know we both grew up in Korea and it was it was nice to see our kids growing up in Korea picking up the language and the culture and and everything that comes and so and the university is is also yeah great and uh I felt like I could um make a difference not just in my uh research career but also in advancing um gender diversity of our faculty um and and and our students as well so with with all of that I we just stayed and I've been here for 15 years now okay yeah that's really great um turning now to the theme of today's discussion which is the crosscultural aspects of llms I want to ask how do we train and evaluate llms uh such that they are culturally aware uh what are the challenges to measure the inclusive inclusivity and cultural awareness of llms yeah um it's a really good question and it's a question for which I don't quite have all the answers um looking at I know it is very tough but yeah yeah which makes it really exciting we're just starting right so um the workshops there's a great Workshop uh called C3 NLP it's going to be held at ACL this year um I think it's in their third year or so um so look so C3 NLP stands for cross-cultural considerations and so um it's it's questions like this so even we well we know that the current llms are very good at fluent language generation you know command R3 command plus command R plus command R command R plus um you know GPT models Gemini all of these models are really good at um answering users questions very fluently uh but when you ask questions that are that require some cultural knowledge um other than the US culture so it's very they're very good at the US culture but if you go step you know right outside of the US culture um for seemingly pretty simple P simple questions such as um what do you know five-year-olds eat for a snack in your in this in Korea um the llms are not very good so that is one way to measure cultural awareness do do these models have cultural common sense but to to be able to do that we need a benchmark data set right and we have some common sense uh data sets for English but again um they don't really exist for other languages and they don't really exist for other cultures because if if you're if you're just talking about English English actually covers multiple cultures and not just the US and so you know what happens in Singapore or or even UK or Australia South Africa um these countries in these countries speak people speak English but their cultural uh common sense is not reflected uh in the English large language models yeah yeah do you know what tasks are particularly sensitive to cultural context yeah yeah that's a good question so so far we have worked on uh Pace speech detection um that's very culturally sensitive so we had one paper in which um like uh the Nazi right so the the whole you know Holocaust so if there is some hate hate speech uh against the Holocaust Survivors uh in English the language models detect that very well but if you translate the same sentence into Korean for example um the translation is perfect and anyone who can understand Korean can know that it is hate speech towards uh the Holocaust Survivors but then the Korean hate speech detector does not detect that and the reason yeah the the reason is it's actually pretty simple and obvious um it's because when we build hate speech detection uh for Korean we get the data the training data and the um test data from Korean uh hate speech which for which we don't really talk about the Holocaust that much um it's just in the history books but we don't everyday people you know we usually talk about or their hate speech comments are directed toward other targets um you know like gender-based hate speech or um or ethnicity is and races that are minority in in Korea so so because this topic is not within the hate speech uh data set in for Korean it just does not detect that as hate speech and so that's haast speech is one topic another um related topic or related task is uh social stereotypes so um there is I in America or in the uh Western culture um there's hate speech to I mean sorry social St negative stereotype towards um African-Americans for example um maybe certain religions like Muslims they say you know are are violent or or terrorists or something like that um if you look at the Korean Society uh we have negative social stereotypes towards um Chinese for example or or Japanese because our they're our neighbors and you know there's some hostility um uh like for historical reasons and so and such and so and so those social stereotypes uh not all of the social stereotypes are culturally dependent uh for example like stereotype against the elderly is universal across at least uh for Korea and and the us but um you know racial e or ethnic uh social stereotypes are very different in in the two cultures and then another um task that we are looking at recently it's not re I I don't know if it's we can call it culture but uh looking at factuality is also one that um you can say is culturally independent so the language models if you ask about certain people um let's say like Barack Obama right Barack Obama is a famous figure across all languages all cultures so if you ask about Barack Obama in any language to these language models um the answers will be very factual uh and and detailed and and very little hallucination but then if you ask about I don't know a a an African leader um a leader of you know Egypt or or Greece or you know some some other country uh let's say in Korean it the the answer is going to be non-factual there's going to be a lot of hallucination and so um I guess it's sort of like geographical no knowledge um or knowledge that is concentrated in in certain geographic regions um they tend to be uh when you ask llms about those types of facts llms are not very good especially in in different languages yeah but it's not only about the multilinguality right it's more about yeah that cultural differences that's right yeah that's right right right so um yeah it's all of all of that it's both it's the combination of the language and and the culture so if you have a low resource language and low or less well-known culture or culture that is very different from the from the Western culture um then the accuracy goes down dramatically yeah okay so that uh brought up a question uh so how do we Define uh a low resource culture and how we can on this yeah um how well we can't really Define I guess so what we're try what we're doing empirically with these llms because we don't it it's very difficult to know uh what the training data are for for these multilingual models and you know it's it's actually the training data that is uh that has the biggest effect on uh cultural knowledge of of these models right so and we know that Wikipedia for example is is a big part of the training data set especially for um languages other than English um but when we look at Wikipedia of course the number of Articles uh about that describe certain cultures I is going to be very small and and and also in languages other than English so that combination of um less well-known culture and less well known language uh I think a lot of it I mean I think one way to um look at the empirical distribution of of the knowledge and the language is just counting the number of Wikipedia articles right so if you have a have a political figure in you know I don't know Greece and um uh and just count the number of Wikipedia articles about that person in the different language versions a Wikipedia it's going to be much smaller than let's say Barack Obama for example yeah yeah yeah whenever I check the same page in Turkish I'm Turkish it's so much shorter than the English version of the yeah yeah yeah you you can't find any details right that's right uh I see the challenges but what are the potential solutions for incorporating crosscultural understanding into llms yeah so that's that's the biggest challenge I think it's going to take some time um because as you know uh data data is is the biggest bottleneck uh and so we are trying um it not just our group but many people around the world many researchers are trying to generate data um using llms um using a combination of of llms and human annotators you know scanning in documents just the combination of all of these and translating um documents from one language to another and then uh after you do the translation maybe some cultural translation as well so just replacing some of the entities um with the entities that are more relevant for for a certain culture so the combination of you know just getting more scraping more data by you know scanning in documents having people human annotators generate data like just asking them to write or speak um to generate the language data and then asking llms to um to generate synthetic data and then having human annotators validate the generated sentences so all of that all of that um can work uh the problem again is for um low resource languages and low you know lower popularity uh cultures this also becomes very difficult because when we when we tried so we we have this we have this paper on Indonesian and sundan and sundan is this local language within Indonesia and um you know Indonesian is like medium resource so language models are really good at Indonesian but when you take sundan which also has a pretty large population I think it's about 30 million speakers um so that's a pretty large population of people who speak this language but the language models are not very good and so when we take um like English Common Sense QA for example and then translate into Indonesian and also into sundon and also um ask llm to you know validate whether this common sense question makes sense in the in in Indonesian and synon and whether it's culturally relevant and so on and and also make it culturally relevant um the language models do very well for Indonesian but but they fail for synon um it just the generated sentence the machine translation isn't good um The Entity replacement that uh to to make it relevant for synth culture doesn't work um there little you know just lots of mistakes so so then you have to resort to um to the human annotators but you know again the problem with these low resource languages is that it's difficult to find the human human annotators too so it's kind of a chicken and egg problem and um but some you know some we we have to get started somehow yeah that's yeah that's very true and um given the significant challenges associated with data set creation in AI research um I really appreciate your effort uh but what drives your commitment to this area um that's yeah that's a very good question um I asked that myself too um because I know it's super hard yeah yeah it's yeah yeah it's hard and it's at the end of the day um I do ask you know how many people is this going to affect are we making progress is this going to be beneficial to my students who are working on these really challenging problems that sometimes it's hard to publish too because we just don't have the reviewers who can who can review these papers um but I think so many of my students work on these problems because it's because these are their own languages so the student who's working on S Denise is uh well so she's Indonesian but her one of her parents speaks us syon and these local languages in Indonesia and many other parts of the world um the the number of speakers is decreasing so the younger generation actually doesn't speak the language um so it's one way to try to preserve the language and make these language models before all of the native speakers disappear right so so that's that's a really uh important cause I think um so the bigger picture is that even um for languages like Korean where we do have a lot of uh resources I mean very small compared to English but compared to other languages Korean is is pretty high resource language and we have these big tech companies in Korea who are working on these Korean llms but we we know that compared to English the the data sets The Benchmark data sets uh are still very very little and so it's still when I compare the Korean llm with the English llm from a user's perspective it's still very clunky and the output of the llm makes me laugh sometimes and there's you know harmful output um there's bias there's all the all kinds of problems and so you know even for these major languages we have these problems so that means when these llms and more generally gener of AI Technologies are going to transform the world um the work that we do you know the way we study all of that is going to be transformed but all the languages other than English and maybe you know Chinese and maybe a couple of other languages all of the users are going to be uh falling behind the English speakers and and that's a big you know AI divide issue that uh is going to be a global um challenge Global obstacle for us to overcome so I'm very happy to contribute anything to to um make that problem uh mitigate that problem a little bit yeah um also uh I think the role of the uh cultural experts is really crucial in the AI research for example uh your Indonesian student and they can really contribute lot uh and a project was one of these initiatives that uh we tried to use these cultural experts and Incorporated their uh knowledge into the AI systems and how what do you think how can their insights be integrated more effectively this is very open question but I'm curious how we can do it better and more effectively yeah I think I have project was wonderful uh it it's the best example that I can think of to get the community involved um and I know so I spoke with Sarah about this and you know and and I read the papers and you know it's a yearlong lots of work lots of um hard work by so many people to to make the aad data set and the models and a wonderful example um I think it will serve as a starting point because I know that the IM am model um you know focused on the the number of languages the broad the breadth of the community um and the resulting data set uh for instruction following is great but we need to do more than that right so we need uh training data sets um on top of or before before the instruction following data think yeah yeah but I think we can learn from that experience to try to mobilize uh an entire community of interested people and make them engaged and I know for from the I project you know um it's very difficult to engage people to to keep keep uh committed to the project um and then there is a similar project uh that Google has done with uh sunipa did this project with Indians and uh people in India uh speaking um multiple languages actually they they were focused on English but there they were trying to um collect Common Sense Knowledge from uh multiple regions within India and that was also another you know Community effort to make this data set so um we haven't actually done in our lab like any Community Based uh data collection but I think that's I think that's the direction that we want to go in yeah or maybe we can do it in our regions in our own languages and and maybe merge the effort um yeah yeah another thing is actually so Indonesian um there is a and and also African languages there are communities of researchers right so there's a uh paper called NSA crowd and it's a bunch of uh researchers and the data sets that they've made from around the world who are doing Indonesian NLP and so getting all of them together into a community of researchers so many of them are local in Indonesia but some are uh in Singapore some are in Korea some are in in the US and so on and um African NLP community Al also uh has has that Community effort of researchers and I think that um when you look at even lower resource languages I don't for example like we work on Bengali also so we it would be really great if we can get the local Bangladesh universities and researchers um and I don't think currently they have the research capacity to do llm research um but you know some if if we can educate them somehow and get them up to speed then then and then collaborate with them and so mobilizing the local research Community plus the local um native speakers uh and just have building this community would be great I think I think that's great idea I I hope we can do that uh very soon um yeah just a reminder for the audience uh feel free to drop your questions to the Q&A uh so we can ask it to ask those questions to Alis as well um also yeah just recently llms has started to be used in educational applications and I know you also have Works in this field uh I want to ask what are the potential benefits and drawbacks of using llms for learning experiences yeah um it's so I we've done some research and you you probably saw the videos from open Ai and also Google so so both of those companies have just made a splash right so I didn't know that they were working I well I knew that Google was working on education um I didn't well and I guess I knew about the con academ um collaboration with open AI but but it was really uh shocking in a good sense to to see those videos and I think those videos really show the potential for uh education personalized education using generative AI so the open AI demo was about um triangle and so like learning about the Trigon trigonometric functions like s cosine um using uh multi modal Lang multimodal model um and I think so one thing that llms are currently not very good at is being pedagogical so these when you when you ask so so in that demo I don't know exactly how they did it but it's very difficult to tell the prompt the llm to give hints but not the direct answer but if you but that's what teachers do right so if you if a student is struggling with the S cosine those you know functions the teacher is not going to say you know this is how you do sign this is how you do cosign it the teacher is going to you know ask just like uh GPT 40 was doing like what is the hypotenuse what is the side what you know relative to this angle Alpha Etc and so how do you get the llm to do that but once you can get the llm to do that um you know speak like a teacher then there's tremendous opportunity for personalized education getting students more engaged um giving education access to a wider uh population around the world there's just so much that that we can do and one thing that we um we did we we had uh English writing assistant so here at Ka um we have students who are not native speakers of English but we do all our education in English so the college freshman have they all have to take some English classes and so for English writing essay writing class we built a web-based platform to um to enable students uh to empower students with with Chachi PT and um ask them to use it in their essay writing and we capture all of their conversations with Chach we asked them for each uh GPT answer reply we ask them how satisfied they are and so and so we we can see like really interesting patterns in their interaction U for example you know they initially start conversing in English um because they feel like you know they have to for some reason and then they quickly switch to Korean once they figure out that you can actually ask in Korean and they feel more comfortable um and then they switch back and forth and then they do this like in the middle of the night you know at 2 am so we know that college students they they work on their essays you know while while all of the Pas and professors are sleeping um they say things to the llm um that they wouldn't say to the professor so for example like is that is that correct I I don't think you're right right like challenging the answer not accepting the answer um as as the correct answer and so just interesting behaviors but we're just starting to realize the potential uh for benefit to education and we're just starting to understand how students may be behave when they're interacting with these uh llms as you know as AI tutors yeah yeah and I think uh they will really help uh in providing scalable and accessible educational resources especially in under resour resourced areas but I think first we need to improve the under resourced areas in llms for that um that's right what what do you think about this um so do do you really think it will I mean increase the accessible educational resources yeah yeah definitely um so you know locally we have this problem um of uh students getting a lot of private tutoring help for for English and math in particular um so we are working with Middle School teacher actually um who wants to use llms in Middle School English classes and when we went into the class and talked to the students their prior knowledge of English is so uh there's a wide spectrum so there are students whose parents you know are fluent in English or they have they take them you know they're very rich so they take them to to the US you know every summer to to get to get exposure to the language or something and and the same thing with math too right so if there are students who have private tutoring so they're very well Advanced and then there are students who um who struggle and whose parents are not very helpful with their homework or something and so if you have a classroom you know of 30 students and there the spectrum of level their level of understanding is so wide spectrum and then you really can't focus the teacher can't really focus on uh each individual student so we're hoping uh llms can help in that space such that um especially the ones who are struggling with these Concepts or with just basic level of English can pick that up quickly without you know having to ask these embarrassing questions in in class and Etc yeah yeah right yeah I was just recing recently talking to a friend of mine who is a therapist and she was uh doing the therapy practices in German to improve her German because yeah she's a therapist and wants to do therapy in German and she she's role playing just asked to role play the llm as a patient uh she was doing the therapy with llm and she says it's really successful um that's great anyway yeah okay uh I want to go to the audience questions now um and H asked there are significant differences in the amount of data between countries and even within each country there are differences in data depending on popularity and exposure is the ultimate goal to overcome these differences however we haven't yet overcome these limitations on the internet so is it possible for AI to to do so the internet is inaccessible in some countries yeah um this is exactly one of the motivations that we we as academics or I guess even researchers in Industry uh should be working on this problem we can't just trust the internet to do the right thing because it's not going to to do the right thing um if we are I mean I don't know if it's the ultimate goal um you know I mean there are thousands of languages it's obviously very difficult to cover all all of them but we have to at least provide the provide sufficient level of accuracy for language models for let's say at least the top 200 languages that are used around the world and as as said you know we don't have data so what do you do um somebody needs to be a champion so you know for example I have students who are championing uh Bengali they say you know we don't have data the language models don't work well we don't have the Benchmark data sets the mlu data set is just translation of English and it doesn't really work for for Bengali so we need to do better and so I hope I hope I can Inspire um lots of students and researchers to Champion one or two languages and you know collect data do annotation train models um do all of that yeah yeah that would be really great uh another question is when we are thinking about cross culture and LP how do we disentangle weaker General ability in non non-english languages versus lack of cultural knowledge yeah um it's difficult to do but there are uh ways to so so there is um when you translate English data sets Benchmark data sets into these languages um you can measure for example um uh the variation in in the answer so like if you ask a math question English and ask the same math question you translate it into uh to Korean and then ask it then the answer when you translate it back it's going to be the same uh hopefully if it's not the same then then then it has it lacks the general ability um but when you ask a cultural question then the answer is going to be vastly different and also um when you ask people these cultural questions people's answers are going to also vary which makes it even more challenging because when you have five different ansers for a question um then are the annotators like did they annotate wrong or are they just dis disagreeing in a good way because of the diversity of of the culture it's difficult to know but when we ask um you know using different prompts using different llms um the same question question and the answer that you get is consistent then you know these are questions for which there is uh sort of relatively less cultural variation and then when when the answers come out different um then you know that we need to concentrate and focus on those questions because they are either you know either culturally diverse uh in in terms of answers or or they're just very difficult questions to answer yeah okay uh another question is I'm wondering if you want to have a culturally aware llm should we GA the same amount of language data sets or apply transfer learning and train on English first the first approach would be difficult for lower resource languages and the second approach might not be able to pick up the cultural information yeah I really think it depends on the on your goal right so and so I think it will depend on um the kind of llm that you want to I think you know we're not seeing it yet it seems like the llms so far are just general purpose llms but we know um llms are now going to be developed in different ways to have sort of different characteristics like a models for example are designed to be just massively multilingual and and and so it's a different I think of it as a different type of llm as opposed to um command R plus which is you know focused on 10 languages and and for accuracy for those 10 languages and so I think we may have models that are focused on just general ability um in including English and just multilingual uh language ability as opposed to which I think most llms these days like gp4 I think is a model that is just focused on fluency across different languages and and accuracy in in their answers but gp4 is culturally very lacking but uh we may design language models like the ones that we try to train um are using you know transfer learning or or um or specialized data sets or in in what way um we're focusing on building in cultural knowledge into these models yeah um okay maybe another question from the chat so how do cultural differences show up in English language llms only and what kind of customizations can be made there Beyond different differences across languages yeah so one paper that we have um it's going to be presented next week I think at nle um is f is on hate speech in English in different cultures and here you know the caveat here is that we for that paper we Define culture at the country level which is still pretty it's it's not exactly but you know if you compare UK versus US Australia Singapore and South Africa these are five Lang five countries in which English is the official language and there is a large proportion of the population that speak English as their as their major main language and so when we take uh hate speech data set um in in English and apply it to these other cultures so we ask people in Singapore for example is this hate speech or not they don't necessarily agree with the American annotators or the UK annotator so these annotators across the different countries disagree on many of these sentences and if when we collect so we collected additional posts that are uh hate speech within those uh five different countries um in the social media channels that that they tend to uh frequently use um there are many words and many targets of hate that are not that we don't see in the American data set so in the Australian data set for example there is hate speech against the indigenous people the um yeah the indigenous um First Nations people and um they're called abos and and the word abos we don't see in the American um data set so and then there other that happens in in all of the countries and there are also local knowledge um and local cultures I guess so for example like in in America there's a lot of gun violence so there a lot of hate speech about gun violence but we don't see that in Singapore or in South Africa or or in Australia because people don't possess guns in those countries and so there's certainly a lot of cultural difference and the llms that are trained um using mostly American data right so the when you ask llms whether to to detect hate speech uh they don't do as well um the accuracy goes down significantly for um Australia and Singapore and South Africa and so on okay yeah this really interesting and requires a a lot of research to do um so I want to move to the last part of our conversation so um we know that research is tough at times sometimes your hypothesis doesn't work and you only can learn it after months of work etc or everyone has ups and downs during PhD for example and what was the most challenging period or thing in your career so um when I started at Ka as Junior faculty um I probably wasn't ready like I didn't do a post after my PhD and I sort of Dove right into my junior faculty years which was very difficult I mean being Junior faculty in a tenure track is is just tough for everybody um but you know I had a I had a baby right at after I got my job so so that added to my my plate of things um and I felt like I wasn't making progress fast enough and and this plagues pretty much everybody in pen your track but um but those were really tough times and um I guess I kind of struggled for at least five years in my uh assistant professor years yeah uh and I guess my tip would be I mean it's it's easier now that I look back it was I don't know if if somebody said this to me I don't know if that would have eased my mind but you know to have a longer term view of your career um even if you struggle in the first few years of your Junior faculty or as a postdoc or something um if you keep uh the one goal that you have in mind and keep working towards it it's going to pay off but it may just take a long time you have to sort of endure through uh the tough sort of initial years and sort of feed yourself a lot of positive feedback as much as possible right so like any positive reward that you get you know you get a paper accepted even if it's just a workshop paper you know it it's positive feedback that your research is going in the right direction and so so you have to sort of enjoy the small wins that you have and have a longer term view yeah I think that's great advice um and I can also ask uh about your advice to students um PhD students or students who wants to apply to your group for example or uh to the PHD position so um yeah what what would be your advices for them so um H that's a tough question are yeah people are mostly curious about how they can start to their AI uh career so where they can start um what's the most important things to to keep in mind and consider so I know it's tough for students these days because I I know even at kaist a lot of the undergrads do research very early on and so you know by the time they graduate from college they already have papers at these you know sometimes these top tier conferences which was unheard of even like I don't know 10 years ago I think it's just recently that this trend has begun um so but I think uh it's not necessary for me for my group at least it's not necessary but I think it is necessary or helpful to know the field know um the research questions that are being asked in a specific group so in my group it's as as we talked about today you know it's using NLP and llms to uh make positive societal impact so what are the important societal problems that we can try to solve with llms right like education is is one domain and so if you have um if you show enthusiasm and commitment to um making positive impact and not just you know make the model better on some Benchmark data set um but really think about the meaning of the research then that is good enough for me um for my group but I know different groups have different standards and requirements yeah okay uh maybe the last question so where do you think AI is headed over the next five to 10 years very exciting places we're heading we're we're I think we're in the middle of a very exciting phase and I know that it can be overwhelming um it's overwhelming for me you know when I saw those uh KH Academy open AI video of the triangle I was so shocked um and I used it in my class like right after it came out uh to show the power of llm for Education uh but I think it's um so there's a lot of potential for making positive progress and really engaging with the world so I think okay the when I say the world the real world so I think a lot of AI progress has been made in the last decade on data sets that are somewhat removed from The Real World so you know think about image net um and even these llms it's just like predicting them the next token right and I mean that's great but it's only when um Chacha PT was released and then you know llama 3 and Gemini and all and all of these models that can actually interact with everyday people like who are not researchers um that we're trying to solve real world problems and I think we're just beginning that exciting journey into you know what are the real problems where can we help um you know drug Discovery um Health Care education uh environmental problems all of these real world very serious problems can be mitigated with llms I mean not to say that llm is the solution to all of these problems but we can use llms to make progress but at the same time um so I was I was just at the AI Sal safety s Summit um with you know the ministers uh of countries around the world just today this happened today um where policy makers are really concerned about the potential harms and risks of AI and you know I don't think they go as far as exist existential risk and I I don't either but there are certainly um potential harm that comes that can come from misinformation disinformation you know deep fakes um potential harmful use uh by malicious actors and and of course biases and stereotypes that are uh prevalent in the llm so so it's an exciting um Journey for the next five years where the llms are going to make so much um difference in so many different domains but we have to be careful uh about the risks and potential harms as well yeah I share the same excitement and concerns with you um yeah and um we just hope it will be really just exciting um yeah thank you so much for accepting our invitation it was really lovely chat and I'm so grateful um and thanks thanks everyone uh who attended and for their interest and questions and yeah thanks again thank you thank you Vasa for having me and for your wonderful questions and comments and thank you everyone for joining and thank you coh here for for having this fireside chat thank you so much bye bye bye oh