When AI meets data: Stack Overflow and Gemini

Key Takeaways

[Music] hello everyone hey how's it going welcome to uh Dex conference hopefully you guys are having fun yeah welcome well so quick show of hands here this might be a horrible experiment we'll see if it works how many people here have used stack Overflow in their care that's pretty much everyone all right awesome that's great to see right that that'll hit our numbers here in a minute as well as we talk about the developer community and and how many people are engaged with the stack Overflow site right um today well I guess we'll kick off by introducing ourselves I always skip that part go ahead yeah everyone my name is Chung I'm product manager at Google Cloud um very nice to meet you all and I'm Ryan PK Chief product officer for stack Overflow um and what we're going to talk about today are a couple of key topics to us things that are really important to us at stack and really important to us in how we work with Google and the Gemini model overall right and so we're going to walk through a couple of key factors one socially responsible AI this is something that I think we've all been hearing about in the news but as an organization this is really important to stack right this is really important to our community as to how we integrate with the major llm providers how we provide our data to people and how we work with them to to further our community so we're going to talk a little bit about socially responsible AI from there we're going to actually walk into some of the mock-ups around how our products are going to be working together um things that we're doing on stack Overflow and things that we're doing uh on Gemini for Google all right uh and so beyond that when we get through all of that we also want to talk a little bit about the power of the data the power power of what we're doing with Stax data and how we use that to create relationships in the llm world to really drive back the value into our community and so we're going to talk a little bit there as well we'll have some time for Q&A so more than happy to open this up to the the crowd and we'll be bringing a mic around um shortly after we get done so please feel free to ask questions when we get to the end I'm going to start off by walking a little bit through Stack's Journey over the last year and a half right um it's no secret that you know when the when a major llm came into the market a year and a half ago um we saw a little bit of waves I wouldn't really call it disruption I'd call it waves in the usage of our our site and things Behavior changes in the users on our site as well right people went out to the llms um to get a quick solution to things but they quickly came back throughout this last year to really learn more about the answers they're looking for right we found through a bunch of different tests on our site if you've been a user you may have uh did anybody sign up for our our uh private Labs tests around AI capabilities of course the stack team but there's a couple other folks in the here as well um we did a lot of different tests using ai ai capabilities on the site what we found were a couple of different things when we we implemented a pure chatbot um we found a really cool link between satisfaction people were getting the answers they were looking for they were getting a quick solution but what we found was they weren't really learning they weren't really moving to that stage of actually digging into things and learning more from the community overall so what we found as well was that really humans are the best source the best resource for providing context and accurate answers to people engaging in the community so we get a quick solution which is amazing from the AI capabilities but when you really want to dig in and learn you have to get into the community itself the AI capabilities also gave people the power of the first draft and we're going to talk a little bit about this as we talk about the different solutions that we're putting into place um but Knowledge Management as a a core element of what we do is still incredibly important right the ability to take a basic solution and then dig in behind the scenes on that interact with community members ask questions engage is an incredible portion of learning and so what we're looking to really drive is that element of learning in our community um aided by AI capabilities another element that we found um we did a our yearly developer survey this is a huge survey we had about 990,000 participants in this um we asked a couple of key AI questions in the last survey in 2023 an interesting one here 70% % of developers um either are using AI capabilities or plan to use it in the near term as this is a couple of months old already they probably more or less more of them are actually in that using category on the other side of this though only 42% either somewhat trust or highly trust the accuracy of the data they're receiving right well trust is something that's interesting because we've seen the AI models gini models evolved a ton over the last couple of years um so trust isn't necessarily purely about the accuracy of the data being received right trust has two basic elements when you talk about Knowledge Management one a simple question why is the sky blue why is the sky blue I'm just screwing with you I'm not going to actually pull you up on stage um why is the sky blue if I ask a friend why is the sky blue right what I'll get is an answer probably if they know enough about this something around you know the wavelength of light moving through the atmosphere and so forth right um I'll probably trust that answer if it's a friend of mine but if it's a stranger what do I do well the first thing I would probably do is is go to the why do they know that what are their bonafides what is the their background that tells me that that answer is correct right well if they answer you have a PhD in in physics then I would say oh your bonafides are amazing right um it'd be cool if you did have a PhD in physics by the way um but uh on the other side of that how how else can I trust that if the bonafides aren't exactly what I'm looking for well I can dig in I can ask how do they know that not just why and go into the details of how they learned that activity I can dig into the knowledge underlying what they're they're providing me the quick solution they're providing me and so what we're looking for here when building trust is providing that how I I expect in the next couple years with Gemini as we look at this people are going the bonafides are going to be there people are just going to start to say well Gemini says it it must be right right um so the bonafides will be there but the how is really what we're looking to provide in Knowledge Management socially responsible AI this is not just a buzzword this is actually a movement in the industry right this is where the industry is moving this is something we've been experiencing and we've been promoting at at the same time right this is a movement towards the major llm providers creating relationships with the communities that they engage with and giving back to those communities while they consume the data from those communities right this shift has been dramatic over this last year um and the relationship with Google is an amazing example of a company taking socially responsible AI um with full power Full Steam full energy and being a huge backer of this one of the key elements that we push really hard is the idea that attribution is non-negotiable pointing back to the source of your data pointing back to where you are actually integrating with um and finding your solution be simply put it this simply um how do you know the answer is key we believe that content itself as we build more and more more content on the web is still best human driven right at the very least we need to have a human in the loop in everything that we're posting right this is a key element of how we are creating relationships with the major llm providers and feedback from communities can be one of the most powerful mechanisms for training and building llms and AI models in the future and so integrating into these communities is going going to be a key element of every llm provider as we evolve what we're looking to do is set the new standard with vetted trusted and accurate data that will be the foundation on which Technology Solutions are built and delivered to our users right so this is a loop for us we want to be the provider of the data we have an incredible data set we'll talk about that in a little bit but why are we providing that we're providing that to create relationships with the ma major providers to then come back around and leverage those relationships to bring more and more capabilities to our community right this is the loop we're looking to provide and this is key to everything we're doing you'll see this on our road map um and when we're communicating with our overall Community this is the foundation of that providing more and more value with the relationships that we create Google's got some great examples here coming up um with how they're do attribution and integrating back into our community um and this is something we're excited to announce at this show now I will talk a little bit about how we're using Gemini as well um oh shoot I'm I'm pre-t talking your slides that's my bad all right I'm gonna hand it over CH thank you Ryan thank you so much now let's talk about what the partnerhip between Google cloud and stagger flow means for you right in your Cloud experiences see that's that's why I thought I wasn't pre-t talking your slides got me you got me that was good um rewind that back um let's talk about how we're integrating Gemini into the public stack Overflow site right now about a year and a half ago this is before I joined we had this concept of what we called staging ground this was a cap ility that was really popular in our community was a basic concept to understand the stack Community you have to understand that the original Mission and the continuing Mission has always been for our community to be the world's software development knowledge store right to be the center of that software development knowledge and one of the key elements to to this we've all kind of experienced it in some ways is that we're relatively strict about what you ask and what you post on the site right which can be a barrier to people this is something that um we take a little bit of roasting on on on the public sites out there for but it's valuable because it drives the quality of the questions and the quality of the answers on our site and so that's an incred that's a valuable Loop to us that we need need to continue to support as we built out staging ground the basic idea here was to help improve the quality of questions by creating a review process where people posted questions and then other community members could help them review those questions give them feedback help them tweak it and get that question in line with what was expected as to a good question a high quality question on our site we launched this uh in our lab's capability about a year and a half ago tested the heck out of this and what we found was that although this was incredibly valuable it was also something that didn't scale right we were having a lot of questions moving through the process automatically because we didn't have enough people to review them on a regular basis this is also a great example of what AI can do for us right something that we can bring these capabilities in um and help the person who is crafting a question both in how they you know how they craft it what they're crafting helping them with their text helping them with their overall context of the question looking at expected outcomes helping them with Framing and formatting and things as an element of helping them craft but it's still their question they're the one writing it they're the one who's engaging with the community this is something that helps them build a better question and helps the community build a better knowledge store um and the implementation of Gemini here is something that is going to be the the first major step in new capabilities on our site now this will include though as I come back to that human element the next stage we we will continue to have a community review of this and our expectation is as we go through the AI review the community review will be much faster and we can scale much much more effectively even though we are still having humans engage with each one of these questions right this is important to us to make sure there's always a human in the loop as we we build new community capabilities now I'll hand it over Chang and we'll go through the uh the over Google Slides now was my turn for real now you can see uh how tightly we have integrated our Sid even friends together now uh but again now uh let's talk about what this partnership really means to you um and how we are bringing the value to Google Cloud customers so first um let me introduce you Gemini clout sist how many of you have been to the opening kyot quick show hands majority of people so this is shouldn't be news to you uh Gemini cloud assist brings a transformative approach to Cloud management experience by leveraging Cutting Edge gen Technologies and our deep understanding of Google Cloud what this powerful combination brings you is really to address the complexity and challenges our customers are facing every day Frankly Speaking uh a lot of pra uh Cloud practitioners in our industry are facing the same challenge um you know Gemini CLA cyst solidifies School Cloud as the best cloud vendor on the market for ease of use and achieving optimal results helping you now let's take a quick look at how Gemini Cloud sist can help you uplevel your Cloud management experience first it helps you to accelerate outcomes uh and achieving your Cloud fast uh achieving your Cloud goals Faster by assisting you through from design to deployment troubleshooting our operations and optimization secondly it simplifies your task management experience by automating and intelligently guiding you through um all your Cloud uh work workflows and third it offers a tailored experience um because Gemini knows who you are knows your resources so they can provide you a personalized experience based on your specific environment and your Cloud resources finally lastly but not leastly from a security and compliance point of view as a Google Google call class customer you can directly benefit uh from Google's best security practices up to-day knowledge and Enterprise great security and compliance that you can trust again uh Gemini Cloud assist is powered by Google's Gemini family of Industry leading uh models it offers Google Cloud white AI assistant for entire Cloud management life cycle experience from design to operations to optimizations and it offers an intelligent and assisted Cloud management experience that cuts costs uh boosts your productivity and Foster customer Innovation lastly but not leastly uh it not only available on the cloud console we also made it available on our Cloud mobile app so all of you can uh enjoy the power of G even on the goal okay let's uh dig a little deeper into how it all comes together on go on Gemini Cloud assist this actually will show you how we integrate Stager flow and bring the community knowledge to benefit your Cloud management experience on Google cloud in the center of the diagram you'll see is our multimodel Gemini models which are train a massive amount of documents books code video audio image uh files and we fine-tune the the uh the foundation model for Google Cloud uh with our specific domain knowledge and then each prompt user submits are also augmented with your specific context uh users information Telemetry data and on the top left corner you'll see Stager flow data along with Google uh additional Google data that we use to actually ground fact ground our response this helps improve our response quality and also this brings a unique set of capabilities to our responses that perhaps other open um other uh gen Solutions don't have the response can also be made on the right hand side you see a variety of experiences and services including both best the desktop as well mobile and various forms uh of experience that user can interact with because Gemini clais is powered by Gemini family models it can incorporate new sources of data to continuously improve the quality of answers we're super super excited to announce that this data now includes the body of stack Overflow posts which brings the expertise of the world largest community of developers knowledge right into Google cl's Cloud console now when Gemini gives you a response that was informed by stacker flow's post you can see a call out Link at the bottom of his response as you seen on the on the on uh on the animation on the right hand side um you will see that link shows up and even marked with the Stager logo allow you to visit the community directly you can get more contexts on you know what this post is about Ser real links and understand how and why Gemini gave you that answer as a summary adding the depth and the breadth of stackl communities knowledge to Gemini helps her user to get better answers to your questions and we're constantly updating our grounding data with the latest post so that you don't have to worry about you know stale answers data out uh data cutouts or scenarios where you're working on something brand new and you can't trust your to have the most Cutting Edge answers because we are constantly updating these data with that Ryan back to you to talk about how this partnership can uh be mutually beneficial to both stack customers and Google call customers very much so so let's talk a little bit about the evolution of the llm and AI market right and the power of the data that is provided by the stack Overflow Community um so go through a couple of the stats here I I liked our our impromptu survey here at the beginning um 92% of developers visit stack Overflow regularly right this is a a huge number and I we were close to 100% in here when we did this the survey we have 15 years of highly accurate high quality trusted knowledge um contributed by the community and let's be realistic here if we had if we went back in time uh 15 years and said we were going to build the the perfect knowledge base for creating data for llms probably couldn't have done it much better than what stack is today um and that's important not necessarily just for the data but for multiple reasons as we we'll talk about as we move forward we have over 58 million questions um and answers on the site about every a question is asked about every 14 seconds um and answers are provided on a much higher frequency 69,000 technology tags different topics on the site um and this is an interesting one the knowledge on the site has been reused over 51 billion times give examples people visiting learning um and then other systems that are engaged or integrated with our products as well this is the foundation of the relationships that we're creating this is the foundation of the relationships that um are driving what we're doing with Google and with other partners this Foundation is important to us not necessarily purely for the data but for the overall capabilities that we can bring to play by leveraging that data so you think of it as we have our community it provides up-to-date information that signals around the value of the knowledge um provides context and it provides content for the llms to consume right our community then interacts with Gemini um and Gemini provides value back to that Community by helping us edit questions by helping us um find new areas that we would want to create questions in there's a whole bunch of different things that we're we're working through here as our overall mission in engaging with AI but the most important factor is that the AI capabilities we put into place are going to help improve interaction in the community help make it easier to interact in the community and help Drive value to the community and that's what this this engagement around data is is giving us we believe in the end the community can be driven to be something that is a collaboration something is a a capability of us working working together with humans and AI learning from each other and helping each other to learn and that learning is is the center of what we're building as part of the stack Overflow Community with that I'll open it up for questions well um while the staff is helping to grab the mic and I just quickly share a few uh QR codes in case you have your phone with with you uh feel free to pull it out on the left hand side is a QR code that we actually have quite a few product engineering teams and researchers are eager to speak with you to get your feedback on how do you see this relationship goes what do you like to see uh you know Cloud uh Gemini Cloud assist can help you in your specific use cases on the right hand side in case you're too busy or have other team member you want to invite you can use that QR code to talk to us uh VI remote or virtual uh virtual meeting and uh before we go to question uh there's also a eBook uh comes with the benefit of this uh conference it's authored by our chief evangelist officer Richard schroer feel free to scan the QR code to download that book questions awesome yeah got one down here thank you for the great presentation I think this is a good way of creating a more collaborative work between llm um or J providers and data providers so I think that's a really good approach I was curious if you're using um on updated stack Overflow data uh fine-tuning on a daily basis or if you rather use an rag approach for the incremental updates yeah that that's I hope people can hear the question I don't need to repeat the question right yeah um so uh currently we're focusing on leveraging stack or full data to help us improve response qualities through use of rag um and we're not using the stacker full data for fine-tuning yet uh we find the rag data is a lot more um uh r as a l more uh allow us to iterate faster uh we do have plan to uh continuously update the source data that we receive from stack R team on a regular basis yeah all right we got right here and right there um as a college students I noticed that most of the questions that I have from where I learn are marked as duplicate when I find like I find the exact problem I have it's Mark said duplicate and then poof it nothing more do you think that they could you could use uh AI as to find a solution to that so as part of the staging ground capability which is definitely a target for us is the idea of also being able to reference out these are other questions that look a lot like yours right and so you should be able to either look at these or or or reference these but then also if your question really isn't a duplicate we can help you craft the question in a way that catches the right responses puts the right tags there and so forth definitely a primary target for us um new users on the site college students what we're really looking to do is is catch developers at the beginning of their careers and help them them grow as part of the stack Overflow community so very much something that's that we're focusing on in staging ground hi uh Adam with well Sky question the stack Overflow data you reference here I'm assuming is from the stock overflow public side MH yes that's the case what's is their future plan on harvest the data in the W um stack Overflow teams the private data So currently our customer customers on teams can access their own data through our apis and so they can pull out their own data and we do have plenty of customers who are using that data to train their own models um internally we are also looking at making it available for people to for internal use only access our public site data as a way of training their own internal models as well but if you are a teams customer today you have access through the apis for that data we'll probably get to bring some new capabilities in to be able to get access to the data in an easier way um possible Direct direct access to the database or something along those lines for training or a data file extract um we are looking at options there and this is something that we want to expand upon as part of our product suite for sure question hey um my name is aiz so you mentioned attribution to users for their post um compensation to users for the knowledge they provide to LMS is a Hot Topic right now I'm curious to know what your perspective is on that are are there any circumstances where I think it makes sense to compensate users um for like the post that they put on stack Overflow and LMS get trained on there's that's a great question I think our primary target is investment in the capabilities that are there for the users um also bringing back capabilities that uh we believe to be valuable to help people grow their careers and really focusing on that developer and technologist Persona to provide more and more value based on the data that is is uh being provided as part of the community so the intent here is investment um if we get to the point where it's past investment then maybe but I think the the primary focus here is is on building out more capabilities for our users and growing the value of the community um based on the data hi thank you for this um the stack Overflow strategy that you've described seems predicated on the assumption that humans are better answerers of coding questions than AIS are which seems true but temporary how would the stack Overflow strategy change if in the next year or two get better than humans at answering the questions yeah there's there's definitely a there's an optimistic and a pessimistic view I think the current media is very much on the optimistic view side um and pessimistic from it's going to you know all the bad things but uh um I think that we we're taking multiple views on this and and combining them into our strategy we do believe that the community as it is has to evolve to integrate and work with AI capabilities and both sides need to be able to learn from each other right and so as we evolve um as a community I believe that those answers are going to get better better and better and that value is going to go up but we still have to actually learn from those answers we actually have to be able to dig into that answer and say how do I know this how do I learn this and how do I take this knowledge and and grow from it not just get a quick answer to my question right and so I believe that Knowledge Management itself is going to be key to that and there's always going to be the case where um we don't get the right answer in the first place even if it's if it's a human or if it's a if it's an llm and at that point we need to be able to then embark on that journey of learning um and I would actually say that even if we do get the right answer in the first place lots of times we should still continue to embark on that journey of learning and we can provide that right if you don't mind I can also add a little bit on the from a Google's perspective right so I think what we see or gen technology really is a empowerment to human potentials were a tool to help human to do things faster more efficient to uplevel human capacity or human potential right so I think there's always a role for human to play uh even with a continuously Advanced gen Technologies I mean if you look at latest numbers uh rgm 1.5 has already exceeded the average uh sort of a there's a there's a benchmark test but that doesn't mean that it's going to replace I think there's always a role for human to play of how do we guide how do we continue larage this to to uh increase our cre creativities and Innovations over here um yeah I had a question from Ryan and uh thanks so much first for your uh insightful um presentation uh I was wondering uh from the time that geni uh platforms like open Ai and and Gemini came uh for Public Access uh what's the trend of um new content addition to uh a sack overflow like how much of the coding issues and problems people go to these chat was and how much do they really come back to Sag overflow and what do you what do you think the trend with um reveal itself in the future yeah I think there's been think there's been a couple of public posts on this one um overall people tracking our traffic and so forth so I don't think this is uh you know I don't think there's any need for for uh holding back these numbers overall but essentially I think the initial drop was in the 30 to 40% range um but what we saw was a pretty quick uptick back from there as people saw you know the initial hype cycle went back and I think the overall drop right now if I remember correctly is about 2 20 to 25% oh oh 15 sorry 15% I got my team down here give me the right numbers that's good don't quote that number um and and I I really think that that is something that overall I think the primary challenge that we need to focus on is making it easier to interact with our community it's not it's not competing with llms it's really working with llms to bring people back to the community and drive them back and when you see what what we're doing in Gemini here that little link there is a big thing right that little link is is a way of having everyone who wants to learn more come back to the stat community and then engage um and do be able to do that as part of their IDE um not leaving their primary screen and be able to come back and actually engage with the community right these are Partnerships that we're looking at across the board because what we want to do is is stay in that first screen with people and have them come from their IDs to us um and engage with the community in their natural environment while engaging with the l M that is providing them answers so hi um you talked about the partnership in Google Cloud assist is that a a plan for the code assist as well U so good question the question is about whether or not the uh partnership would benefit the code assist product um so the data that we how we use Stager data is to put into our common rack um retri augmented uh uh Generations so that's actually a common module that's shared for um all gamini products across Google Cloud uh eventually all those benefit will show up uh but it's not going to be in the first wave we're building out the product pipeline as we go so just stay tuned for future announcement but uh but the short answer is yes it will benefit all Jam products for Google Cloud yeah thank you hello I'm wondering uh what call be the main difference between Consulting information through sh Cloud the between Consulting directly from stack Overflow website I mean ER with your AI uh offering well so our offering is not intended to replace the Gemini offering so we aren't doing our own chat-based interface on our site and so if you would be engaging with the chat based capability in Gemini and then coming to US based on the answers um and engaging with the community on our site we're really focused on assisting capabilities around um people engaging with the community and uh what we find is that in the IDE or in the in the console for uh for Gemini um it's a great place to start but in our community we're try trying to stay away from actually um doing chat-based interfaces so hi thank you for the great presentation um one quick question how do you deal with deprecated data I'm just curious right because you know the versions of the software are changing you know many things are changing you got to need it's a you want to you want to roll as a product manager or engineer at uh at stack um so it's a it's a constant uh Focus for us it's something that we're we're now of course the site has the capabilities built in right as as questions change people can downbad answers um and as they evolve as those answers evolve there's new interfaces new you know new versions of products and so forth the new answers should float to the top but there are challenges with that right we are looking at better tagging the ability to show versions for um uh for questions so a question based on a version of a software interface or something along those lines to make it easier to then keep the historical record but start to generate new questions based on on new versions of products and so forth so it's definitely on our road map and something we're focused on solving that problem specifically all right oh we we almost had a hand back there but now I'm gonna all right all right if no more questions we definitely would appreciate your providing feedback to us if you could uh you know uh uh let us know how it goes for you and thank you again so much for coming to our session uh hopefully enjoyed the Potter part party tonight and have a rest have a great rest of your next conference [Applause] [Music] thanks