Remote Software Development

Key Takeaways

it was mid-february 2019 when i first heard word of a serious virus spreading through the world i put the entire data skeptic team on a temporary work from home schedule that turned permanent a few months later when we gave up our office lease oh yeah this season we're calling it physically distributed the pandemic and its demand for people to work remotely was such a major stair step change that a generation of researchers will be annotating this on their time series plots explaining away a large change in the data generative process with the simple footnote reading covid19 the supply chain game has changed forever as has education and it's quite likely your industry your field has evolved a more physically distributed approach to its human capital this season is an exploration of the data and practices for remote education remote collaboration and an overall more distributed sometimes asynchronous way in which people are finding success working together without physical proximity there are now two years of longitudinal data to be studied surely researchers have interesting insights about this change if we call it that in how we work i spend most of my time writing code and i benefit greatly from the online communities i was already a part of for learning things or getting advice it seems those places are going to be even more vital to me now that there's less water cooler talk about how to fix this or that exception so that's where we're starting this season i connected with danae ford robinson who's going to share her research in the human computer interaction field specifically around these digital online communities hi i'm danae ford robinson a senior researcher at microsoft research and an affiliate assistant professor at the university of washington i study software engineering and human computer interaction mostly focusing on online programming communities or where developers or technologists share their identity and their technical knowledge the intersection of hcl and software engineering is super interesting to me for listeners that don't know the space well could you share what are some of the outcomes or research questions or areas that you look into at the intersection of hci which is human computer interaction and software engineering i'm mostly focusing on how people engage in software engineering systems mostly how developers or programmers as i like to call them because everyone may not consider themselves a software developer are building systems understanding how their identity is deployed in these systems how does that impact how they work how they way find so a lot of how humans use technological systems so when i'm on a larger software team i might find myself in a couple such systems you know source control like github maybe i'm on slack or teams i'm in a ticketing tool like jira and discord my identity seems kind of smeared across all of these digital spaces how do you look at it yeah smeared is an interesting word to describe it so what i'm most looking at previously to the study we're talking about today is looking at how developers across the gender spectrum the racial spectrum the geographic distribution how they engage in online communities such as stack overflow and github so stackoverflow is particularly interesting because it is a platform that immersion developers or even expert developers from across experience levels ask questions answer questions and give their knowledge to help others so that's what's really interesting there is how people can be vulnerable in ways that they ask questions and also be helpful in ways they support others so people helping others is really interesting to me i find that threat is common throughout all of my research i also work in other open source communities you mentioned version control so i study how software developers and emerging programmers engage in open source software where developers can build systems that can be deployed and anyone can collaborate on them together and also evolve them over time why have personally benefited so much from stack overflow in my life most of the time from questions somebody already asked but several times from questions i got to pose are there any trends you've observed or insights in the engagement you know are there groups that are not engaging as much on average that maybe need encouragement or what are the general statistics tell you a long time ago i think once that my advisor kind of threw out there was about on stock overflow about 65 percent of the people answer five percent of the questions so it was a small this is like three 2010 but this is a small population of people who are answering questions on the platform so experts users who are engaging there from the annual stackoverflow developer survey that they've been launching for a couple years they're seeing a range between 5.8 and i think last time's below nine percent engaging of women on these platforms which is also kind of disturbing because when we see in offline settings and software developers women mostly make up about 18 to 20 percent of the setting so it's really interesting to see that on the internet where you can be anyone you're also still seeing a decline in the gender distribution of who's engaging in these platforms yeah it's like you're double underrepresented first of all in the profession and then even more so online is strange and the point that is really interesting there is that when you're engaging in these platforms like stack overflow you're seeking help you're using psychoflow as a resource well if you're not using the main popular resource that majority of the developer community is using then where are you going for help how are you getting unstuck what resources are you using to stay within this field of software development so there's a lot of broader questions there about the tools people can use the resources people feel comfortable using to maintain their career in software development there were an easy fix all like you know we just need to make the font larger we would have figured this out by now it's obviously a complex issue how does one approach it from a scholarly research point of view one way that i've actually been able to investigate this platform so i've done some research investigating how the barriers what are the barriers that people have to engage on this platform started off looking at women and identified about 14 barriers and then from there identifying that these barriers that we identified for women had a broader impact as in they also impacted men and people across the gender spectrum turns out they also impacted novices right so a lot of these marginalized groups and these platforms have had challenges engaging but it's interesting to see that it's not just the marginalized people who are impacted here i think the most interesting part about this is that using that set of 14 barriers as a framework we've been able to understand how you can flip the bit of each barrier so something that was negative figuring out how to improve them by figuring out how to reframe this fear of failure for example or the sphere of understanding how to engage in a large community how do you make the community feel smaller and we use techniques like a online mentorship program so sakofu's first mentorship program i was collaborating with them on that to figure out how to increase novice participation and what we ended up seeing is that we were able to increase the score of novices by who participated in this program by 50 on average so i should also take time to explain what the mentorship program looked like so on stack overflow platform we were able to take advantage of the chat feature so if you go to stackoverflow.com and you see that there's sub chats for each type of programming language topics that developers can just spin up and they're essentially chat rooms synchronous chat rooms we use those instances or four instances of those to figure out how we can get expert users and novices in the same room to guide each other to help them craft better questions so that they can feel like they're getting more out of the platform so i think that there is a value there in understanding how sub-communities or creating safe spaces or helping people understand how to maximize the value of communities can also help them feel more comfortable being members of the software development community at large but also wanting to give back later on as other mentors what sounds like a great program and pretty impactful then if that's improved their scores by 50 do you know what any next steps are there are there ways to grow it so the next step that i've been really considering now is figuring out how it looks like on other platforms like github for example which actually had a large growth let's see since they had github discussions which is something that is a another version of a asynchronous chat that was launched about maybe a year and a half ago where on github discussions you can ask questions and anyone can ask questions which is separate from issues but it's really geared at how do you help onboard people to communities how do you create a space where you can ask the quote-unquote silly questions that may not belong in issues because it's not a bug but it's really about how do i engage in this community where are the contributing guidelines how do i get best in contact with a maintainer figuring out where those types of conversations are happening and allowing them to be more accessible to the first time contributor good advice all around well these communities have existed for a while i've benefited from things like github and stack overflow for do you want to say at least a decade now which means we have a longitudinal data set that covers both before and after the natural experiment we've all been a part of with the oncoming pandemic and the switch to remote work i was very happy to find the survey and many of your other papers and work in this area but starting from that point i was wondering if we could dig into a tale of two cities software developers working from home during the covet 19 pandemic a topic i'm sure many listeners are first-hand familiar with so we all bring our own experience to the table to color it but it's interesting to see it in survey form and learn a little bit more about some of the trends before we get into those could you tell me a little bit about what the goals were what did you want to learn when you set out to do the study so pre-covered so before january of 2020 even before december a couple of my collaborators and i were investigating how remote work or again engaging online communities can really be a gateway for software developers so specifically we were looking at how remote work can be an advantage for trans developers and figuring out how they can foster communities that empower them to find flexibility of economically stable work find that being a software developer remotely can also give them autonomy to disengage and re-engage and also have control over their identity i want to also set the stage for this pre-covered remote work because there has been an incline in the amount of programmers who were able to work from home before the pandemic versus other careers like in art design and media personal care and business finance so i'm one to set the stage that software development has already been in this globally distributed format especially when you think of how people contribute to open source where people are collaborating across time zones there are people who have been contributing our team members who have never met each other in person so i think developers are a great case study for understanding how and what remote work or hybrid work can look like successfully before the pandemic and give us a nice template for what we can look for after the pandemic so going back to the tale of two cities paper the goal was to really understand what are some of the challenges the benefits and even some of the improvements room for improvement that developers have had during the pandemic but i also want to set the stage that remote work is not the same as pandemic work right there are several other factors at play for the most part to date i mean before march 2020 we've been talking about remote work as this own separate entity of like you have work siloed and everything else is kind of does not impact that pandemic work is not that pandemic work is people are working remotely they're also managing their families at home their well-being the well-being of their family the environment in previous remote work settings people were able to go to coffee shops for example you couldn't do that in the pandemic you know your physical work setting looked a lot different and it impacted a lot of how you worked so through the tale of two cities paper we went to gain a little bit more insight into what that looked like what are the new benefits what are the new challenges people are facing and how have software engineers their self-reported productivity how has that changed since working from home well let's get into productivity first maybe that's the one i think a lot of developers get so central to because we have good metrics on our work right we really track what we do and we want to feel like we're progressing and learning all the time was that possible for people during this as you say not just remote time but remote plus pandemic time yes so i would say that overall we've seen this dichotomy of experiences where for the productivity change that we ask developers compared to working in office how is your productivity change this will be access them in our survey and we're seeing this 30 if you will have felt more productive about 30 percent have felt about the same and thirty percent felt less productive so again this time frame for the survey was about april to may of our second analysis in 2020 so this is a while ago however i have also read recent surveys that people have also seen that they felt a split still as well so i think it's also interesting to note that a lot of the things that we found early in the pandemic have continued and i wonder what they look like for hybrid work settings but we can cover that later so for the productivity changes we've seen that some of the benefits that folks have mentioned as well are they feel like their health has improved and they get more asleep they feel like they had more schedule flexibility and autonomy to determine what their schedule looked like they also felt like they were able to get more heads down focus work and have more control over their environment as i mentioned so their apartment feeling less jury than the actual physical office and also being able to spend many more moments with their family children and pets overall we saw that all the benefits that we identified did have a positive delta on the productivity change i could probably link the paper as well but there's a cool figure chart in our paper that identifies the survey benefits with respect to productivity and we saw things like this ability to spend less money flexible hours closer to family or all had there were significant differences there i think it's important to acknowledge this as well because we did a separate analysis understanding the cost of benefits with less time better focused time like how much time they're spending on deeper work but also their ability to have less distractions or interruptions so the fact that working from home was able to when people who did discuss that they had an increase in productivity they also had better focused time they sell also had less attractions interruptions on the opposing side some of the same benefits for some were also challenges for others so there were several participants who mentioned issues with connectivity and being able to not access the physical work devices if they were kind of remoting in also the same opportunities people had with spending more time with family children and pets could also be quite a challenge because if you needed to support and help your family or your child with their homework or their time to get them connected to their internet for their virtual learning in addition to your time to get connected for your work right we also saw a significant amount of challenges with communication figuring out a new set of communication standards and guidelines for how you were going to connect with your team and also other teams as well because a lot of developers work across teams of the challenges that i like technology the most are there was a significant difference in missing social interaction and we also seen this missing social interactions play a very important part in the team productivity as well which is a second survey that breakdown you'd mention that there's a split between people feeling more the same and less productive obviously the numbers may have evolved a little bit but if you're the same i guess you can't complain if you're more productive congratulations but for that significant slice of people that found themselves less productive and maybe are still in that boat have you learned any general attitudes or ideas about how to deal with that or make improvements yes so actually we kind of came up with the set of recommendations and improvements that we've worked with microsoft to kind of determine and they have also evolved over time one of the main ones is more of a charge to managers to really figure out how to best support their employees during this time so creating settings and having designated time between meetings for example to have effective breaks so there was a study that came out a couple months ago from microsoft where researchers were investigating how the science of breaks so they conducted a study with about 14 participants from microsoft and connected them to electroencephalograms or eegs to be able to understand how their brains interacted between breaks between meetings so breaks between meetings versus breaks outside of meetings breaks without meetings and they found that those who were able to have breaks between meetings so that five minute gap that happens between if you have a 25 minute meeting before your next one you can start to see that there is a less heightened amount of stress that happens between so you're able to transition better you're able to kind of ease that stress rather than going back to back means we have to pivot your mind quite a bit pivot your brain activity to think of different tasks and i can link that study as well [Music] thanks to our annual sponsor weights and biases the developer first ml platform weights and biases is an enterprise grade platform that gives you experiment tracking data set versioning and model management their solution integrates with tensorflow sklearn xgboost hugging face and literally every other framework i could think of integrating with your model training process takes just a few lines of code and there's even a nice live notebook on their site that'll show you how to get it going once your machine learning code base or notebook is wired up you've just unlocked the next level of productivity the weights and biases platform collects aggregates and beautifully displays all of your key metrics and telemetry so you can track your model training in real time and compare different runs have you ever had to screenshot a matplotlib craft that you'd drop in an email or in slack to show your boss the results of your work that's like serving a gourmet dinner in a cardboard box with weights and biases all of your collaborators can see the results of different runs with an idealized interactive suite of tools you no longer have to be the single source of truth for your whole organization about how the model's doing if you're on or leading a growing team of machine learning practitioners you need to investigate if weights and biases is the right solution for you get started at w and b dot me slash data skeptic and mention us when you request a demo thanks to our sponsor arctic wolf 2021 was a landmark year in cyber security the rapid piecemeal changes businesses made on the fly to infrastructure applications and access controls during 2020's huge shift to remote work were now under scrutiny while security teams were still busy defending against the litany of attacks and attack methods as the size scope and sophistication of attacks continued to grow many organizations spent the last year overwhelmed by perpetual change wondering when where and how things will ever improve in early 2022 arctic wolf conducted a survey of over 300 global i.t security decision makers to understand their priorities and anticipated challenges for the years ahead our goal is to understand their security roadmap for 2022 and to get a first hand perspective on their current objectives and potential obstacles looming on the horizon visit arcticwolf.com skeptic to find out what they learned that's arcticwolf.com skeptic that's a nice insight then and something that i almost wish we could just broadcast to all managers of the world but there's no such channel could we approach it from a software point of view so at microsoft we have the viva insights teams which is really centered around figuring out how to make people productive not just software developers how can you kind of give people small incentives to modify their calendar in order to create concrete boundaries of how people do this so i mentioned this five minute between meetings what we've seen a lot of and this is actually from a nature paper from the microsoft electronics and devices team that they put out they were looking at how people's calendars regardless of role as a software engineer or not and looking at the results from people pre-pandemic and post-pandemic they were able to see that people were in more meetings longer and there's another study that showed that there is actually another third peak of people meeting at or having a lot more work to do after they're nine to five from these findings they have been implementing this toggle you can add to your calendar where you can change the settings and set meetings to default to be 25 minutes or the traditional 30 minute meeting to be 25 minutes and then the traditional one hour meeting to be 50 minutes so by default when people create these blocks on your calendar or you create a meeting it will default give you that five minute gap again encourage people gently nudging them to have that time for themselves to recoup to get up and get a glass of water to get up and stand up and stretch for a little minute also on microsoft calendars we have things like the focus time that is default setup and i haven't used this myself but several of my colleagues have where from the viva insights team is able to extract what gaps you have in your calendar aside from your meetings and recommend to you that you set these up as focus time so you have deep time to focus on what you want to work on rather than engaging deeply in meetings which is also valuable but it's a different type of productivity that's happening mention some communication challenges that came up as well could you expand on what you learned from the respondents the communication challenges we were seeing were essential for new employees coming into microsoft there was also a set of team collaborations of people who were already ftes at part of microsoft for the new employees coming in the communication challenge that they were having is i'm new to this organization or new to this team i'm not quite sure about the tone the cadence the communication style of people on the team so there was a lot of that went unsaid if you will that is quite a challenge for people who are trying to get the groups really figure out how the team interacts one solution that some have come up with is setting the set of communication standards or guidelines for their team and that was one of our strong recommendations that we find from participants but we also is a great recommendation for how to be effective in your hybrid settings so in creating these communication standards for example you can define what your working hours will be for example like as a team are we going to accept that wednesday afternoons or no meeting wednesdays so we shouldn't expect to hear fast responses from people on wednesday afternoons versus okay well mondays are going to be from 9 to 12 we're going to expect more people to be available to answer synchronous questions on teams for example having those set of standards expectations of what you as a team member would expect from other people as well as what you're setting for yourself gives each other a common language it also allows the manager to understand what is the best way to interact with their employees and give their employees trust you know give their ics trust likewise some of the other recommendations for improving team productivity on the communication side was managers holding weekly one-on-one meetings and keeping hold of their regular team meetings at whatever cadence made the most sense for their team because again although we are working deeply there's also the chance of that isolation that happens that a team member can go off and work on something that they needed to touch base and they didn't have the opportunity to do that so when you're having those regular team meetings basically your scrums for example you're able to really connect back and see how the work that you're doing can impact others on your team which was a example of a communication loss during the pandemic earlier we touched on those missing social interactions and maybe that managers could create settings for those to exist those will have to be done in some new digital way and it's not that we don't have tools for that we can obviously send bits back and forth to each other but it still somehow doesn't for many people myself included feel like it has parody to the real world what's missing and will we ever achieve that i can tell you a little bit what people have done before and then maybe we need to figure out what that transition looks like for hybrid settings so for before some of the early recommendations we had was including social activities so a lot of people were trying to figure out different ways to engage their team however a lot of people were doing those outside of their set working hours so if you're working 95 they wouldn't have to be hour five well if happy hours at five i'm missing that important team bonding time well i had to pick up my kids from daycare you know which is daycare being my neighbor's house next door right that's the time i started interacting with my family so by default when we set some of those social activities outside of the work time some people got excluded from that so not only were they missing that from their core nine to five they also missed out on the bonding time which is essential to building that team rapport so one of our core recommendations was including the social activities as part of work and also thinking beyond what this happy hour looks like happy hour is fun all but everyone doesn't drink and there's other ways to actually have fun there's different painting things you can do right there's the paint classes there's the cooking classes on airbnb for example which allow you to virtually travel to a place the most exciting recommendation that we got from others was kind of having those social activities be a part of the core work time and then actively in doing so you start to think about what are the ways i can actively be inclusive to my team what can i do to maintain team culture in a way that's open and receptive for everyone so everyone feel like they can be a part of the team but now think about what it looks like through hybrid settings which is i think essentially your question is a whole another like beast if you will because in hybrid settings some folks may be in office physically in in the facility and then others may be remote so what are our meeting rooms set up how are they set up in order to allow us to do that effectively so that everyone still has the voice when we were all remote everyone could raise their hand virtually to engage what does that look like in in person in facility settings when also one person or three people are remote i know there's been work at microsoft looking at how we can incorporate that back into our hybrid meeting settings and there's people like shawn mintel and abigail selin at microsoft research who are looking at that however the biggest hybrid encouragement people are giving is give each other grace as we figure this out because everyone's not going to be it's not going to be an even split between who's remote and who's in person so when we have these social activities as part of a work setting maybe we should have all those people are in office have them bring their laptops to the room right so they can also be able to have that you see their face during meetings and they can also feel like they're on a level or similar platform as others because everyone's going to have a different expectation of what hybrid looks like and everyone's not going to have the same type of hybrid lifestyle as we may have appeared to have sometimes in the remote work setting so we need to give each other grace in those cases well i've heard various reportings about stats on which companies are going back or going to be hybrid or what the future looks like do you have any sense since you've been doing survey work and thinking from the user's perspective what does nothing in need workers typical but what does the typical worker want or are there trends you see there and people having a desire to go back and be hybrid keep things the way they are or whatever the case may be i would say that there is quite a mix and this has actually been kind of highlighted in some of like the hybrid work material so microsoft for example will be moving to stage six quite soon they've announced that i think that's been kind of highlighted in the news what we set up even pre before we arrived at stage six was a system a framework for talking to your managers to figure out what does hybrid work look like on an individual basis because hybrid could be a hybrid worker who's working from home some days or working from the office in some it could be a hybrid team where some people are permanently remote and some are physically in the facility and that threshold of time looked different for everyone so during the pandemic workers were thankful for the flexibility right now as folks move towards stage six which is starting to come back to office slightly they are still seeking that flexibility they've had that autonomy that has been really great for the morale and being a member of the team and figuring out what their new work setting looks like they are trying to figure out what it looks like in the hybrid setting post pandemic if we ever get to that full stage so i would say that workers are looking for a flexibility that is dependent upon what works for them at the individual level there is no one size fit all then that's that's what researchers have found there is no one-size-fits-all for hybrid and flexible work environments for folks it's going to be up to managers and their organizations to be able to be responsive and supportive of what that hybrid setting looks like for folks well if i were an employer looking at these and trying to make some decisions one of the areas i'd zoom in on were those benefits you highlighted earlier and even if i was ebenezer scrooge you're only concerned with the bottom line when i hear there's more deep work i know that's good when i hear the employees are healthier and getting more sleep i know that's good maybe their flexibility in schedule doesn't suit me if i want these rigid meetings but two pluses and a minus it seems like there's an overall net positive here is there any guidance you would give to employers about how to better structure their organizations as they're thinking through these issues the best guidance i would give them is to and maybe this is my call to action is to conduct your own surveys with your employees because they'll tell you what they want figuring out how to best support them in hybrid settings asking them how often they feel like they maybe want to come into the office offering them opportunities to have hybrid settings because that was something that i think a lot of folks and something we learned from open source communities who have been doing this hybrid remote work for a while now right where there was a remote work set up that folks were empowered to use and take advantage of so that they could be effective at home or effective wherever they're working from organizations like github and stock overflow these companies that i've mentioned earlier in the talk their employees were already remote first pre-pandemic they already had different advantages of special cameras they used or a nice hybrid setting at home where they had a chair that was ergonomically sound so what does that look like for the range of employees a larger group of employees and the only way to know that is to ask them and i think that was one of the advantages of our study and where we got so much great feedback from other survey responders taking our survey like thank you for asking right they were like we definitely got some one-off emails that were just like i appreciate you all inquiring about this and sharing this up because at the time not that many people had asked this specific question and i think that employees would do the best work when they feel like they're heard and respected and they feel that they have a little bit more autonomy over their flexibility and their employers care about that it seems so obvious now that you've said it to ask the employee what they want great advice indeed similarly you uh had a follow-up survey i'm looking now at the how was your weekend software development teams working from home during covet 19 paper i'd love to actually get towards the regression in the learnings there but to start off is there anything we should highlight about this survey in contrast to the previous yeah so this survey the core findings that will highlight is that we were really focused on how teams work together or how people feel like the pandemic may have changed or shifted their relationship with their team how they collaborate so we had some findings from our first survey which is from our broader work from home how are your productivity changes how you plug your meeting milestones in our second survey which is the how was your weekend paper we found that 74 of these respondents again missed the social interactions with their colleagues 51 reported a decrease in their communication ease with colleagues as well and then from our regression model we found that the ability to brainstorm with colleagues the difficulty in communication and then also the satisfaction with those social activities were important factors to consider so again it was kind of interesting to see that the same factors we were identifying across we're still persistent in thinking about how people collaborate when we started diving deeper into that and can we talk about some of the results from the regression and the analysis what are some of the key factors that jump out from the data so first i mentioned we had the 608 responses we created this model that we looked at for all the team culture factors which reduced to our on a reduced model and we were looking at what explained team productivity so the factors that i want to highlight the most would be this idea of the less awareness of what colleagues are working on and this kind of results from how folks are communicating within their team so i mentioned a little bit earlier about if bob went to the scrum meeting but hasn't connected with the team for a whole week they haven't come back to the manager to the whole group to know that oh actually what you're working on has shifted quite a bit because they've been siloed right so there's been a really big challenge with people feeling siloed not just like emotionally isolated but their work not really being able to have the tie back to the latest project goals expectations and outcomes you can consider this if you were in person you're walking down the hall and you may have connected with someone from a team next door and they're talking about how their project may have shifted a bit that's that hallway water cooler conversation that keeps you also abreast of what's coming up down the pipeline and how your work can impact that people were missing that in order to schedule that on teams for example you had to actually knew who exactly was working on that on the other team catch them at the right time schedule a meeting awkwardly come to this team's call that was previously just like a walk past them in a hallway conversation so a lot more friction was there that inhibited how people connected so that's some of the friction that being remote did cause also one of the other factors i want to highlight is this impactful contributions to the team so as the individual it's hard to know how impactful your work has been to the broader goals of the group because not only are you not able to have those side conversations with other group members where they're able to say you know hey you know that work you did was awesome you know it really helped me understand how to do xyz there was no assessment it was harder to assess how your small part of the project was able to help and really be a pivotal role to the broader goal so other factors that we were kind of having my second group here were again the external team bonding activities the ability to make decisions as a team the start and end of meetings of non-work talk so that beginning transition of meetings where you're asking people how their days are or you talk about the latest show you were watching those kind of help the transition and those little mini opportunities to build that team bonding were struggling those were challenges or great examples of this difficulty communicating amongst each other is there anything you can tease us from your current research or stuff that's coming down the pipe pre-pandemic as i mentioned a little bit i was working a lot on how who was engaging open source communities who was engaging on these online communities that programmers were asking for help giving help supporting one another lately i've been thinking about how people trust these communities how they find trust in different communities and the systems that they're using and where they feel comfortable asking those tough questions lately it's been questions about how different ai assisted agents can be supporting developers so i'm currently looking into projects that can help us understand and build metrics around what trust looks like in some of these systems the same way people have been really skeptical or engaging in trust in how they ask general questions about like linked lists and loops on stack overflow but what does that look like and what forms are people using to ask that about ai assisted agents so that's quite a little bit of a different tangent but it fits into the broader work that i'm working on is what is the next generation of developer communities who is in the next generation of developer communities and how they can be and continue to be a resource for others where can listeners follow that work or anything else you're doing online so online you can follow my research my website which is danae4.me danae ford being my pen name all my research should be published under that name i'm also on twitter at danaeforwardrobin and a lot of the research blogs will be posted on the microsoft site so i'm looking forward to getting connected with folks if you're interested in any of the recent research projects i mentioned i'm happy to stay in touch lenae thank you so much for coming on data skeptic and sharing your work thank you so much for having me [Music] that concludes another episode of physically distributed thanks to our sponsors weights and biases and arctic wolf myself claudia armbruster as associate producer vanessa bly guest coordinator david bembe show notes and our host kyle pulich [Music] so you