Inclusive Study Group Formation at Scale

Key Takeaways

thanks to those that responded if you want your voice to be heard at survey.data skeptic.com this week I'm asking about chat GPT but my first question in the last week's survey that we're going to share the results for now which statement best describes your belief about the emergence of artificial general intelligence eight percent of you think AGI is impossible and will never occur in what I assume is a coincidence eight percent also believe that AGI has already been created but everybody else picked an option that had a year in it so let's break that down in fact let's take this in order I offered the years 2023 24 and 25 individually nobody picked AGI being created in 2023 which I guess implies that the people who thought it has already been created think it was created before January 1st but anyway five percent said twenty twenty four or Twenty twenty five an aggressive timeline I then did some ranges how many of you think AGI will be created in 2026 to 2029 kind of a weird range but I wanted to get onto the decade system that's a surprising 24 of you but not yet our most popular answer know the most popular answer came next a whopping 33 percent of you reported that the best statement that reflects your beliefs is that AGI will be created in the 2030s and I'm going to group the rest together the remaining 24 give or take percentage of respondents believe AI will be created in 2040 or later that includes later than the year 2200 which a little under three percent of you picked gosh I wish we had done this a year ago or two years ago I'll do it in a year's time and see how it Compares actually I'm interested to do it in 2035 and see where we're at since the most popular answer is that we're going to be speaking to artificial general intelligence in the next decade I forget what I voted because my view on this is evolving my research why does my views have evolved quickly on recently it's education I don't think anyone would say that any class is perfect I guess I care a lot about in my mind the Improvement is doing a more complete or more efficient or in some way better job of transferring information into the students study groups because I was looking tutors using surveys so I spend a lot of time to gauge the success of this project in the methodology we're going to talk about this scale relatively simple feel included and welcome and meet everyone where they're at definitely I'm curious is there a long history of scholarship on this it seems like it could be a new field that even though it's been there the whole time how to optimize teaching yeah I think optimizing teaching is something that in terms of scholarship there's a long history of work I mean at Berkeley we have an entire Graduate School of Education you know there's lots of different theories of how a student learns we haven't had classes at the scale at which we're teaching until very recently and I think that there are a lot of innovations that we're doing but I don't want to claim that we're the only people doing these kinds of things I think Berkeley is very is different in that it's one of the first universities to have these thousand plus student classes we've definitely been innovating in terms of trying to get the scale to work out in terms of the history of scholarship I'm not sure that there's been a lot of work studying how to optimize thousands of student classrooms yeah I there there might be but I guess my my training and my PhD and all of my research so far has been largely in you know information Theory theoretical control machine learning so that's kind of the focus of my research and I've gotten into education and teaching more as a practitioner in the sense that well as part of my job as a professor I teach I care about my students and so I want to do right by my students hence I think about how I should teach best so in some sense I'm coming at this very much as an engineer and then there's some research questions that fall out of it what we see is that you know students coming in from high school or maybe they're transferring to UC Berkeley from a community college they're extremely excited and you know eager to learn but Berkeley has a tough curriculum and we find that sometimes it's shocking to students and we see students coming to us from the smallest Rural High School in California as well as from very established high schools in say Silicon Valley people come to us from a wide range of backgrounds and when you have a thousand students you have extremely diverse backgrounds and personalities and level of preparation and one of the key questions is how can you in such a diverse setting make sure every single student can find something that's new where they can grow and no student should feel like they're out of place everyone should feel like like they belong so it's a lot of layered teaching and trying to think of intentional interventions that can level the playing field for students who are coming in from such vastly different places well if you are one instructor with a thousand students I don't think it's mathematically possible to give each of them a dedicated hour of your time it's not so what are some of the mechanisms that we can use to help promote things in the course study groups are a huge focus of what I've been doing recently of course we have a lot of teaching assistants you know we have training intensive training for the teaching assistants on how to teach at this scale but study groups is kind of a new direction that I that I moved in quite recently in the last few years to try and get personalized experiences for students in the classroom study group project really started from the experiences of of some of my students so I had a student Gloria who was actually a co-author on this on our paper who told me her story of struggling to find study groups and and people to work with and when I started looking into things I actually found a news article from 2013 believe it or not so 10 years ago where I can read this quote out to you the then director of the African-American student development office said a black student might be in a science course and the professor says okay everybody has to have a study group nobody picks them for a study group they first have to show they can get an A before they get selected and then we had actually another student write an open letter to the Department about their experience this was the president of the black graduate engineering Student Association at Berkeley and they said I've stood by black computer science undergraduates who are constantly ostracized in group projects often the last ones to be picked their peers regularly perceive them to be less knowledgeable and capable in carrying coding projects I've had to console crying teenagers scarred from The Experience questioning whether being black was worth it imagine that their only path to recognition in that environment was through a bogus burden of Excellence whatever happened to the opportunity of being average yet respected and then the pandemic hit and we had to Pivot from you know having all of these thousand students in one room where at least you had the opportunity of sitting next to someone to all of these thousand students being in their own homes uh not really being able to connect with anyone it felt absolutely imperative that we try to get people together in some kind of structured way and so I think about our kind of we call it our automated inclusive study group at scale project I guess our goal is that it should take no overhead for a student to find a high quality study group they should be able to you know meet a group try it out if it works for them great if it doesn't work for them you know find a different study group this is not a hard problem to solve in the context in in software in some sense like we have a lot of different ways in which we can think about matching students together one of the easiest things you could have done is just randomly assign students at the role of a dice to groups that would have you know met a version of the goal and so maybe that's our control group or something like that what sort of improvements might we want above that students schedules are extremely complex and so in our matching system we went a different route from random assignment even though I can talk more about random assignment and I think that there is different ways that you can in which you can make that succeed in our system we decided that if you cannot meet each other then clearly this is a failed Endeavor and we had students all around the world we had students in India in China in Europe you know in all time zones in the US so we have students first and foremost what time zone are you in then we asked them to tell us what their course schedule is when do they want to study when do they want to meet up so we said do you want to meet on Monday mornings Monday afternoons Monday evenings and we tried to group people together based on their schedules in addition we asked them some simple questions about how much do you care about this course we wanted to have people who were really excited about the course they were really gunning for that a to be able to work together and then you know there's always people who don't necessarily feel excited about the course material and are taking the course because it's a required course and we didn't want to necessarily create friction in the in the study group because the people in the study group had differing goals and so we wanted to be able to create groups where everyone wanted the same thing and finally we considered what we call Singleton avoidance um a single 10 in a group is someone who is the only representative of say their particular demographic group maybe race maybe gender maybe other identity that might not even be obvious and there's a lot of literature that says that people who are Singleton groups experience more stress have lower performance and what we wanted to do is we wanted to make sure that we didn't do this to our students and so we explicitly created groups such that for example you would never have a group with one woman and four other or five other men in the group you would always have pairs of people so you could have two women and two men we could do the same for people's racial backgrounds and racial identities this is something that that really benefits from the fact that we are teaching courses at the scale of a thousand students because one percent of a thousand is ten whereas one percent of 100 is one right so you know even if you have an identity that is one percent of your entire class you can still find Partners absolutely and so that's something that I find very exciting about this system all right everybody today's podcast was supported by the University of San Francisco and their new Masters in applied economics degree so if you're considering grad school and you're interested in data science let me tell you a little bit more about why an applied economics degree could be the way to go in the new digital economy everything is about the platform as you may know I wasn't an economics major myself but I did take a lot of econ classes and I gained a strong appreciation for econometrics and that benefited me greatly in my career for example my understanding of the Vickery auction helped me to work in search engine marketing since at least at the time that was the primary auction mechanism of the platforms and as the digital economy grows and evolves I've been excited to participate in projects related to tracking of reputation online experience and causal inference and those are just some of the tools businesses are using to make decisions today not to be outdone the University of San Francisco's new Masters in applied economics degree is also going to teach you machine learning using R and python this is a stem designated program which I appreciate very much you can get an application fee waiver by visiting this link usfca.edu skeptic once more from the University of San Francisco and California usfca.edu skeptic thanks to today's sponsor notion.ai listeners the team and I over a data skeptic we love notion our sponsor today we use it every day for organizing the show and today I'm excited to tell you about an incredible new addition to the notion Suite of tools notion AI notion AI helps you work faster write better and think bigger doing tasks that normally take you hours in just seconds and that is totally true I've asked it to make to-do lists for me to break down problems into steps to rewrite an email in a more professional way you just tell notion AI what to do the more details the better or you can start a prompt and go from there have it write a blog post make an outline brainstorm ideas the options are endless and for a limited time you can try notion AI for free when you go to notion.com data skeptic that's all lowercase notion.com data skeptic there you can try out the incredible power of notion AI today and when you use our link you're supporting the show so please use this limited time offer try notion AI you can try it for free at notion.com data skeptic so I can see how you would gather data from students about their you know availability and identity and things like that and how they want to be matched and we've talked about some of the virtuous things you want to do like eliminate Singletons and things like that in these groups what's the algorithm that gets that all that work done so our basic algorithm is a partitioning algorithm it's very very simple uh we basically partition people according to First their time zone then their schedules and then within each of those kind of if you think of this as a tree structure that you know how like you have a big initial group of everyone in the class and you say everyone who's in the Pacific time zone is in one partition within that partition I put all the people who want to study Monday morning in the partition so on and so forth now that I've identified people who want to study at the same times I now go through and find what we call partners for students so we basically you know think about tying together students who share racial and gender identities so that we can make sure that every group does not have any Singletons of course in this case you might have you know there's only one student who identifies with a particular group only one woman wants to study on Tuesday afternoons I don't know right so what do you do in this case so in this case what we try to do is we allow for a partition violation and we you know try to say well maybe we will put the student with another group just in this case to make sure that they might be able to find someone with their same identity and this usually worked out because students didn't always identify with just one scheduled time so they would say that I can work Tuesday afternoon or Wednesday morning or Friday afternoon like they would give different options and we would move them around but of course there were cases where you know it it simply didn't work out and so we had maybe very few groups with some kind of Singleton presence because we decided to prioritize scheduling if you can't actually meet up then you know the fundamental of the idea of the study group is just gone and in terms of Engagement I I don't want to take for granted that every student wanted a study group maybe there's some circumstances people had chose to or needed to just be independent but what sort of Engagement did you see yeah that's a great point so this system was entirely opt-in the ones was required we had exactly 477 students who consented to our research study out of about a thousand but the engagement was much higher than that we had more people participating than consented to participate in the surveys and can you tell me a little bit about the surveys what did participants have to give you as feedback participants had multiple surveys that they filled out they filled out basically first their preference survey tried we tried to make that extremely lightweight and we just made it a small homework assignments on their first homework they just got through the surveyed um and it was just a little homework question with a couple of these few questions and then we asked them for a midpoint evaluation and a final evaluation both of which asked the following series of questions we asked them about their comfort asking questions in the group and ask them if they agreed or strongly agreed disagreed or strongly disagreed we asked them about their comfort sharing ideas with their group mates we asked them about how often the group actually met we asked them about how frequently students participated so by that I mean how engaged students were in the group did most students participate did only a few do all of the talking and the others were mostly passive listeners and then we asked them hey do you want to continue the study group the next time you're taking a course would you consider working with these people again so these were our evaluation surveys and we also had what we called reassignment surveys and reassignment surveys were for the express purpose of asking students hey is this study group working for you two weeks after the first assignment was sent out we asked students if they enjoyed the group or if it was working for them or for whatever reason they kind of didn't have a a positive experience they could fill out the reassignment survey which was very similar to their initial preference survey except with one extra question of what was going wrong in the previous study group and then we would rematch them with a new group so that they got multiple opportunities to meet various different groups of people and do you have a sense of you know in summary what some of those reasons were it's quite shocking we thought that there could be a wide variety of reasons but the largest reason people ask for reassignment was that their schedules didn't work out with the new group you know even though we had prioritized scheduling scheduling was number one and number two reason was that people ended up in an unresponsive group so they would say that I sent an email to my group members but no one actually replied and I think this is because student schedules just change and student priorities change and the first week of semester you're very excited you want to meet a lot of new people you want a study group but maybe two weeks in you know you actually found out that there's someone down the hall in your dorm that is in the same class and you don't need a study group anymore sure so actually a big focus of our future work is to try to think about how to address it if we can come to that and then you can ask us students I guess in like an exit survey about their enjoyment can you talk about that results in any other ways you have a measuring success in terms of results on the survey what we or it results about our our intervention one of the most interesting things was that women and students from underrepresented racial groups opted into the system at a higher rate than those from majority groups the people that we thought needed help forming study groups actually do need help forming studies or at least they're excited about joining such a system so for example we had about 74 of women versus 67 percent of men opted into the system similarly we had 85 percent of black or African-American students opted into the system versus 67 of Asian students opted into the system and at Berkeley Asian students are actually one of our majority demographics that's I think is one very interesting result and then I think there's a bunch of interesting results in terms of how students actually told us that what students actually experienced in terms of their comfort in the groups as well as the interaction frequencies and we tried to figure out how do we evaluate how these groups did it's hard to do a true control experiment a true controlled experiment would say well I take my class I give some students study groups and I give other students no study group and I see how they experience the class that would be the true way of understanding how people did sure but that's clearly unethical if a student wants a study group like I want to give them a study group and so what we did to compare was we tried to understand how do students from majority demographics do as compared to students from non-majority demographics and so for example what is really cool is that we found no significant differences between men and women in the groups in terms of how they answered questions such as were you comfortable asking questions in the group how often did your group meet would you like to take future courses with your same group men and women answered quite similarly uh there was no statistically significant difference between their answers similarly if you compare racial demographics again for all of these questions we found that black students as well as latinx Hispanic students did not show any statistically significant differences from the two majority demographics that we compared against which is Asian and white students in our classes so I think that's something very interesting the last demographic group breakdown that we looked at was by student year and here we did find significant differences in fact we found that the freshmen did like just were happier about everything as compared to more senior students okay and I guess that's not that surprising we find that freshmen are often the most enthusiastic students in the classroom and therefore we're more likely to be enthusiastic while probably making friends it was their first year and they really appreciated the system one last thing that I could add in this in terms of our results was that we also tried to understand whether this impacted learning outcomes which I think is in some sense a Holy Grail like if you can make students learn better wow yeah and it's difficult of course to have a true control study but what we found was we found a correlation between those students who were in high comfort or high interaction groups with better performance on their final exam so of course I cannot find causation here because maybe students who were doing better in the class anyway were more comfortable in their groups but it's still something that is positive and makes me feel that the groups are a good thing to do well do you have any broad recommendations of rollout whether it be you know even at your own University or at other places how can some of the learnings here be applied elsewhere at even bigger scales so this is a big direction of future work we started out with having this system be implemented in just one course my course and now we're doing it in six different courses in our department and we're looking to build out a scalable system that can be ported across universities easily one of the challenges in this is that we have to build usable software and right now our software is a little bit of a research prototype software uh so it's a big it's a you know research stress for us to make this software easy to use for instructors there would be also some interesting questions to ask about you know could we learn from our rollouts of this program in terms of what groups work and what groups don't work can we improve the experience of students through the data that that we're collecting what is important to students what isn't important students this requires asking more nuanced questions than we've been asking so far but that's another direction that I'm looking to expand in I guess the other last idea that I think would be very very cool is that there's been documented evidence that shows that cohort effects are very positive so this means that for example you know having a bunch of students from similar identities taking a course at the same time for example or in terms of hiring for there's advantages to hiring if multiple women at the same time instead of have hiring you know a few women every year for instance and if we could use a similar strategy to form cohorts of students that I think would be very exciting what we find is that you know sometimes in Upper division courses in in a freshman course you have a thousand students and there's a lot of different people but as you go to Upper division maybe there aren't that many students from a particular identity group for example I remember as a woman when I was a student as I got to more and more specialized Opera division courses I was more and more likely to be the only woman in the in the classroom but maybe instead of you know every woman taking the class at a different time or according to a different schedule if we cohorted these folks and said like instead of everyone take half of you taking in the fall and taking in the spring if we could all have you take that course together in the fall semester we might be able to give students a much more positive outcome so I think that's another direction for future work and I'm curious if you see the possibility that uh maybe your partitioning algorithm could be used in groups outside of just the classroom yeah I think the algorithm has potential very interesting applications in the context of micro work or Flash teams that one might be pulling together there are platforms out there for example insta work that can that let's say you have a big event at a stadium and suddenly you need a large staff for a particular event at a particular time there might be value to constructing teams that are based on you know demographics previous experience familiarity with each other or the same kind of Singleton logic that we've been thinking about different contexts will of course have different preferences and different requirements but I think the same style of partitioning algorithm might actually lead to workers in Flash teams or micro work scenarios having more positive experiences in their work and you know therefore having increased productivity all around and so I think that would be a very cool kind of future direction also for this to build and is there anywhere listeners can go online if they want to follow the project my website which is just eeks.berkeley.edu kirija would be a great place we recently have a publication in the sigzi conference so that's that's online and I'm looking for collaborators I'm looking to interact with people who might be interested in you know understanding the future of this I think we have a very cool potential for understanding team Dynamics and group work in real world settings so I'm looking to collaborate with researchers from the social sciences or Business Schools as the case may be so anyone who's interested in studying group dynamics through surveys or otherwise in the context of classroom work I'd be very interested to connect yeah a lot of exciting opportunities there I think yeah and there is this one story that just kind of has has stuck with me that I would I'd love to share oh yeah this is one of the one of the students that that you know shared this with us which he said that starting my freshman year of freshman year of college virtually due to the pandemic made me feel really disconnected from the Berkeley Community but luckily being part of a study group in my first semester really changed that as a woman and underrepresented minority in stem my study group gave me the support system necessary to succeed in an extremely rigorous and minority isolating major this group was essential when it came to having people to complete assignments learn Concepts and study for exams with but it was also very helpful in me finding lifelong friends through my study group I met my first and closest friends at Berkeley who would help me adjust to college in an online world and who later became my supportive and caring roommates in my sophomore year at college and this is just a dream for me as an instructor right for me to be able to facilitate and not me really but just this structured intervention to be able to facilitate significant friendships is just amazing I mean in my dream world of success no one would need to sign up or opt into a study group system because they all already had such strong networks so that would be perfect if I could if that could somehow happen absolutely no I think it's a great vision and a lot of ways to produce even more qualified Engineers through the whole process fingers crossed thank you so much for taking the time to come on and share the work yeah thank you well that concludes this installment of data skeptic all about surveys please head over to survey.dataseptic.com and answer three questions I have for you about chat GPT we're going to share those results next week that survey.dataseptic.com [Music]