April Wang - Redesigning Notebooks for Better Collaboration | JupyterCon 2020

Key Takeaways

hi everyone i'm april wang a phd student from university of michigan today i'm presenting my research work on redesigning notebooks for better collaboration first let me introduce myself and my team i'm a phd student from school of information university of michigan my broader research area is human computer interaction with the focus on enabling and encouraging collaborations for exploratory programming i'm advised by professor steve only and professor chris brooks in particular i'm a member of the spot lab where we focus on understanding the factors that make programming tools usable and designing new tools for programmers i'm also a member of the educational technology collective lab where we explore intersections of technology with teaching learning and education with a particular focus on learning analytics educational data mining and collaborative engagement my work draws upon human-centered design techniques to explore barriers in different aspects of collaboration practices and opportunities in redesigning this state of art tools to support collaboration today i'm presenting two of our recent work along that threat the first work is an exploratory study on how data scientists use computational notebooks for real-time collaboration computational notebooks allow data scientists to create computational narratives this image shows jupyter notebook the most widely used notebook platform recently tools like google cloud have demonstrated the possibility for synchronous editing multiple users are able to edit the same notebook while changes are updated in real time this may bring new collaboration practices for data scientists however synchronous editing comes with its own challenges and may not always improve work efficiency studies have identified several issues with synchronous editing in other contexts such as collaborative writing and programming for example in a synchronous code editor things may become more complicated while one person's compilation error may completely block others work data science doesn't just involve the form of writing and coding it also includes the reasoning process to extract insights from data thus we believe that there can be unique challenges and opportunities for synchronous editing in the context of collaborative data science in particular we explore the following research questions what tools and strategies do data scientists currently use for collaboration compared to working on individual notebooks in a collaborative setting how does synchronous notebook editing change the way data scientists collaborate in computational notebooks what challenges if any do data scientists perceive in synchronous notebook editing to understand the tools and genealogies data scientists currently used in practice we first conducted a survey to ask their previous experience with collaboration our high level takeaway from the survey is that synchronous notebook editing tools are relatively new and not widely used among data scientists most data scientists work in the traditional collaboration setting where team members work on individual jupiter notebook and update each other's work asynchronously to further compare how their collaboration styles varied between different approaches we conducted an observational study with 24 data scientists working in pairs to solve a predictive modeling problem we assigned groups to work in either the shared condition or non-shared condition in a non-shared condition participants walked on individual notebooks in a shared condition a jupiter extension was enabled on the server to support synchronous editing in notebooks the extension shares notebook edits and actions in real time it executes code on a single interpreter and updates the output and runtime variables among collaborators on high level we discovered full collaboration styles and align it with an existing framework of collaborative writing groups in the shared condition use the single authoring pair authoring and divide and conquer style groups in the non-shared condition use the divide and conquer and competitive authoring style next we looked at how participants communicated with each other participants in the non-shared condition relied more on slack to send files code snippets and outputs while participants in this shared condition are less distracted by those low-level coordinations and have more time to focus on more important issues this indicates that working in a shared notebook may reduce communication cost by establishing a shared context this relates to our analysis of the final submission we looked at the error scores of each submission the number of alternative models participants have explored and the lines of content in the notebook the result suggests that working in a shared notebook encourages groups to explore more solutions and leads to a better result to summarize our main findings indicate that synchronous editing helps data scientists maintain a shared understanding while reducing communication costs it also provides the flexibility to branch through tasks thus synchronous editing in notebook improves the overall efficiency of collaboration despite all the benefits that collaborative editing features offer with respect to sharing contacts and improving productivity we discovered several challenges in using the collaborative notebook let's take a look at an example when working on individual notebooks and not clear about the plan bob and alice both go ahead and explore the task on their own even if they end up duplicating each other's work while working in the same notebook bob is unlikely to duplicate the work if knowing that alice is already working on it without splitting the task well bob may not know what to do until alice finishes her part they may end up with alice doing 80 percentage of the work and above doing 20 percentage of the work besides the example we just discussed our paper summarizes six challenges of synchronous editing in notebooks for example working on synchronous notebooks is more likely to lead to unbalanced participation where one team member does the majority of implementations and ideations in summary our exploratory study investigates how synchronous editing in computational notebooks changed the way data scientists work together compared to working on individual notebooks working on synchronous notebooks improves collaboration but also introduces new problems the challenges in using the current real-time collaboration features suggest that we need better collaborative editing features for computational notebooks in the next project we aim to address one of the specific challenges discovered in previous study improve explainability of the collaborative notebooks when teams of data scientists collaborate on computational notebooks their discussions often contain valuable insights into their design decisions these decisions not only explain analysis in the current notebook but also alternative paths which are often fully documented however these discussions are long and disconnected from the notebooks for which they could provide valuable context for example there is a relevant message saying there is a bug in the second cell but we didn't know what the second cell was by the time that the message was sent team members typically need to have a shared context to make sense of the discussion this can be particularly challenging as a notebook evolves in this project we propose to improve collaborative data science by connecting discussions with computational notebooks to better understand how discussions can be useful for explaining the data exploration process we analyzed the 760 chart messages collected from three data science group projects we coded messages based on their purpose their relationship to the evolving notebook and the specific aspect of the notebook dimension in summary we drive three design implications from our findings chat messages are useful for explaining the exploration process but they're difficult to follow notebook elements are frequently referred to in chat messages based on the findings we designed the cholesterol a plug-in for jupiter notebook cholesterol augments jupiter with several collaborative features first it allows users to share notebooks collaborate in real time and discuss with collaborators callisto adds a share button that generates a unique url for collaborators to join the shared notebook session users can see a panel listing all the collaborators cholesterol synchronizes edits runtime variables outputs annotations and cursors among collaborators jupiter doesn't have built-in messaging features to capture discussions callisto embeds a synchronous chat panel directly in the shared notebook on the left side callisto includes a panel showing users and its histories to provide awareness of collaborators activities after introducing the basic collaboration features of cholesterol let's take a look at how callisto captures the connection between messages and notebook content first users can make direct reference to notebook elements in their chat messages by clicking the magic wand and then selecting the relevant notebook elements callisto enables five types of references code references cell references annotation references snapshot references which point to a previous notebook version and the div references which point to a comparison between two history versions now that we're able to capture the connections through explicit references created by users what about other messages that do not have explicit references when users send a message it is likely to be relevant to the currently selected cell thus callisto automatically attaches a cell reference to the currently selected cell if users do not add an explicit reference users can also manually correct errors from automatic references now that we have created connections between messages and notebook content how can we use this connection cluster enables two broader uses of these connections to understand in the context of a given message or to find the part of the discussion that is relevant to a specific part of the notebook the connection from messages to notebook content demonstrate what changes were made when a user clicks on the reference cholesterol highlights the relevant elements in the notebook similarly when a user clicks on the message callisto highlights the relevant cell in current notebook messages might become out of date if they reference an element of the notebook that is later modified or deleted clisto allows users to view the snapshot of the notebook when the message was sent in some cases users may want to understand what collaborators were doing between messages when selecting two messages kalisto can navigate users to the diff view comparing two snapshots in diff wheel code differences and output differences are highlighted for the other way around the connection from notable content to messages explains why changes were made callisto allows users to see which parts of the discussion are relevant for a particular part of the notebook users can enable the filter mode to only see messages and edits relevant to the selected cell to assess how cholesterol assisted new collaborators when joining the collaborative notebook we designed a two-stage evaluation study we first observed the participants working in pairs on a data science task in real time to test coisto's usability we then conducted a comparison study with the third individual joining the shared project for using cluster or a light version of the system with no contextual links in stage 1 we observed that participants are hesitant to make accurate and polished references on average each group created around seven messages that contain references out of a hundred messages and most of them are the default cell pointers with most connections are made by automatically inferred references we manually went through the references and find that 92 percentage are connected to the correct context in stage 2 we evaluated how a new collaborator followed up with an ongoing collaborative project we asked participants to first explore the notebook and answer five questions related to prior analysis then we asked them to use the tool in depth to follow up on the ongoing project we compared two versions of cholesterol in this stage the full version and the light version where we replace manual references with a textured description we find that participants in the experiment condition understood the ongoing collaboration process better participants in both conditions reported a need to check the chat messages even though the notebook already contains some code comments and explanatory text comparatively participants in the control condition have trouble following the chat message participants in the experiment condition benefit from established connection between messages and notebook most of them kept the filtering mode enabled and went back and forth to check the context of messages they also used the diff view to better understand how a code change resulted in an output change for example participants in the experiment condition were more aware of the relevant message the result looks much better that are connected to this cell some participants selected this message and the previous message to generate a diff view they were able to see that prior collaborators applied a log transformation which made the distribution plot look the less skilled in summary our paper contributes the design of cholesterol with a set of features to make chat messages more useful for understanding the past exploration process in the notebook our evaluation provides evidence that creating mappings between messages notebook elements and versions helps data scientists understand and follow up on the exploration pipeline along this line i would like to introduce some ideas for future work the two studies i presented focused on professional data scientists what's missing here is that many roles who are not skilled at data scientists collaborate with professional data scientists we hope to investigate the communication challenges between data scientists and other stakeholders furthermore we would like to explore approaches towards automatic refactoring computational narratives let me give you an example this example shows three different levels of explanations for the code cell the first one explains the process we drew a distribution plot the second one explains the results the sale prices are right skewed and the third one explains the reasoning the sale prices are right skilled this was expected as few people can afford very expensive houses we hope to explore how code summarization techniques can be used to generate different levels of explanations with that i would like to end my presentation and open for questions thank you for listening