Data Science Project Workflow | Behind the Scenes at Coursera

Key Takeaways

[Music] Have you ever wondered how data science projects make it from conception to completion? What's the secret sauce that transforms raw data into businesschanging insights? Today, I'm going to take you behind the scenes of the data science workflow. The structured process that turns promising ideas into powerful solutions. In my years as a data scientist, I've seen brilliant analyses fall flat because of poor workflows. And I've watched seemingly simple projects create massive impact because of exceptional organization and process. Whether your code is flawless or your statistical knowledge is unmatched, without a solid workflow, your data science project might never reach its full potential. Let me start by explaining what a data science workflow looks like on my team. We begin every project with a shaping document that outlines several critical elements. Project objectives, scope, stakeholders, actionability, roadmap, and milestones, and what's explicitly out of scope. This document serves as our northstar throughout the project. Once we have our shaping document, we create a Jira epic for the project. This epic is essentially a container for organizing related tasks in our project management system. Each step becomes a task that can be completed in a few hours of focused work. This approach helps us track progress, check off completed steps, and always know what remains to be done. Our workflow then typically follows these steps. We write SQL queries which we check into GitHub and then request code reviews from colleagues. Next, we develop pipelines in tools like Airflow, again checking into GitHub for code reviews. We create Jupiter analysis notebooks, also submitting them for review. If needed, we build Looker dashboards which undergo the same review process. Finally, we create decks of reports that we share with our immediate team for feedback before presenting to stakeholders. An often overlooked but crucial final step is adding our documentation to Confluence, our centralized knowledge sharing platform where information is organized and preserved. Before closing any project, we ensure others can find it and understand what it is long after we've moved on to other work. This institutional knowledge is invaluable. In my experience, two components of data science workflows stand out as particularly important. Version control and code review. Version control is the most straightforward way to enable code review. Having another data scientist review your code and analysis provides feedback that helps you improve. It also creates a repository for storing all documents related to your analysis, keeping everything in one place and well documented. I found that feedback from teammates is often the critical factor that elevates an analysis from good to great. Colleagues can suggest more rigorous statistical approaches or ways to optimize code performance. They also ask questions that reveal what you haven't documented clearly enough which you can then fix. Version control is also essential for recovering from accidents. When something gets deleted or changed unintentionally, you can always revert to an earlier state. Let me share a personal experience about workflow challenges. At the beginning of the pandemic, I was tasked with building daily dashboards for a university while everyone was remote. At that time, I didn't know how to set up automation for job runs. So, I woke up at 5:00 a.m. every day to manually rerun the code that updated multiple dashboards with new values. Every morning I contemplated setting up automation, but figured it might not be worth the time investment since we might return to in-person work soon, and I only needed to keep the dashboards updated while we were remote. I eventually did automate the job runs, but I wish I had done so from the beginning. I would have gotten so much more sleep during that time if I had. This experience taught me that investing time in automating repetitive tasks pays dividends quickly, even if you think the need might be temporary. For those looking to improve their data science workflows, I'd recommend reading platforms like Medium, where data scientists often write up their projects end to end. These resources help you learn new tools and methodologies that other professionals are using successfully. Seeing how others structure their work can provide valuable insights for refining your own approach. The key components of an effective workflow really depend on the project's objective. Different projects require different approaches, but the constants are clear documentation, version control, peer review, and thoughtful planning from the outset. By integrating diverse skills from data loading and manipulation to visualization, statistical analysis, and even AI assisted coding, we can build comprehensive solutions to complex problems. This integration supported by robust workflows is what makes modern data science so powerful and effective. [Music]