User retention analysis framework | data science product sense

Key Takeaways

hello everyone my name is sophia i am a data scientist welcome to my video in this video i plan to talk about a retention analysis framework from a data science perspective i will talk about the following topics three dimensions of measuring retention time user status and action analytical frameworks of retention analysis how to find the aha moment and a happy moment through analysis and investigate why users stay and leave so if you're interested in learning about retention analysis or interested in learning how to answer the business business growth frameworks aarrr stands for acquisition activation retention revenue and referral but when you think about it it doesn't make sense to put acquisition first it doesn't matter how many new users you get it doesn't mean anything if you can't keep them stay and also of course acquisition strategies like ads are very expensive it is a lot cheaper to retain a user than to get a new user and for many other reasons some people later reprioritized the aarr framework to the rar framework with retention as the number one priority the rar framework stands for retention activation referral revenue and acquisition so the first question that comes to mind might be how do we define and measure retention what is retention retention measures how many users return to your product over some specified time if you look at this definition you might see some problems with this definition what do we what do we mean by users what do we mean by coming back and what do we mean by time those are really good questions and those are the exact three dimensions i'm going to talk about understanding those measures and dimensions will help you choose the right one for your product the first dimension is time the most popular retention measure that everyone uses is the end day retention it measures that among users who first use the product at day zero or proportion of them are still active at day in here day could also be week or month for a one-week retention measure or a n month retention measure whether to use daily weekly or monthly attention totally depends on your product and how often users use your product for example for gaming products with high stickiness it is typical to measure and day retention on a daily basis unbounded retention also called rolling retention measures among users who first use the products at day zero or proportion of them are still active on and after they end bracket retention is more flexible you can define whichever time frame you're interested in for example pinterest measures the percentage of new signups that are still doing key actions during doing a one-week time window of 28 to 35 days after sign up we'll talk about what is the key actions later the second dimension i want to talk about is user status in addition to time user status is often another important dimension to consider so how do we define user status there are many ways to define user statuses you might see different definitions from different companies and different products people have their own way to define user statuses so here is just one way and one example that user status can be defined new user churn user inactive user reactive user and active user it is often important to calculate retention for different user statuses let's think about it new user retention measures the proportion of new users who stay active active user retention measures the proportion of active users who stays active in the example of feature changes we might see that active user retention goes down because users have already gotten used to the product however in the meantime the new user retention could go up suggesting that this feature change could be an improvement improvement in the long run the third dimension i want to talk about is action when we say users use a product we didn't define what we mean by use should we define use as visiting the product page staying for a certain amount of time conducting certain actions or purchasing a product we call the actions we use to calculate retention key actions with so many measures and all those dimensions which one do we use and where do we start for the first dimension time i will start with the classic end day week or month retention and investigate the rest later to determine the time interval i would plot how often users use the product plot the percentage percentage of users who use a product x amount of times with various intervals and plot retention with those various intervals to see which one makes sense for your product it's okay to include more than one time interval in your analysis for the second dimension user status i will start with the new user retention and investigate the active retention active user retention later especially if you are developing our new product third for the third dimension uh user actions it actually depends on the product goal are you more investigated in monetization values or are you at a stage to grow and retain your users through certain actions um yeah so really depends on your product and your business case next let's talk about the analytical frameworks of retention analysis the first thing is retention curve which is widely used everyone probably knows a retention curve plots the retention over time with time on the x-axis and retention rate on the y-axis ideally we would like to see a smiling curve when users come back more and more over time a declining curve signifies danger and a flight flattening curve signifies a healthy product the goal for a product is to shift the curve up and flatten or up till the curve it is often common to also do a retention cohort analysis cohort analysis tells us where do we see products do well and not well we can define cohorts from many different categories such as demographics acquisitions and behavior in terms of the graphical representations we can represent the retention cohorts in a grouped retention curve or a triangle retention chart here is an example with the acquisition time as the cohort and finally we have statistical analysis like any other types of analysis we can start by calculating descriptive statistics and correlations we can also conduct survival analysis to determine which attributes matter in addition the rfm retention frequency monitoring analysis framework is often popular for user segmentations so where should we start with analysis a starting point would be to see a dashboard showing the retention curve and triangle retention charts for all the interesting cohorts and groups then we can generate generate insights from the dashboard and starting from there well the important value of retention analysis is that we can do some really concrete analysis to generate insights for example for new users we can try to find the aha moment throughout our analysis for new users the most important thing is the onboarding experience that happens within the first few days or weeks almost all products wants to help users to achieve the aha moment as quickly as possible the question is how do how do we define a user's aha moment what is the user's behavior that contribute to the aha moment for example facebook focuses on getting seven friends in 10 days the behavior getting friends and the magic number seven friends in 10 days indicates users future success i don't think that we all should find a behavior with a magic number for all of our products but i do think it is important to figure out what behavior contribute to the aha moment from our data then the product team can use this information to improve our onboarding experience a starting point of this analysis is to find the user event and the onboarding behaviors that are related to retention in future weeks and month so simple correlation analysis or descriptive statistics will show you this kind of insight for long-term users we want to find the habit moment how do the most successful long-term users form a habit to use a product who are those users what features do they keep using what attributes do they have and what is the user journey for them the goal is to find successful feature features and the user journeys investing those successful features and encouraging new users to follow successful user journeys again we can do some descriptive analysis on long-term users to identify those user behaviors and features and then conduct cohort analysis to find user journeys for those people okay so you might wonder for a given product why do people choose to leave and stay there might be many many reasons why people leave and stay there might be many reasons why people choose to leave and churn for example there might not be they might not understand the product the product might not be might be hard to use people prefer a competitor's product the product might have some issues like bugs or being slow um the new user is not the target user there could be a mismatch between the users and the core features and also people might only need a product for a very short amount a short period of time and they don't need our product anymore and then there could be many reasons why people stay for example people may love the product the personalized notifications work it has become a habit for people to use the product um [Music] and many more those reasons behind why users stay and leave might be hard to derive directly from the product data instead doing user research and experiments can help for example conducting experiments on a better onboarding experience might help people understand the product the product better doing during onboarding experiments on push notifications might help remind people on the values of the product and experiments on that on various product features might help identify and promote the key features people love understanding the reason is hard collaboration here is the key to understand the users we can collaborate with the ux research team to help design the surveys and interviews and generate insights to run experiments us data scientists can collaborate with the engineers to to design experiments and analyze results hopefully this video provides you some perspectives on how to get started on your retention analysis and how to deep dive for future analysis please let me know if you have any thoughts or comments i would love to hear them thank you