Successful machine learning models: lessons learned at Booking.com
📰 Hacker News · joker3
Learn 6 key lessons from Booking.com's experience with 150 successful machine learning models, including the importance of business value, model performance, and iterative development
Action Steps
- Read the paper to understand the 6 lessons learned by Booking.com
- Apply the lessons to your own ML model development, focusing on business value and iterative development
- Use techniques like randomized controlled trials to test the business impact of your models
- Prioritize model performance and prediction serving latency
- Get early feedback on model quality to improve development
- Integrate ML development with other disciplines, such as product and engineering, to drive business impact
Who Needs to Know This
Data scientists, machine learning engineers, and product managers can benefit from understanding these lessons to improve their own ML model development and deployment
Key Insight
💡 Driving true business impact with ML is hard, but an iterative, hypothesis-driven process integrated with other disciplines is key to success
Share This
🚀 6 lessons from @Bookingcom's 150 successful ML models: focus on business value, model performance, and iterative development 📈
Key Takeaways
Learn 6 key lessons from Booking.com's experience with 150 successful machine learning models, including the importance of business value, model performance, and iterative development
Full Article
Title: 150 successful machine learning models: 6 lessons learned at Booking.com
URL Source: https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/
Published Time: 2019-10-07T05:00:00+00:00
Markdown Content:
[Skip to content](https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/#content)
[the morning paper](https://blog.acolyer.org/)
a random walk through Computer Science research, by Adrian Colyer
Menu
* [About](https://blog.acolyer.org/about/)
* [Archives](https://blog.acolyer.org/archives/)
* [InfoQ QR Editions](https://blog.acolyer.org/infoq-quarterly-review-editions/)
* [Search](https://blog.acolyer.org/search/)
* [Subscribe](https://blog.acolyer.org/email-subs/)
* [Tags](https://blog.acolyer.org/tags/)
* [Privacy](https://blog.acolyer.org/privacy/)
# 150 successful machine learning models: 6 lessons learned at Booking.com
[October 7, 2019 October 4, 2019](https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/) ~ [adriancolyer](https://blog.acolyer.org/author/adriancolyer/)
[150 successful machine learning models: 6 lessons learned at Booking.com](https://www.kdd.org/kdd2019/accepted-papers/view/150-successful-machine-learning-models-6-lessons-learned-at-booking.com) Bernadi et al., _KDD’19_
Here’s a paper that will reward careful study for many organisations. We’ve previously looked at the [deep penetration of machine learning models in the product stacks of leading companies](https://blog.acolyer.org/2018/12/17/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/), and also some of the [pre-requisites for being successful with it](https://blog.acolyer.org/2017/10/03/tfx-a-tensorflow-based-production-scale-machine-learning-platform/). Today’s paper choice is a wonderful summary of lessons learned integrating around 150 successful customer facing applications of machine learning at Booking.com. Oddly enough given the paper title, the six lessons are never explicitly listed or enumerated in the body of the paper, but they can be inferred from the division into sections. My interpretation of them is as follows:
1. Projects introducing machine learned models deliver strong business value
2. Model performance is not the same as business performance
3. Be clear about the problem you’re trying to solve
4. Prediction serving latency matters
5. Get early feedback on model quality
6. Test the business impact of your models using randomised controlled trials (follows from #2)
There are way more than 6 good pieces of advice contained within the paper though!
> We found that driving true business impact is amazingly hard, plus it is difficult to isolate and understand the connection between efforts on modeling and the observed impact… Our main conclusion is that an iterative, hypothesis driven process, integrated with other disciplines was fundamental to build 150 successful products enabled by machine learning.
In case you’re tempted to stop reading at this point, please don’t interpret that quote as saying that investing in machine learning isn’t worth it. On the contrary, developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness as all the other characteristics of high-performing organisations highlighted in the [State of DevOps reports](https://cloud.google.com/blog/products/devops-sre/the-2019-accelerate-state-of-devops-elite-performance-productivity-and-scaling). (And by the way, wouldn’t it be wonderful to see data confirming or denying that hypothesis in future reports!).
### Context
You’ve probably heard of Booking.com, ‘the world’s largest online travel agent.’ Delivering a great experience to their users is made challenging by a number of factors:
* The stakes are high for recommendations – booking a stay at the wrong place is much worse than streaming
URL Source: https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/
Published Time: 2019-10-07T05:00:00+00:00
Markdown Content:
[Skip to content](https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/#content)
[the morning paper](https://blog.acolyer.org/)
a random walk through Computer Science research, by Adrian Colyer
Menu
* [About](https://blog.acolyer.org/about/)
* [Archives](https://blog.acolyer.org/archives/)
* [InfoQ QR Editions](https://blog.acolyer.org/infoq-quarterly-review-editions/)
* [Search](https://blog.acolyer.org/search/)
* [Subscribe](https://blog.acolyer.org/email-subs/)
* [Tags](https://blog.acolyer.org/tags/)
* [Privacy](https://blog.acolyer.org/privacy/)
# 150 successful machine learning models: 6 lessons learned at Booking.com
[October 7, 2019 October 4, 2019](https://blog.acolyer.org/2019/10/07/150-successful-machine-learning-models/) ~ [adriancolyer](https://blog.acolyer.org/author/adriancolyer/)
[150 successful machine learning models: 6 lessons learned at Booking.com](https://www.kdd.org/kdd2019/accepted-papers/view/150-successful-machine-learning-models-6-lessons-learned-at-booking.com) Bernadi et al., _KDD’19_
Here’s a paper that will reward careful study for many organisations. We’ve previously looked at the [deep penetration of machine learning models in the product stacks of leading companies](https://blog.acolyer.org/2018/12/17/applied-machine-learning-at-facebook-a-datacenter-infrastructure-perspective/), and also some of the [pre-requisites for being successful with it](https://blog.acolyer.org/2017/10/03/tfx-a-tensorflow-based-production-scale-machine-learning-platform/). Today’s paper choice is a wonderful summary of lessons learned integrating around 150 successful customer facing applications of machine learning at Booking.com. Oddly enough given the paper title, the six lessons are never explicitly listed or enumerated in the body of the paper, but they can be inferred from the division into sections. My interpretation of them is as follows:
1. Projects introducing machine learned models deliver strong business value
2. Model performance is not the same as business performance
3. Be clear about the problem you’re trying to solve
4. Prediction serving latency matters
5. Get early feedback on model quality
6. Test the business impact of your models using randomised controlled trials (follows from #2)
There are way more than 6 good pieces of advice contained within the paper though!
> We found that driving true business impact is amazingly hard, plus it is difficult to isolate and understand the connection between efforts on modeling and the observed impact… Our main conclusion is that an iterative, hypothesis driven process, integrated with other disciplines was fundamental to build 150 successful products enabled by machine learning.
In case you’re tempted to stop reading at this point, please don’t interpret that quote as saying that investing in machine learning isn’t worth it. On the contrary, developing an organisational capability to design, build, and deploy successful machine learned models in user-facing contexts is, in my opinion, as fundamental to an organisation’s competitiveness as all the other characteristics of high-performing organisations highlighted in the [State of DevOps reports](https://cloud.google.com/blog/products/devops-sre/the-2019-accelerate-state-of-devops-elite-performance-productivity-and-scaling). (And by the way, wouldn’t it be wonderful to see data confirming or denying that hypothesis in future reports!).
### Context
You’ve probably heard of Booking.com, ‘the world’s largest online travel agent.’ Delivering a great experience to their users is made challenging by a number of factors:
* The stakes are high for recommendations – booking a stay at the wrong place is much worse than streaming
DeepCamp AI