“Won’t running all those experiments slow down our product development?” When speaking at conferences, I get this question a lot. And my answer is always the same: yes and no. Let me explain. There is a difference between “speed” and “velocity”. Speed is how fast something is going, while velocity…

Three things I wanted to experience in my next job — One month ago, I started a new job. I left a place I love to go exploring. After eight years of following a clear path, I was ready to be left to my own devices and seek out a new course. What came after was to choose the next step…

Research and experimentation are the “peanut butter and jelly” of customer insights — here’s why — Written by Molly Stevens and Lukas Vermeer. User research and experimentation are both powerful tools that allow us to narrow down and select product options. Each tool has a range of core capabilities that ensure our customers have great experiences and the business reaches their goals. At Booking.com, our teams…

FOMO made me do it — At the end of this month, I will be leaving the company where I have spent the past eight amazing years. I am not sure I can adequately explain my motivation (decisions of this nature rarely have a single cause) but I will try. Not all decisions in life can…

Kickstarting and keeping the A/B testing momentum — Written by Aleksander Fabijan (Mircosoft), Benjamin Arai (Microsoft), Pavel Dmitriev (Outreach.io, and Lukas Vermeer (Booking.com). Originally published on the Microsoft Experimentation Platform blog. At the time when we published the Experimentation Evolution Model in 2017 [1], we never expected such an interest in it. Practitioners from hundreds of companies and…

How I learnt to stop worrying and made the inbox-as-to-do-list approach work for me. — I know. I know. I know! Using my inbox as my to-do-list is considered bad practice, and I should feel bad for even trying. Several well known time management books explicitly advise against this approach. Based on this advice, I have tried different tools and techniques to create a to-do-list…

Mediation Analysis to Disentangle Direct and Indirect Effects — Written by: Tolga Oztan, Zoe van Havre, Chad A Davis and Lukas Vermeer. At Booking.com, one of the key ingredients to customer-centric product development isn’t just bright minds having great ideas, but collecting the evidence to support these ideas. We test each idea addressing a customer pain point via an…

How we use online controlled experiments at Booking.com to release new features faster and more safely — Written by Iskra and Lukas Vermeer. At Booking.com, experimentation is an important part of our product development cycle. On a daily basis we implement, deploy to production, execute and analyse hundreds of concurrent randomized controlled trials — also known as A/B tests — to quickly validate ideas. …

Categorising Common Challenges — Written by Timo Kluck and Lukas Vermeer. Online experimentation platforms abstract away many of the details of experimental design, and their users do not have to worry about sampling, randomisation, subject tracking, data collection, metric definition and interpretation of results. The rapid adoption of these platforms in the industry might…