Research and experimentation are the “peanut butter and jelly” of customer insights — here’s why

Written by Molly Stevens and Lukas Vermeer.

User research and experimentation are both powerful tools that allow us to narrow down and select product options. Each tool has a range of core capabilities that ensure our customers have great experiences and the business reaches their goals.

At Booking.com, our teams collaborate together within an insights organization, focusing on how to measure and understand the customer experience across all aspects of our business. When we (Molly and Lukas) started working together a year ago, people expected that we would not get along. This expectation stemmed from the mistaken idea that somehow…


FOMO made me do it

At the end of this month, I will be leaving the company where I have spent the past eight amazing years. I am not sure I can adequately explain my motivation (decisions of this nature rarely have a single cause) but I will try.

Not all decisions in life can be data driven. I am leaving because it simply felt like it was time for me to leave. I will use three metaphors to try to convey how I feel. I hope this may help others who are in a similar position.

A picture of me looking out over Half Dome in Yosemite National Park.
A picture of me looking out over Half Dome in Yosemite National Park.

I am leaving for the same reason that I love travel.

Booking.com’s mission is to make it easier for…


Kickstarting and keeping the A/B testing momentum

Written by Aleksander Fabijan (Mircosoft), Benjamin Arai (Microsoft), Pavel Dmitriev (Outreach.io, and Lukas Vermeer (Booking.com). Originally published on the Microsoft Experimentation Platform blog.

At the time when we published the Experimentation Evolution Model in 2017 [1], we never expected such an interest in it. Practitioners from hundreds of companies and several different industries reached out with thoughts, concerns and tips on how to kickstart and scale an A/B testing program from “Crawl” or “Walk” milestones, to the “Run” or “Fly” stages [1]. Without doubt, scaling A/B testing in an organization is a challenge with which all of us can empathize…


How I learnt to stop worrying and made the inbox-as-to-do-list approach work for me.

I know. I know. I know! Using my inbox as my to-do-list is considered bad practice, and I should feel bad for even trying.

Several well known time management books explicitly advise against this approach. Based on this advice, I have tried different tools and techniques to create a to-do-list separate from my inbox. Yet the honest truth is that all of these attempts didn’t work nearly as well for me as my current Gmail setup.

I learnt to stop worrying and made the inbox-as-to-do-list approach work for me. I will try to explain why and how. …


Mediation Analysis to Disentangle Direct and Indirect Effects

Written by: Tolga Oztan, Zoe van Havre, Chad A Davis and Lukas Vermeer.

At Booking.com, one of the key ingredients to customer-centric product development isn’t just bright minds having great ideas, but collecting the evidence to support these ideas. We test each idea addressing a customer pain point via an A/B test using our in-house experiment platform. This platform is able to test thousands of changes simultaneously, with real customers, collecting data on the outcome within minutes of being implemented. However, as Booking.com …


How we use online controlled experiments at Booking.com to release new features faster and more safely

Written by Iskra and Lukas Vermeer.

At Booking.com, experimentation is an important part of our product development cycle. On a daily basis we implement, deploy to production, execute and analyse hundreds of concurrent randomized controlled trials — also known as A/B tests — to quickly validate ideas. These controlled experiments run across all our products, from mobile apps and tools used by accommodation providers to customer service phone lines and internal systems.

Almost every product change is wrapped in a controlled experiment. From entire redesigns and infrastructure changes to the smallest bug fixes, these experiments allow us to develop and…


Categorising Common Challenges

Written by Timo Kluck and Lukas Vermeer.

Online experimentation platforms abstract away many of the details of experimental design, and their users do not have to worry about sampling, randomisation, subject tracking, data collection, metric definition and interpretation of results. The rapid adoption of these platforms in the industry might in part be attributed to the ease-of-use these abstractions provide. However, there are common pitfalls to avoid when running controlled experiments on the web, and one needs experts familiar with the entire software stack to be involved in the process.

In this post, we argue that these pitfalls are not…

Lukas Vermeer

I help people run experiments to make better products for customers. I explain science using historical narratives and teach statistics through storytelling.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store