Organizations are embracing product experimentation as a tool for continuous improvement
Companies that embrace product experimentation are able to capture highly accurate data about markets and product ideas and gain intelligence about market trends. As a result, these companies are able to continuously deliver innovation and measure business outcomes.
Industry leaders such as Netflix, Instagram and Facebook have used product experimentation to maximize a positive customer experience, increase their user base and elevate their market positions to the highest of levels.
Let’s say you want to be one of those companies—or at least you and, hopefully, your team, want to experiment with product features in hopes that your company will embrace experimentation on a larger scale to achieve continuous improvement.
Great. But now what?
Just thinking about learning the basic concepts of experimentation and figuring out where to start can seem like an overwhelmingly daunting task. Fortunately, there are a few tried and true steps you can take to get on the path to product experimentation for continuous improvement.
Measure First
“You have to walk before you can run.” That old adage certainly applies here. Rather than attempting to jump headfirst into experimentation, organizations first must ensure they are measuring things. Are you tracking things such as users per day, trends, dates and conversions? The idea here is to become data driven, because before an organization can run effective experiments, it first must gather quantifiable data; understand what this data means for overall growth, user retention and long-term projection; and then base business decisions on this data.
Buy or Build
No matter the size and scope of the experimentation throughout the organization, even if it is just one team or department, you need a platform or tool to experiment on. Many of the most progressive organizations that have based their growth and continuous improvement on experimentation have built in-house solutions to facilitate this. However, this can be time-consuming from the start and the platform will continue to require attention as it evolves. This evolution also requires a specific skill set to make it happen—specifically, data science and engineering—to ensure the experimentation platform is statistically sound.
For most, investing in a robust tool or platform built specifically for experimentation is the best option. Most experimentation platforms allow you to plug and play, which provides the right features and capabilities to get an experimentation started quickly.
Hypothesize
The next step is to create a testable hypothesis and then record and analyze the data during testing. It’s important to begin thinking in terms of hypotheses, as framing plans and ideas in this manner makes them easier to reuse. In other words, being able to say, “Based on this, we predict we can move that metric,” makes your processes very repeatable. Getting into this mindset and having a standardized format—in terms of how things are designed and reported on—makes the lives of everyone on your team easier, as they can simply replicate similar experiment designs. And they will have fewer excuses not to.
Keep It Simple
Now that you have quantifiable data and some hypotheses to work with, you are ready to dip a toe into the experimentation pool. It is always a good idea to start with a single simple experiment first, no matter how small. For example, experimenting on a known pain point in your product is a nice place to start. Once you can successfully run an experiment that allows you to fix something in your product that you know is a problem, then you are ready to move on to bigger things such as revenue creation.
Choose the Right Metrics
As your team’s product experimentation efforts become more sophisticated, choosing the right metrics—and more importantly, your overall evaluation criteria (OEC)—should become a top priority in continuous improvement. Your OEC comprises the most important metrics of your experiment and it’s a measure of long-term business value or user satisfaction.
In choosing which metrics to use, there are three essential characteristics of an effective OEC to consider.
The first of these is sensitivity. This means the metrics you have chosen should be adjustable due to small changes in business value or user satisfaction, which helps ensure the result of an experiment is clear within a reasonable time period. For example, let’s say a customer only buys something from your website once or twice year. That’s not a very useful metric for an experiment since it would be difficult to influence.
Next is directionality. Your metrics should be moving in one direction or another based on whether your overall business value increases or decreases. That said, the direction some metrics take can be ambiguous. For instance, say you are attempting to measure customers’ engagement with your support team. An increase in that metric could be a positive thing because it might be an indication that more people are engaging with your core product. On the other hand, a decrease in this metric could be an indication that your core product has become easier to use and therefore now requires less support.
The third characteristic to consider is understandability, meaning the metric should be easily understood, both in terms of its direction and its definition, by your entire organization. Moreover, you should ensure it ties the experiment directly to business value.
Of course, much more will need to be done to transform your organization into a place that truly embraces an experimentation culture and reaps the benefits that it brings in terms of customer experience and beyond. But these steps should set you and your team down the right path.
