Summary
Data analysis might sound a bit nerdy, but it should be part of every product manager’s and product owner’s tool box. The idea is simple: Investigate the data gathered, learn form it, and use the new knowledge to create a successful product. In theory, that’s easy. But in practice, it can be challenging. The following tips help you get the most of your data analysis efforts.
But in practice, it can be challenging. The following tips help you get the most of your data analysis efforts.
Have a Clear Research Goal
Collecting the right data and analysing it effectively requires a clear research goal – understanding the reason why you carry out the work, and what you want to achieve. At the early stages of creating a product, your goal is likely to validate a critical assumption. This could be the product’s value proposition, the main revenue source, or an aspect of the user interaction design. Lean Startup captures the research goal as a hypothesis, and Scrum as a sprint goal.
Without a research goal you are in danger of collecting the wrong data, drawing the wrong conclusions, and moving your product in the wrong direction. In a sense, you are just trashing around hoping that the data will magically tell you what to do.
Separate Data Analysis from Data Collection
Once you have gathered the relevant data – for instance, by observing users, demoing a prototype, or tracking user behaviour using an analytics tool – step back, and carefully reflect on it before you make any decisions.
If you come from a Scrum background, then separating analysis from data collection may be new to you. In Scrum, data is traditionally collected and analysed in the sprint review meeting without always clearly separating the two activities. This carries the danger of rushing or skipping the analysis work, and making suboptimal or wrong decisions.
I hence recommend that you first collect the relevant data and then analyse it. Use the new insights to change the appropriate artefacts, for instance, your Vision Board, Product Canvas, or product backlog, select a new research goal, and start the next cycle.
Keep an Open Mind
Keeping open mind may sound trivial, but clinging to an idea – not the lack of a fancy analysis technique or tool – is the biggest barrier to drawing the right conclusions in my experience. I know what I am talking about: When working on a new product, I feel strongly about my own ideas, and I sometimes have a hard time changing my mind. But being too attached to an idea, or being too eager to succeed carries the danger of rejecting any data that challenges it, which may well result in a poor product.
Before you carry out any analysis, take a deep breath and relax. Whenever you get tense or worked up about the data, tell yourself that it is not you, who is being challenged, but ideas, assumptions, and concepts. And ideas, assumptions, and concepts don’t have any pride; they don’t want to be right or wrong. They are just thoughts.
Mitigate Cognitive Biases
My fourth tip is to be aware of the cognitive biases we all have. A cognitive bias is a fault in our thinking causing us to draw the wrong conclusions. Confirmation biases, for instance, is the tendency to search for or interpret information in a way that confirms our preconceptions, and self-serving bias is the tendency to claim more responsibility for our successes than our failures.
Maybe the worst thing you can do when employing a framework such as Lean Startup or Scrum is to run iteration after iteration only to look for data that confirms your ideas – and to reject the rest. This is likely to result in late failure, which is just as painful as in a traditional, sequential approach.
A great way to mitigate cognitive biases is to analyse the data collaboratively. This tends to balance out individual preferences, believes, and preconceptions. Consider therefore involving the development team in the data analysis, particularly when you validate critical assumptions.
Clean the Data
Don’t forget to clean the data. Remove data whose quality is too poor to interpret it correctly, and discard irrelevant data. This should be easy enough – unless it’s an idea from an important customer or a powerful stakeholder. But saying yes to every idea is not going to result in a great product but in a cluttered piece of software with a poor user experience. As Steve Jobs once said:
Innovation is not about saying yes to everything. It’s about saying no to all but the most crucial features.
Stay true to your vision, and leverage your primary persona to determine the right features.
Pivot or Persevere
When analysing the data, ask yourself if it invalidates your strategy, for instance, the customer segment chosen, the product’s value proposition, the anticipated user experience, or the revenue source. If it does, pivot and change your strategy. This typically requires big amendments of your planning artefacts including the Vision Board and the Product Canvas. If your strategy is valid, persevere and refine the appropriate documents.
Pivoting is never easy, as it require us to accept failure, and to let go of assumptions and ideas we may have grown fond of. But it is often a necessary step towards developing a great product. If you find failure scary, then don’t take it personal, and don’t identify yourself with your ideas: It is not you who has failed, but an idea or an assumption has turned out to be wrong. That happens even to the likes of like of Einstein who famously said:
A person who never made a mistake never tried anything new.
But if you keep pivoting repeatedly, stop and reflect. Check if you are really moving towards a successful product, or if you are chasing an ever-changing dream.
Learn more
While there is more to data analysis then I can cover in this brief blog post, I hope that the tips above help you analyse your data and create a great product.
To learn more attend my Agile Product Planning training course. Please contact me for delivering the training course onsite, and in form of interactive virtual training sessions.
