The “right” data — An analogy to explain the importance
This is a well-known image; the backstory is familiar too. This story exposes what is known as the superior bias while looking at data and analyzing them (superior bias is a situation where we look for data of situations that succeed whilst conveniently ignoring those that cause failures). This story goes back in time, during World War II; many British bombers were shot down by the German and then the ones that came back had severe damage. The war-time Data experts/researchers painstakingly analyzed each and every bomber that got back, analyzed the damage caused due to the German attack, and reinforced them to ensure successful operation (next time around).
Mr. Abraham Wald, a Hungarian statistician re-analyzed the data that was put together and unearthed the superior bias. Whilst, the researchers analyzed the bombers that got back, what about the ones that did not get back? Were reinforcements done at the right places, were actual vulnerable areas missed by the analysts? Truth be told, YES.
With this story as our backdrop, Let’s analyze the importance of “Right” data in product development. As Product managers/UX experts/Definition experts/GTM experts, we rely heavily on data to take the right decisions (Sometimes aided by human research and sometimes supported by machine algorithms); here are few examples of our dependence on data:
- Product managers: How do we decide the next big function/feature in the product? How do we analyze product consumption data to arrive at a data-driven product definition?
- UX experts: How do we change the experience of a feature or a function? Our rationale can take us some distance but not the whole journey?
- GTM experts: How do we decide to upgrade a customer to a premium package or how do we decide when to run a promotion or how do we determine the right discount?
In all the situations mentioned above, data plays a critical role in the decision-making process, and analyzing the right data is imperative.
Let’s look at situations (1) & (2); product managers and UX experts rely heavily (if not, must rely heavily on data) to define the future course of the product (from a feature/function and experience standpoint). How can they go about doing the same?
The first version of the product is a coming together of the product manager/UXer rationale and some science. What about the next version? Can rationale alone help? Absolutely not, that’s where the “ Voice of Product User” plays a major role and must not be ignored. “Voice of Product User” enables product managers and UXers to gather, aggregate, analyze, and follow through on received feedback. Like the British bomber example in our storyline, the “ Voice of Product User” could be misleading leading to incorrect interpretations. I have tried to solve this conundrum by doing the following:
- Step 1: Collate data from various feedback sources.
- Step 2: Create “ Feedback Cohorts”, coming together for feedback items with shared characteristics and contexts (sample cohorts could be “Adoption”, “Acquisition”, “Renewals”, “Data and insights”, etc.).
- Step 3: Try to suppress the superior bias by performing qualitative user research of various items (at the cohort and item level). This will enable you to separate the noise from the signal.
- Step 4: Reorganize cohorts and associated items based on suppressed superior bias.
- Step 4: Rank cohorts based on product and market vision.
- Step 5: Triage items within the cohort to create an execution plan.
I am sure there are many more approaches to achieve accuracy. The most important point here is to recognize the fact that whilst data is important, data can also mislead.
Originally published at https://www.raghsforte.com on February 20, 2021.