The tech industry prides itself on being “data-driven”. We’re so data-driven, in actual fact, that there are a entire bunch of startups building analytics tools (Section by myself lists over 100) and A/B attempting out tools (~30). We each and every snicker at but also secretly savor stories savor Google A/B attempting out 40 shades of blue for its hyperlinks. A unparalleled person-product tech firm might perchance well per chance also merely be running anywhere from dozens to hundreds of A/B tests concurrently, and inspecting hundreds of metrics to concentrate on what to launch.
At the floor, it makes sense. “Records doesn’t lie”, we’re urged. Records, we’re promised, will relief us overcome our luxuriate in a huge sequence of cognitive biases. This is in a position to per chance well gash through team and firm politics. Searching on who you question, data is the fresh oil, data is the fresh gold, or data is the fresh religion. Every product decision you invent, every OKR you attach—ought to be grounded in data. “You would’t argue with the information!”
I’ve labored at a couple of person internet corporations, and I’ve considered it first hand. I joined the cult and drank the Kool Encourage. And, I love data. I’m an engineer at heart. Records works. Nonetheless savor any religion, data might perchance well per chance also be taken to the unparalleled, with awful consequences.
So I’m no longer announcing we ought to throw the miniature one-data-Messiah out with the bathwater. All I’m announcing is that data is a tool, and you ought to exercise it as such.
Accept as true with you’re a product manager at a person internet firm. Your activity is to construct a touchdown internet page to secure users to register on your product. So you attach a lot of precious data on that internet page. The conversion rate is low. You scoot an A/B test with a bunch of adaptations, and you see that withholding severe data boosts the signal-up rate. You scoot more A/B tests. The connection holds. Much less precious data, more signups. Outdated to your perceive it, you’re a burly-fledged touchdown internet page click-bait artist. Your internet page is shit but you nailed the conversion rate!
“Wait a minute,” you’re announcing. That is a self-discipline that might perchance well per chance also be solved with more data. And yes, it’s seemingly you’ll per chance well be ready to beginning up measuring downstream metrics savor retention, and so on, and most definitely you be taught that tricking your possibilities into signing up by withholding data ends in decrease retention. Nonetheless now you’ve shifted the matter downstream, and what is going to seemingly happen is that you just (or any other product manager) will now be tasked with rising the downstream retention, and yet again, the information guides you towards more dusky patterns. Because your entire funnel is now grounded in dusky patterns. And now any time you absolutely strive to convey precise worth to users, your metrics fall.
If this instance sounds cartoonish and laborious to judge, I swear you I’ve considered it (or one thing same) happen a couple of cases at very legitimate corporations. We have to note that data isn’t any longer a change for anything. It’s no longer a change for thought your possibilities and their problems. Records isn’t any longer a change for supreme judgment. Records can if fact be told turn into a crutch that gets in the potential of self-discipline-solving. More data can lead to data hoarding and choices to the detriment of your possibilities, your product, and your firm.
Records also ends in huge, monopolistic person internet corporations that have misplaced explore of the matter they’re seeking to resolve and in its attach lawful need to select their metrics. It also ends in upset workers. You exit and hire the smartest, most passionate other folks it’s seemingly you’ll per chance well be ready to search out, and flip them into A/B attempting out monkeys. Before the entire lot, they admire it—they create adjustments, they survey numbers bound up. They secure promoted. Nonetheless they flip off the section of their mind that cares or thinks seriously. Records isn’t any longer a change for motive. Care for all shallow gamification, the pause at final wears off.
Use data as a tool. It’s miles extremely efficient. Don’t exercise it as a religion. Work with other folks and corporations who realize that. Work with of us which might perchance well presumably be if fact be told centered on solving a self-discipline. Use data to validate the matter and the alternatives, but don’t let it lead you blindly.