Data Myths: The Misconception of Intuitive Decisions

A lot of the discussions I hear about data and analytics revolve around what and how to measure, and many interesting startups deal with creating new data sources. We deal with clicks, interactions, graphs, heat maps and surveys. We look at networks, assess nodes and links, and analyze service providers and browser information. We create masses of (often useful) information – but what do we do to organize and make sense out of it? While measuring and tracking is important, excess data can drive people to either give up on using it completely, or turn to use complex, sometimes very unfriendly analysis tools that require a lot of effort and ramp up time.

The most common claim by those who give up on using data is that talent or experience replaces data with “intuition”, and the rest of us should succumb to the wisdom of those who have good intuition. Indeed, it is mind boggling to work with highly talented people that seem like they can make correct decisions in a split second, without really being able to articulate their decision (“I just know!”). But what is this intuition? Actually, it is far from something supernatural. As discussed in research, intuition is a result of micro-learning that one might not be able to articulate, since it differs from standard and identifiable learning setups (read more in Matthew Lieberman’s paper here. Careful, PDF for download!). We learn from example but often unconsciously, and those result in intuition that seems to transcend logic.

Furthermore, since intuitive decisions are usually taken under stress, they often have a positive effect in preventing decision biases that arise when you rationalize or over-analyze your decision. I really like the Cook County Hospital example from Blink since it’s a great case of how a succinct procedure, applied by experts, removes the potential bad effects of excess data and over-thinking. And lastly, like in everything else, there are people who are better at this learning than others; they “see the matrix”, so to speak, and understand patterns better than the rest of us. But intuition is hard to quantify, and finding people who can both understand patterns and articulate them in a way that makes sense is very, very hard even for experienced modelers. Getting the “I just know” mantra is much more prevalent than finding an expert you can use, and the result is that such real intuition is often either lost or applied only by a few that are well capable, if they are lucky enough to get into influential roles.

How do you find the right people do approach data “intuitively”, but at the same time be able to articulate what they understand? I suggest you start with your customer service reps. Generally speaking, if you want to learn about customer behavior you talk to the people who talk to them on a daily basis (and make sure that all of your people do) – obviously – but day to day interaction with users causes reps to develop keen insight, intuition, as to what this customer will do next. Granted, not all of them get it and certainly not all of them can translate that into actionable patterns – but some do. And those that do are your key to making sense of data quicker and in an actionable way. Translating this knowledge into automated, actionable insights, however, is a completely different issue.