Too much data, too little information

So, you have this big 1000 user system, with its flows and checkpoints and flags and pointers. If you've grown it well you have a dashboard showing you login numbers, counts of transactions, dollars moving around. You control it all from your NOC, pressing the little red buttons whenever necessary, moving dials and reading graphs. But the thing is, that seeing the bits and pieces of online life on your screen doesn't necessarily, and sometimes doesn't at all, help understand what's going on.

What IS going on in your system? What are users doing, and will that translate into the bottom
line? What can the numbers tell you?

Well, we've been through a few ideas. Experts knowledge ties symptomatic indicators with identities and with what they intend to do, so that you can at least start making sense. Collecting the data is one aspect, and using it to understand is a whole new area. When we reach tips and tricks on how to develop your own methodology, some of this might start ringing a bell. But this post is about one system that shouldn’t be adopted as your main tool if you’re the risk management expert – it’s about advising you to not count on hindsight based on business results.

No, no, don’t get me wrong – business results are important, one of the most important aspects of the business (and some will argue – the single most important – but that is another discussion). But using the bottom line (or even a highly detailed version of it, including a drill down of, for example, every auth rejection code) to indicate what the risks are in the system or worse yet – to indicate what needs to be fixed – is a call for bad judgment. Consider my favorite example, a hospital. If you needed to weigh two hospitals one against another, would you use the percentage of deceased patients as an indicator? Would it matter that one has an oncology department and the other doesn’t? Would it matter that one is in Mozambique and the other is in Mexico? Of course it would, since when all else is equal (in staff, training and tools – like your company compared to other retailers), fraud-on-entry (the hospitals’ location and the indigenous diseases you’d expect) and fraud MOs (the types of diseases that are actually seen and treated or not treated) have a big impact on the bottom line. Trying to use the numbers post risk controls, chargeback, CHB dispute and collections to understand what could have happened is trying to pin down a moving target – and the wrong one at that. Worse of all would be trying to design future systems based on the current snapshot, since you do not have any indication of what users do – just how much money it costs you, and user behavior is much more volatile than your incoming chargeback count.

When you come to understand what’s going on, business results are highly important. But letting them steer all of your team from looking at user behaviors will put you exactly where you don’t want to be – patching up holes in your system using a highly delayed hindsight mode. To be successful, combining data analysis and behavioral research is a must.

Who are these guys?

The investigators were baffled. After 3 hours of investigation, they still haven't made any progress in understanding who should they be looking for as the prime suspect for the assault case. The problem? No, not a mismatching DNA sample. Not a picture that's not on the immediate suspects list. Not even scarcity of able people willing to crate a drawing based on the description from the victim. The problem, suprizingly, was that the victim would not let out any revealing detail about their assailant: gender? against the sexual harrassment act. Skin color? Dude, we're against any type of discrimination. Religion? get out of here. Lucky for the investigators, the guy (oops) was of average height. At least that went through.

Imaginary? Indeed. Possible? Of course. I once spent 15 minutes listening to a friend of a friend describing a very similar case, until I was able to understand what the person's profile was. Because in social interactions PC sometimes deters us from using specific observations. Makes sense. In the world of Risk management, however, such a starting point can be the blow of death to your ability to understand what exactly is attacking your system, and stop it.

Profiling is the name of the game, and some of us are not playing it, and are wrong at doing so. Because fraudsters lie every time they need to. They lie about their identity, they hide their connection, they use other people's details and they will come back to demand the service you are not giving them and might end up convincing you. But what's their motivation? Is a WFH scammer the same as a 419 fraudster or a WOW gold trader, or for that matter - a cusotmer that maliciously reports not receiving an item they have in fact received? Of course not. They have different starting points, different sets of tools and conceptions, they might even be from completely different regions of the world (quick hint: they are). And that renders behavioral attributes that either are not reflected in your analysis (beacuse the fraudster's age and favorite social network do not reflect in the account time-on-file or time before a Chargeback comes in).

When not profiling, you are bound to looking at losses as they appear, and then reverse engineer them using business dimensions to try and understand what going on. You might discover that your UK market for new intangible item transactions is high on chrageback rates. Is this a bad finding? Absolutely not, data driven analysis HAS to be the first step of any research - because segmenting the world is the first step toward prioritizing work and creating a souns results-related risk policy. But whan you don't ask yourself "why is this happening" and "who are these users causing losses" and even "what's their story?", you are missing on three big things:

1. the ability to further segment the world based on how bad user behaviors look in your system, and differentiate malicious intent from system errors (classification errors and others) and mistakes (the human factor - flakes, friendly fraud and others)
2. the ability to identify the good guys, and provide them with better treatement, even when they resemble bad guys in business segmentation
3. the chance of foresight - understanding where the bad guys might go next

Behavior based analytics isn't the sole answer to all BI problems. On the contrary - without a proper data driven segmentation, experts' intuition is both invalidated (and though usually is useful, is risky when it's the only thing you're using for long term planning) and will take a lot more time to create (since prioritizing where to look first is the proper use of your Oracles). But it is the single most important frame of thought your risk management team is probably not using - and whether this is happening because of PC, lack of domain expertise of just disinterest, you cannot let it pass you by.