Dr Anton Chuvakin Blog (Original)

Security writing since the mid-2000s. New posts are mirrored from Anton on Security on Medium.

Sad Hilarity of Predictive Analytics in Security?

By Anton Chuvakin
Historical Archive Note: This post was originally published on the Gartner Blog Network on March 31, 2016 by Anton Chuvakin.
Original URL: http://blogs.gartner.com/anton-chuvakin/2016/03/31/sad-hilarity-of-predictive-analytics-in-security/ | Archive.org Snapshot

After spending a week in Siberia, I am ready for more fun blogging – and of course for more drama that is our industry (GO CYBER DRAMA!). In any case, the topic is PREDICTIVE ANALYTICS in SECURITY: What is it? Can we have it?

What is it?

First, I’ve encountered a few “false predictive” examples like “if you don’t patch your Windows for 3 years, I *predict* that you will be hacked” (common sense predictive) and “if I see this attack first, I can tell you and thus you can *predict* it in your environment” (my past is your future predictive). Frankly, I don’t think this is real prediction, like predicting the weather. Finally, some people treat other types of analytics as predictive: for example, they treat some UBA / UEBA tool output “user jsmith maliciousness score is 73%” as 73% that this user will go bad… To me, even guessing the likelihood of some event isn’t truly prediction, since the event will either happen or it won’t [update: OK, maybe I was too harsh here, but simply saying some known outcome can happen with X% probability is often unimpressive predicting].

What is real prediction then? I’d rely on Gartner definition of predictive analysts that “create models that anticipate future behavior, or estimate unknown outcomes” (here; also see this). In other words, it better be about goddamn future! (“Prediction is very difficult, especially about the future.” – maybe Niels Bohr)

Can we have it?

Depending on the definition, the traditional reason for why not (= because there is a smart, evil-intentioned and possibly psychopathic human on the other end of the wire) may not apply since predictions of customer behavior (e.g. see this) has been done for a while. Still, if you predict the future activity based on the past record and some context, you are bound to be wrong in some cases, especially when a) you don’t have enough information about the subject of that activity – the adversary and b) said subject is actively trying to avoid being seen.

Frankly, I’ve seen idiots and their little idiot AIs make judgments like “because 100.0% of DDoS you were hit with so far was ICMP, we predict that your next DDoS will be ICMP.” And then – BAMM! – a massive DNS flood blows them away…. And this example does not even start to take into account any active deception and misdirection by the attacker. On the other hand, the past clearly tells us something about the future, and thus maybe there is some hope here [“Organizations that use traditional predictive analytics [tools] for security? You mean ‘both of them?’” (source)]

There you have it – what do you think? Can we have real [clearly defined] prediction in information security?

P.S. Now back to our EDR research … much excitement is coming! BTW, our updated IR paper is in editing already, but if you have any fun last second comments on this, please send them over.