Friday, September 14, 2007

Once More on Failure of Academic Research in Security

Many people, myself included, have bemoaned the complete failure of academic research in information security. The main reason for this is a complete disconnect of academic security research from real-world threats and vulnerabilities (e.g. I still see people publishing papers inventing signature-based network IDS systems, reinventing MAC/RBAC, neural nets to catch hackers, etc - and if I hear about the Lincoln labs 1998 intrusion detection data set again, I will screeeeeeeeeeeam! :-))

This fun post brings a few more examples. It gets to "Security against real threats is the point where scientific integrity, method and rigour unravels" and continues to "The academics have presented stuff that is sometimes interesting but rarely valuable. They've pretty much ignored all the work that was done before hand, and they've consequently missed the big picture" and even "The academic world of security is simply too far away from their subject. " (which all ring sadly true)

What is much more exciting is that there is finally an explanation for this "induced stupidity" phenomenon, that puzzled so many in the security field: "Academic work is only serious if it quotes other academic work. The papers above are reputable because they quote, only and fulsomely, other reputable work. And the work is only rewarded to the extent that it is quoted ... again by academic work. The academics are caught in a trap: work outside academia and be rejected or perhaps worse, ignored. Or, work with academic references, and work with an irrelevant rewarding base."

You know what? I think the above does explain it!


kurt wismer said...

i think there's some wrong-headed assumptions in that article and i suspect your agreement with it indicates an agreement with those assumptions...

what exactly do you expect academia to be able to do with respect to security problems like phishing? are you expecting them to solve the problem? do you expect they at least have the capacity to solve it if they freed themselves from the constraints that are holding them back?

consider the possibility that the problem is not actually solvable...

are you expecting academia to be able to bring more to bear on at least mitigating the problem than big business can? why? are academics magical? are there powers of logic and deduction that can only be held by an academic and are lost once the person is recruited by a security vendor?

consider the possibility that academics are just people at the end of the day and don't have the intellectual super powers outsiders would ascribe to them...

are you expecting everything academia produces will be useful and practical?

aside from the fact that that would be a highly comical expectation given the definition of academic, consider the apocryphal story of alexander graham bell taking a different view of his many failures to create a light bulb... academia has generally done a pretty good job at discovering what won't work, and by extension what limitations we face...

Augusto Barros said...

Must say that you are completely right again. I'm finishing my Master thesis (about detecting insider threats by log correlation) and I'm having a very hard time to find "academic works" useful enough to be cited and used as base for my work. I have tons of good ideas from MJR, for example, but almost everything from him are forum messages or informal articles. It's funny that citing him will be badly seen but anything from the "academic world" will be accepted.

By the way, if you know good "academic content" about the subject...

Anton Chuvakin said...

I don't really know how bad is academic research in phishing (I am relying on the original blog post author for that). However, I read countless academic papers on intrusion detection and my expectations were that:

a) useful thing show up in commercial solutions
b) useless things will stop being researched.

Neither a) nor b) seem to happen all that often...

Also, I expect these same links to work in sec:

physics academic research -> new technology
biology academic research -> new drugs, etc

and not

infosec academic research -> either trash bin or purely humorous matter

>are you expecting everything
>academia produces will be useful
>and practical?

God no!!! But I expect that the percentage will be decent ...

I think this justifies a longer post, so I will shut up now and revisit it later.

Anonymous said...

The disconnect exists because infosec is not a discipline (yet) in the academic sense.

Hence, you have people in academic infosec who are computer engineers, computer scientists, EE people, physicists, economists, etc.

I needn't explain to a physicist that disciplinary boundaries exist to satisfy the human need to feel that all problems may be understood if not solved using the tools of the discipline one is a member of. :^) This is all well and good -- to a point, but it needs to be counterbalanced by actual practitioners who know what the nascent discipline needs to do to be relevant. Unfortunately, there is often a culture clash between these practitioners and academics, which stands in the way of what Dan Geer refers to metaphorically as "hybrid vigor" from developing. USENIX is as close as we have gotten, although like Bejtlich I think Ross Anderson and crew have plenty of real world cred.

Anonymous said...

It would be nice to have some help, academia has not provided too much so far. As Blaine Burnham says "we are in a position where we have two working mechanisms - one is the reference monitor and the other is crypto, and I gotta tell ya, it ain't enough. IDS was supposed to be the last resort not the first and only" and so on.

The "its perfect or its broken" mentality does drive me a little bats, but before we throw academia under the bus, the other areas in computers like programming maybe did not get so much help either from academia and yet they deliver stuff all the time - Java came out of Sun, C came out of Bell Labs, C# came out of MSFT, CORBA came out of committees, and so on. Maybe waiting on academia is the wrong answer and it is up to all of us in the marketplace to come correct and build better stuff.

Andy Steingruebl said...

All the same if I go back through ACM, IEEE, and Usenix conferences for the last 10 years I'll find a number of interesting academic style papers that border on the purely theoretical that have turned into real shipping products and/or open-source solutions to things.

- Application systems call profiling
- Analysis of CPU instructions necessary to support true virtualization

I can probably come up with a few more, but at least a few papers from 8-10 years ago have turned into real products in the last few, or so I'd claim I suppose.

Yes, there are areas that seem to not get good quality papers and real practical thought (IDS as you mention) but there are others that are useful.

In the phishing areas, and areas that require extensive testing of hypotheses, etc. the only people publishing any useful information in this area *are* academics. Look at 2006's paper from the Harvard folks on Phishing and the usability research they did. No one outside academia has the time and energy to do user behavior modeling at this level in the security field. Well, almost no one actually.

So, they do a few useful things anyway especially in the area of human behavior.

jbmoore said... has some decent theses published all the time. Thorsten Holz and Neils Provos show that practical IT Security solutions can come out of academia. Considering that securing college campuses' networks are nightmares, it may be that many academics have given up or the problem is not considered interesting enough. Consider the nooks subsystem that would offer increased driver reliability and possibly security to the Linux kernel has never been adopted by the Linux kernel community, yet buggy drivers cause more crashes than any component in an operating system, especially for Windows. Yet, no one cares. Microsoft hasn't built a nooks-like subsystem for Vista even though they know buggy drivers are their biggest headache. They don't remove tftp.exe from their default installs even though OEMS transfer the OS via imaging programs directly to the hardware, not via tftp. Then too, NSF funding is atrocious right now. Perhaps most IT security proposals aren't being funded because the grant money is being allocated elsewhere. Or, the cutting edge research is classified by NSA and the DOD.

Anton Chuvakin said...

Wow, such a great discussion! I agree that there WAS useful stuff produced (and app call profiling does fit the bill); I also agree about USENIX.

Still, I feel like seeing something more ... I dunno... mind-blowing :-)

kurt wismer said...

@dr anton chuvakin:
"Still, I feel like seeing something more ... I dunno... mind-blowing :-)"

which, to me, still sounds like you're expecting something preternatural...

Anton Chuvakin said...


Yea, I think you are right actually. I think it comes due to my physics background where science is 2-5 steps :-) ahead of the currently running technology, not 1-2 behind like [sometimes!] in infosec...

Dr Anton Chuvakin