So, when I posted this blurb on anti-virus missing malware, I didn't mean to whip people into a frenzy. I really didn't - I just wanted to express my genuine shock about how poorly the tools, built for blasting away the threats of the 90s, fare against the threats of 00's. In fact, I myself naively thought that a typical AV tool will catch 60-80% of serious in-the-wild malware today. Some of my readers were surprised by the numbers and some were not, stating that it matches their experience as well. Many probably choose to stick to "my anti-virus is fine, go away!" illusion.
It is also bizarre how some people chose to interpret my blog post as biased: "i saw where this was going early on (the original question was obviously loaded)." I would like to assure them that while I did state my initial question in a somewhat emotional manner, this was not due to any inherent bias I might have had, but due to my deep surprise. I myself hate people saying things like "today was a hot day -> obviously global warming is here" :-), but in this case what matters is not "statistical significance", "sample selection bias" or "test-bed integrity", but the fact that if you deploy anti-virus on your systems and run it according to the "directions on the label", your system will soon "change hands" :-) This doesn't point to any global emerging trend, but just to a fact, observed by the author of the study (which, BTW, I just read, not conducted myself...)
I later learned that a major analyst firm, that will remain nameless for now, proclaimed in their recent piece: "By 2009, anti-virus as we know it will be dead, succeeded by a new generation of protection technologies, and many of today's anti-virus vendors will be extinct."
Some folks have asked me a sensible question: what is the alternative? At this point in time, the alternative for most people is fairly unpleasant: you are going to get 0wned :-) Go update your incident response (IR) plans and sharpen your IR skills. Learn to detect 0wned systems.
Over the long term, I am willing to bet on some fancy "whitelisting" approach (e.g. this) or novel heuristics (e.g. here) or something else (e.g. here), which is still being forged in secret underground labs of some nameless security start-up :-)
Overall, it seems that "classic" (e.g. "blacklisting") anti-virus technology does indeed work as stated by its purveyors. It is just that modern malware no longer does ...
13 comments:
"It is also bizarre how some people chose to interpret my blog post as biased"
of course it was biased, you'd already made up your mind... most people are biased in that sense, but that's not the real issue...
"but in this case what matters is not "statistical significance", "sample selection bias" or "test-bed integrity""
it amazes me that you seem to genuinely not understand the implications of the methodological problems i underlined... to put it bluntly, the results don't mean what you think they mean (and in fact don't mean much at all)...
the self-selected sample alone is enough to rob it of any deeper meaning than the obvious fact that anti-virus products don't catch all viruses... you might as well take an ad out in the paper asking for volunteers in a study about illiteracy (given that those who are illiterate won't be able to read the ad)...
only sampling malware from compromised systems where there already was protection in place means that the tester was not using sample that represented the set of malware in the wild but rather s/he was using a sample that represented the subset of malware capable of bypassing the protection on those systems...
"but the fact that if you deploy anti-virus on your systems and run it according to the "directions on the label", your system will soon "change hands""
this may come as a surprise to you but the same can be said about all malware prevention techniques/technologies... no single technology is able to protect you from everything... multi-layered approaches have been the proscribed way to go since the last millennium precisely because for each technique there are classes of things which that technique cannot handle...
>of course it was biased, you'd already made
>up your mind
Huh? I saw the results, got very surprised and blogged about them. Where is the "making up my mind" part? The results were certainly contrary to what I expected...
>only sampling malware from compromised
>systems where there already was protection in
>place means
That is actually a very good point, which I admit to have totally missed :-(
>this may come as a surprise to you but the
>same can be said about all malware prevention
> techniques/technologies..
No surprises here - I totally agree with that!
However, the issue is that AV is now sold as "anti-malware" and people EXPECT it to stop most of currently in-the-wild damaging malware (exclude adware and other low-risk stuff). In reality, it does not seem to...
"Huh? I saw the results, got very surprised and blogged about them. Where is the "making up my mind" part? The results were certainly contrary to what I expected..."
you saw the results, took them at face value, and incorporated the conclusion into your world view... and then you wrote a few blog posts about it...
"However, the issue is that AV is now sold as "anti-malware" and people EXPECT it to stop most of currently in-the-wild damaging malware (exclude adware and other low-risk stuff). In reality, it does not seem to..."
does not seem to according to which metric? according to this particular test whose results were (intentionally or unintentionally) cooked? the subset of itw (in the wild) malware used in this test suggests anti-virus products don't do very well, the subset of itw malware used in certifications like vb100 say the exact opposite... without a truly representative sample of what malware is in the wild we have no objective way to judge whether anti-virus products are effective against itw malware or not...
that said, the range of detection rates for this test bear a remarkable similarity to the range recorded at av-comparatives.org in their retrospective testing - which is designed specifically to test av products against viruses they haven't been updated to detect (which leads me to once again suspect the samples were too new - not that new malware isn't in the wild, but there's a heck of a lot of old malware in the wild too)...
Wow! You said it best
>which is designed specifically to test av
>products against viruses they haven't been
>updated to detect (which leads me to once
>again suspect the samples were too new - not
This is EXACTLY where the point is!!! A threat model changes (and took AV with it):
a) 1999 - if your AV is out-of-date, you have only yourself to blame; make sure you update it ASAP and then you are protected!
b) 2007 - your AV is out-of-date, and you just updated it! There is A LOT of malware that your AV vendor haven't even seen and has no signature for!
Thus, saying "testing outdated AV is dumb" was absolutely true in 1999, but in 2007 - ALL AV is out of date!
">which is designed specifically to test av
>products against viruses they haven't been
>updated to detect (which leads me to once
>again suspect the samples were too new - not
This is EXACTLY where the point is!!! A threat model changes (and took AV with it):"
umm - i think you've missed the point entirely... retrospective testing is done for the sole purpose of measuring how effective heuristics are... it has nothing to do with changing threat models (new/unknown malware has been an issue for a long time), it's simply a way of constructing a test to test a specific part of an anti-virus product...
"b) 2007 - your AV is out-of-date, and you just updated it! There is A LOT of malware that your AV vendor haven't even seen and has no signature for!
Thus, saying "testing outdated AV is dumb" was absolutely true in 1999, but in 2007 - ALL AV is out of date!"
congratulations on twisting the meaning of "out of date" to suit your argument... in reality, "out of date" means the same thing in anti-virus software as it does in every other type of software - specifically that there is a newer version available... if you just updated then you cannot be out of date, by definition...
there has always been malware your av hasn't seen yet and there will always be malware you av hasn't seen yet... if the existence of such malware is supposed to kill anti-virus then anti-virus should have been still-born... is the volume of such malware increasing? yes... is the window of opportunity such malware exploits shrinking? yes... should you be using complementary technologies to deal with that particular subset of malware? yes... should you have been doing that since the beginning? yes... has the technology been there in the past? yes, even in the (relatively) distant past...
is the "anti-virus-is-dead" mantra the product security faddists who've decided it's time to find the next big thing? yes... malware prevention is about building layers of defenses, not playing musical technological chairs...
>is the "anti-virus-is-dead" mantra the
>product
>security faddists who've decided it's time to
>find the next big thing? yes.
No, sir!
"AV is dead" is not the faddist mantra. This is a mantra of people who spent years paying to a "Big 3" AV vendor for "protection". And who were, indeed, protected for a good number of years. But not anymore...
This is also a mantra of people who's fully patched boxes with updated AV are owned by stock malware.
And all they hear from their AV vendors is "ah, its not really a virus, its a Trojan." Come on (emotional argument alert! :-)), will your grandmother care about such arguments?
So, to summarize:
1999 - run and update AV -> [reasonably] safe from malware
2007 - run and update AV -> owned by malware and have financial info stolen
The latter paradox is what I call "AV is dead."
"No, sir!
"AV is dead" is not the faddist mantra. This is a mantra of people who spent years paying to a "Big 3" AV vendor for "protection"."
please tell me you didn't just follow up a faddism denial with a supporting argument that invokes the big (as in biggest market share aka most popular) 3... or at least tell me you didn't have a straight face when you did it...
"This is also a mantra of people who's fully patched boxes with updated AV are owned by stock malware."
stock malware? when did stock have it's meaning changed to "new/unknown"?
"And all they hear from their AV vendors is "ah, its not really a virus, its a Trojan." Come on (emotional argument alert! :-)), will your grandmother care about such arguments?"
i haven't heard an av vendor make the 'we detect viruses not trojans' argument for many years... all mainstream vendors have now included non-viral malware in their focus...
"1999 - run and update AV -> [reasonably] safe from malware"
highly revisionist... perhaps you just weren't paying attention to the malware domain back then, but av failed frequently enough then for it to be clear that av should not be solely relied on....
"2007 - run and update AV -> owned by malware and have financial info stolen"
with the exception of the organized crime aspect, this was equally true in 1999...
"The latter paradox is what I call "AV is dead.""
the fact that people are only beginning to see the limitations of anti-virus now and mistakenly think it's something new is what i call a fad wearing out...
>the fact that people are only
>beginning to see the limitations of
>anti-virus now and mistakenly think
>it's something new is what i call a
>fad wearing out...
Well, if it is indeed true, then the correct "mantra" is not "AV is dead", but "AV was dead" :-)
I still think that a chance of being owned while running and updating AV is HIGHER now than in 1990s. If this statement is NOT true, then I would, in fact, fully agree with you ...
well, if you want to quibble over degrees of risk then sure, it's probably somewhat higher now than it used to be... but it was high enough back then...
i say "probably somewhat higher", by the way, because despite there being more malware in the new/unknown category now, most of it is non-replicative which inherently lowers the risk of exposure to any given piece of malware, especially in the long term...
This is where we actually come to an agreement, finally :-)
>well, if you want to quibble over
>degrees of risk then sure, it's
>probably somewhat higher now than
Well, actually, a degree of risk is kinda the whole point here (not so much the "uselessness" of "death" of AV)
>i say "probably somewhat higher",
> by the way, because despite
>there being more malware in the
>new/unknown category now, most of
> it is non-replicative which
>inherently lowers the risk of
Ah, this is 90s thinking speaking. Today, non-replicative malware is all the rage - meaning that a typical IT user is much more likely to be hit by non-replicative malware.
"Ah, this is 90s thinking speaking."
yeah, right.... or maybe it's my ability to separate aggregated risk (the one that's probably somewhat higher) from per-instance risk (the one that's inherently lower) that was speaking...
clearly you didn't like my last comment enough to publish it so perhaps you'd prefer to hear straight from the folks at virustotal what they think of using virustotal as a tool to perform comparative analyses...
The free DropMyRights program is an excellent addition to the arsenal for protecting a Windows XP machine. You can use it to run the obviously dangerous applications with lower rights.
http://www.cnet.com/defensive-computing/8301-13554_1-9756656-33.html
Post a Comment