A long, long, long time ago (eh, like, in 1997 :-)) Marcus Ranum said in his post "artificial ignorance: how-to guide": "... you build a file of common strings that aren't interesting, and, as new uninteresting things happen, you add them to the file." And then you watch for other log records, which are then presumably interesting. This introduced the concept of "negative" filtering i.e. looking NOT for what you know to be "good."
This technique is indeed extremely effective for finding "interesting" (i.e. unusual + actionable; see more, for example, here [PDF]) events in logs. I even wrote a few prototype tools myself - including filters such as "new log record type for this port", "new for this network segment", "new for this sensor", "new for this IP range", etc or even two-dimensional ones such as "new for this port and destination IP address" which worked really well. All approaches had adjustable timeout after which the tool would consider an event to be new (e.g. "new for this week", "new for this month", etc). However, few vendors chose to incorporate such analysis approach in their products (bizarre, isn't it?).
And now, 10 years later (!) somebody finally did. In his post on 'Finding Events that have "Never Been Seen" Before' Ron Gula (with Marcus Ranum standing over his shoulder, no doubt :-) ) talks about implementing this in his product. What took you so long? :-)
They say "Regardless if your event logs are from UNIX systems, router access control violations, wireless access DHCP logs, intrusion detection systems or so on, after a certain period of time, the same events tend to repeat themselves. This is because most of our networks run controlled and automated processes. With this in mind, finding out when something "new" occurs could indicate a security or administration problem. " Right on! There is more fun stuff in the follow-up post 'More on "Never Before Seen" Log Events'.
4 comments:
On commercial products this NBS style analysis can be new, but ossec is doing that since 2004. It allows alerts to be generated on new IDS, authentication events and etc. You can also combine them with srcips, destips, usernames, etc.
*eg: First time user A logged to system XYZ. First time IDS event B was triggered against system C.
*On ossec, we call that FTS (first time seem).
Thanks,
Daniel Cid
That is really COOL! When I looked at OSSEC last time (maybe in 2004), I definitely missed it there.
That is indeed awesome!
A really long time ago (like around 1990) a similar idea was applied to virus detection (there was at least one popular antivirus program which used it - I do not remember its name). It made use of regular monitoring of file system on a PC and detecting changes in files (and in other parts of the file system) for which modification is unusual or not supposed to happen. For some reason, this idea did not become popular (not for Windows platform, at least).
Yeah, sad to say, but it didn't work too well in this realm - there is way to much stuff which is legitimated + new....
Post a Comment