Friday, August 25, 2006

On Common Event Format (CEF)

So, Common Event Format info is out. Do I like it? Do I think it sucks? Does it matter? Here is what I think.

There was a lot of talk about such standards. On our corporate blog we said that "Mary-Ann Davidson, CISO of Oracle, has been promoting an audit log standard for years. Others include a spring initiative by NIST to launch Common Logging Interchange Format. SANS deserves credit for picking up the ball where NIST left off. They brought together a wide range of users and "loggies" to debate standards at the recent log management summit [where I presented as well]. And, Amrit Williams from Gartner also published on the topic - such as his May 2006 Gartner publication #G00139205 on log output standards." And there are always IDMEF (largely RIP), SDEE (dormant? dead?) and WELF (not too relevant and not going anywhere fast) ...

Obviously, I'd also prefer something more vendor neutral, but, what is more important I think we should first go for a content, not format standard like CEF. I usually like to structure it like this - one can standardize logs in:

  1. Format
  2. Transport
  3. Content

IDMEF? CSV? CEF? WELF? You are talking formats here; slashes, dashes, pipes, spaces, tabs and name=value pairs.

Syslog UDP 514? SQLNet? HTTPS? This is about transport, how log bits travel from there to here and back :-)

How about content? There is really no credible log content standard that defines what is in the logs (and what it means) and not how they are formatted, apart from a long dead and forgotten CIEL. And that is where the greatest need is (and has been for a while)!

I can spew XML and CSV just as well as the other guy, but this is not the problem that people our there are facing. Our log parsing capabilities allow us to chew through any of the current log formats, admittedly, with varying degrees of difficulty. However, understanding logs, whether parsed or free-form English is still a challenge for security and operations folks. If some industry group can standardize the log content, the world would be a much better place. And SANS did pick up this challenge so there is hope that this will finally move forward.

So, as a result, everyone would be able to understand, just by looking at each log record, whether it relates to access failures, system changes, attacks, service restarts, system rebooting, account creation, etc via a clear and open log taxonomy, and to do all that without diving into thousand-page documents.

Can it happen? Yes, if we make it happen!

tags: , , ,

3 comments:

Anonymous said...

Well said! We've pushed for a standard event classification/identification system for a looooong time. Maintaining a textual representation of events, once events are being handled at an automated level (realtime, in aggregate, over-storage and processing-resources) is asnine. The same amount of work that has and will continue to go into log entry analysis/regexp/rules/etc to relate what the message is trying to say could/should have been spent on organizing and tagging types of events for ease of relational analysis, and reduced storage. Do it once, maintain as required (which I would have to think would be less than system/application level translations/interpretations), and spend time on writing worthwhile alarm traps and reports. You can always get a textual representation back for viewing purposes - given a system or app-level event id.

Unknown said...

I don't think you've read the CEF definition. The Common Extension Dictionary is an attempt to streamline the log content amongst vendors. It's exactly talking about that : content.

But your problem is that if all vendors agree on the same format (and content), your company (LogLogic) won't earn as much money.

Anton Chuvakin said...

>But your problem is that if all
>vendors agree on the same format
>(and content), your company
>(LogLogic) won't earn as much money.

OK, I will address this point in a separate blog post since it is common (but still absurd!) to think that log management and SIEM vendors make money off non-standard logs.

In reality, we will benefit the most from log standards - but watch for a new post on this very subject.

And, come on, CEF is NOT about content! Go and ask Raffy, he wrote it, for gods sake.

Dr Anton Chuvakin