On Free Log Management Tools

I completely forgot to repost my list of free log management tools to the blog from my consulting site. Here it is (original that is updated periodically):

This page lists a few popular free open-source log management and log analysis tools. The page is a supplement to "Critical Log Review Checklist for Security Incidents" that can be found here or as PDF or DOC (feel free to modify it for your own purposes or for internal distribution - but please keep the attribution to us authors). The log cheat sheet presents a checklist for reviewing critical system, network and security logs when responding to a security incident. It can also be used for routine periodic log review. It was authored by Dr. Anton Chuvakin and Lenny Zeltser.

The open source log management tools are:

1. OSSEC (ossec.net)  an open source tool for  analysis of real-time log data from Unix systems, Windows servers and network devices. It includes a set of useful default alerting rules as well as a web-based graphical user interface. This is THE tool to use, if you are starting up your log review program. It even has a book written about it.
2. Snare agent (intersectalliance.com/projects/index.html) and ProjectLasso remote collector (sourceforge.net/projects/lassolog) are used to convert Windows Event Logs into syslog, a key component of any log management infrastructure today (at least until Visa/W7 log aggregation tools become mainstream).
3. syslog-ng (balabit.com/network-security/syslog-ng/) is a replacement and improvement of classic syslog service - it also has a Windows version that can be used the same way as Snare
4. Among the somewhat dated tools, Logwatch (logwatch.org), Lire (logreport.org) and LogSurfer (crypt.gen.nz/logsurfer) can all still be used to summarize logs into readable reports
5. sec (simple-evcorr.sourceforge.net) can be used for correlating logs, even though most people will likely find OSSEC correlation a bit easier to use (or even use OSSIM below)
6. LogHound (ristov.users.sourceforge.net/loghound) and slct (ristov.users.sourceforge.net/slct) are more "research-grade" tools, that are still very useful for going thru a large pool of barely-structured log data.
7. Log2timeline (log2timeline.net/) is a useful tool for investigative review of logs; it can create a timeline view out of raw log data.
8. LogZilla (aka php-syslog-ng) (code.google.com/p/php-syslog-ng) is a simple PHP-based visual front-end for a syslog server to do searches, reports, etc

The next list is a list of "honorable mentions" list which includes logging tools that don't quite fit the definition above:

• Splunk is neither free nor open source, but is has a free version usable for searching up to 500MB of log data per day - think of it as a smart search engine for logs.
• OSSIM  is not just for logs and also includes OSSEC; it  is an open source SIEM tool and can be used much the same way as commercial Security Information and Event Management tools are used (SIEM use cases)
• Microsoft Log Parser is a handy free tool to cut thru various Windows logs, not just Windows Event Logs. A somewhat similar tool for Windows Event log analysis is Mandiant Highlighter (mandiant.com/products/free_software/highlighter)
• Sguil is not a log analysis tools, but a  network security monitoring (NSM) tool – it does use logs in its analysis.

For a list of commercial log management tools go to Security Scoreboard site. A few of the commercial tools offer free trials for up to 30 days.

Feel free to suggest your favorite tools and I will update the list!
Possibly related posts:

• Simple Log Review Checklist
• SANS Top 5 Essential Log Reports Update!

Next Career Post: “Gartner-heads” vs “Packet-heads”

Who do you want to be  when you grow up, “a gartner(*)-head” or “a packet-head?“

Huh!?

Over the years, I realized that even in our mixed-up field of information security there are essentially two paths (that is, provided you do choose to follow a path as opposed to just “dabble in security” or be an “I just work here” kinda guy…)

• Instead of starting from asking a question of “do you even need a path?” or “is security your career or your passion?”, let’s assume that it IS in fact your passion. It might vary in strength from all-consuming mental affliction to a mild case of “securitis? (or “securosis”, per chance? :-)) - but it is a passion nonetheless.

How do you plot your course through that passion without losing your mind and then switching to real estate  career (BTW, a real case I’ve heard of)? And how do you stay on your path without diffusing your efforts, losing focus and becoming “aware of everything and expert in nothing.” As I mention, there are two paths:

1. A path towards super-deep technical kung fu in one or very few related areas. It does not have to be exploitation (even though that is a popular choice), but can be about network packets, web app security, malware reversing or something even more fun (eh…logs?). This is what I humorously call “The Path of a Packet-head”
2. A path towards … well… let’s call it “strategy”, even though the word is heavily abused. This is where “CSOs-from-god” and good security product leaders come from. This is what I humorously call “The Path of a Gartner-head”

It goes without saying that suffering through a few hex dumps or through a few policy rewrites, does not put you on the path. And neither does reading an exciting piece from … well.. Gartner. I am talking here about a commitment to become one of the best in the field [BTW, I hate “be the best you can” theme – for many people it just means “you’d still suck”… but I guess that’d be an unamerican thing to say, so I won’t say it :-)].

But here is the trick – there is some MAGIC in carefully blending the two paths a bit. The trick is in NOT losing focus on your path WHILE blending in (but not dabbling!) something from the other path.

A simple example: if you spend 12 hours a day looking at the smoking guts of malicious software, try reading what some analyst firm wrote about the anti-virus market – even if it sounds a bit boring at first. Does it make sense to you (or not)? Does what they say match your experience?

An opposite is even more true: if you spend 8 hours a day writing policies and connecting pieces together into “a big picture”, why don’t you pick one of said “pieces” and look what’s inside? Does it have code? What does it do? Does it really work? And how do you know?

Thinking about things like that has a potential of moving you forward on your path, however counterintuitive it might sound. It will also give you career advantages without failing into the “generalist expert” crap….eh….trap.

At the risk of praising myself too much, only now I fully grasped the compliment somebody gave me a few years back “… you can switch from reading packets to reading Gartner in a second – and not even flinch” :-) Let’ consider this an inspiration for this post, nothing more…

(*) no offense to esteemed folks from Forrester :-)

Possibly related posts:

• Skills for Work vs Skills for Getting Hired
• A Myth of An Expert Generalist
• All posts related to career

Nobody Is That Dumb ... Oh, Wait XIII

Perhaps surprisingly, but “Information Security” magazine allowed me to restart my long-forgotten “Nobody Is That Dumb ... Oh, Wait” series. The last post in the series was a long time ago, so thanks to them we now have the  #13. Hurrah!

So, their latest issue has this brilliant piece of sheer idiocy:

Do you really need me to comment? Just laugh… TrendMicro gets a Silver Prize in SIEM category … WITHOUT EVEN HAVING A SIEM PRODUCT. And “reported dead a few times” Symantec SIM gets a Gold Prize, but that just gets filed under “insult to injury” category…

So, even though my subscription has expired, I just updated my address with them so that they can send me some of the stuff they are smoking.

Possibly related posts:

• Everything else labeled “stupidity”

Two Fun Presentations Today

Just FYI, I am doing two fun PCI DSS presentations today:

#1 LogLogic’s PCI 2.0 - What's Next? (register)

The PCI DSS standard is evolving, with version 2.0 due some time very soon. The summary of changes has just been issued. Do you know how it affects you?  Dr Anton Chuvakin, author of the book “PCI Compliance: Understand and Implement Effective PCI Data Security Standard Compliance” will talk us through what’s expected, how you should respond, and how you should target your efforts. The focus of course will be on audit trails, tracking and forensics within a best-practice framework provided by LogLogic.

and

#2 BrightTalk’s  What PCI DSS  Taught Us  About Security (register)

This presentation will derive some useful lessons from our industry experience with PCI DSS. Organization can use these lessons to improve their security programs and reduce risk as well.

The first one is more useful and the second one is more … fun!

Enjoy!

Possibly related posts:

• All recent presentations
• LogChat Podcast 1: Anton Chuvakin and Andrew Hay Talk Logs
• Compliance Poll Analysis: PCI DSS Wins!

Compliance Poll Analysis

A while ago, I did this quick poll on regulatory compliance – and here is the result analysis.

The “winners” are:

1. “No brainer” winner: PCI DSS with 59% – it is indeed ‘forevah’
2. ISO2700x is a surprising silver medalist with 36% (more than half of PCI?)
3. ITIL holds an even-more-surprising 3rd spot with 19% – at nearly 1/2 of ISO again
4. A bunch of supposedly “cool” regs share #4 spot with 12%-15%: FISMA, HIPAA, SOX
5. …and the same percentage (15%) is held by “I don’t care about that compliance sh*t”

Notable write-ins were:

• NIST (in general, I guess beyond just FISMA)
• Red Flag (financial)
• CFATS (?)
• PHIPA, MFIPPA  (?)
• EU Data Privacy laws

What does it tell us? What can we hypothesize based on our totally unscientific compliance poll?

• All this talk about PCI DSS impacting security at large is very real – now and likely in the near future. I might argue with Josh about whether the impact is positive or negative – but it is HUGE. It definitely goes way beyond retail and ecommerce.
• ISO27001 came back to life somehow. That’s probably a good thing….
• Not sure what the lesson from ITIL being #3 is – that folks from UK read my blog? :-)
• Finally, I think the people who don’t care about compliance split into two opposite camps: people who don’t EVEN CARE ABOUT COMPLIANCE (much less security) and people who care about security and operational excellence which gives them compliance [not for free, mind you!] So, 19% covers both of these camps.

Any other thoughts?
Possible related posts:

• All posts on polls and their analysis

The End of An Era: ArcSight Goes to HP

The era has ended: the last independent software SIEM [worth buying] is bought. The biggest SIEM game “winner” (ArcSight) is acquired by HP for about $1.5b. As people are already calling me en masse to comment, here is the post with a random sampling of conclusions, predictions and “lessons learned”:

• Do something better than everybody else and you can win big – even if you start late like ARST did (this comes direct from the Cap’n Obvious, of course :-)) For example, focus on a good UI usable by your target audience as early as possible!
• Appliance SIEM battle was - until now-  a sideshow to the SIEM “classic” battle (IMHO). Yes, despite the volume of appliance sales, distributed software SIEM was still seen by many as “the real thing” and appliance SIEM was seen as “maybe for SMBs?” And now appliance SIEM guys get to fight the main war!
• Will HP screw it up? Hmmmm..... with their record in security.... oh, wait, they have a record in security? :-) No further comment.
• It is official: SIEM market again has no leader (at least until HP figures our what to do with ARST). Will anybody else stand up and take the reigns while HP is “sorting things out”?
• What is the fate of the appliance SIEM (Express) and log management appliances (Logger)? Well, the answer lies deep inside HP, but my guess is that they will not fare better than they fare now. HP “the home of OpenView” will probably like big messy software more than the boxes.
• Q: Can I please say something related to the news with the word “cloud”? A: Sooooorry, nothing cloudy about it whatsoever.

Winners:

• ArcSight, of course. Big congrats to the crew!! I competed with you a few times, but that does not mean you are not awesome :-)
• Kleiner Perkins with about 20x on the investment; even CIA made some money (via In-Q-Tel), I guess.
• SIEM players close to the top of the totem pole. All will now claim “ah, we are the leader now!”

Losers:

• Whoever was on the shortlist with ArcSight to be acquired by HP. Oops!
• Current HP “SIEM” partner - this vendor now gets to add their own name to the list of failed SIEM vendors :-) Bummer!
• Whoever else wanted to buy ArcSight. Oracle?
• SIEM players close to the bottom of the totem pole. Even fewer people will buy your wares now, especially if HP discounts Express aggressively.

More would be added as I think about it and talk to people. Other fun coverage of the matter would be added below as well.

Speaking at SANS in San Francisco on November 9

Just FYI, I will be speaking at SANS San Francisco about SIEM. Come see me there!

Topic: Got SIEM? Now what? Making SIEM work for you!

Date: Tuesday, November 9

Time:  7:00pm - 8:00pm

Location: Hilton San Francisco Union Square

Abstract: Security Information and Event Management (SIEM) as well as log management tools have become more common across large organizations in recent years. SIEM and log management have also been a topic of hot debates. In fact, you organization might have purchased these tools already.

However, many who acquired SIEM tools have realized that they are not ready to use many of the advanced correlation features, despite promises that "they are easy to use." So, what should you do to achieve success with SIEM? What logs should you collect? Correlate? Review? How do you use log management as a step before SIEM? What process absolutely must be built before SIEM purchase becomes successful?

Attend this session to learn from the experience of those who did not have the benefit of learning from other's mistakes. Also, learn a few tips on how to "operationalize" that SIEM purchase you've made.

More details and how to sign up here.

Possibly related posts:

• Another Fun SIEM Whitepaper
• How Do I Get The Best SIEM?
• LogChat Podcast 1: Anton Chuvakin and Andrew Hay Talk Logs (my new podcast on logs and SIEM!)

Log Standards and Future Trends

As some of you know, I’ve done this BrightTalk Log Management web conference the other week. My presentation was about “Log Standards and Future Trends.” Here is an embed of my presentation with voice.  If you just want this slides, go check the Slideshare version.

A BrightTALK Channel

Enjoy!
Possibly related posts:

• Log Awesomeness - On August 19!
• CEE Architecture Overview FINALLY Out!
• CEE Update - Aug 2010

Monthly Blog Round-Up – August 2010

Blogs are "stateless" and people often pay attention only to what they see today. Thus a lot of useful security reading material gets lost.  These monthly round-ups is my way of reminding people about interesting blog content. If you are “too busy to read the blogs,” at least read these.
So, here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month.

1. My super-rant about log analysis “Pathetic Analytics Epiphany!” has shot to the top like a pig kicked up in the ass by an irate giant. It is about how after looking at logs for so many years, we still use primitive approaches and primitive tools.
2. Not surprisingly, my belated reading of the Verizon Breach Reports 2010 (“Verizon Breach Report 2010 OUT!”) is in my Top5. VzDBIR is pure awesomeness, as always!
3. “Updated With Community Feedback SANS Top 7 Essential Log Reports DRAFT2”, “SANS Top 5 Essential Log Reports Update!” and their predecessor  “Top5 SANS Log Reports Update DRAFT” finally beat the previous champion of a few months “Simple Log Review Checklist Released!” Now I just need to document all the chosen favorite reports and submit it for community release.
4. Career posts always get top scores automatically and “Skills for Work vs Skills for Getting Hired” is no exception. Just as its predecessor, “Myth of an Expert Generalist”, it got on my monthly Top 5 posts immediately, was featured on Reddit.com, etc, etc. The next career post is coming soon…don’t despair :-)
5. News of sinking SIEM and log management vendors alluded to in “To Those Escaping from Sinking SIEM/Log Management Vendors” somehow made it to the top. Maybe links to SIEM jobs did it?
6. “How Do I Get The Best SIEM?”, a companion to “On Choosing SIEM“, went to the top like lighting a few months ago and stayed there this month as well. If you are thinking of getting a SIEM or a log management tool, check them out and also look at related resources at the end of these posts.  “The Myth of SIEM as “An Analyst-in-the-box” or How NOT to Pick a SIEM-II?” and ““I Want to Buy Correlation” or How NOT to Pick a SIEM?” also stay at the top – it seems like smaller organizations are looking at deploying SIEM and log management and there is a lot of interest in simple guidance on this.

Also, below I am thanking my top 5 referrers this month (those who are people, not organizations). So, thanks a lot to the following people whose blogs sent the most visitors to my blog:

1. Michał Wiczyński
2. Raffael Marty
3. Dancho Danchev
4. Cédric Blancher
5. JP Bourget

 See you in September; also see my annual “Top Posts” - 2007, 2008,  2009!
Possibly related posts / past monthly popular blog round-ups:

• Monthly Blog Round-Up – July 2010
• Monthly Blog Round-Up – June 2010
• Monthly Blog Round-Up – May 2010
• Monthly Blog Round-Up – April 2010
• Monthly Blog Round-Up – March 2010
• Monthly Blog Round-Up – February 2010
• Monthly Blog Round-Up – January 2010
• Monthly Blog Round-Up – December 2009
• Monthly Blog Round-Up – November 2009
• Monthly Blog Round-Up – October 2009
• Monthly Blog Round-Up – September 2009
• Monthly Blog Round-Up – August 2009
• Monthly Blog Round-Up – July 2009
• Monthly Blog Round-Up – June 2009
• Monthly Blog Round-Up – May 2009
• Monthly Blog Round-Up – April 2009
• Monthly Blog Round-Up – March 2009
• Monthly Blog Round-Up – February 2009
• Monthly Blog Round-Up - January 2009
• Monthly Blog Round-Up - December 2008
• Monthly Blog Round-Up - November 2008
• Monthly Blog Round-Up - October 2008
• Monthly Blog Round-Up - September 2008
• Monthly Blog Round-Up - August 2008
• Monthly Blog Round-Up - July 2008
• Monthly Blog Round-Up - June 2008
• Monthly Blog Round-Up - May 2008
• Monthly Blog Round-Up - April 2008
• Monthly Blog Round-Up - March 2008
• Monthly Blog Round-Up - February 2008
• Monthly Blog Round-Up - January 2008
• Monthly Blog Round-Up - December 2007
• Monthly Blog Round-Up - November 2007
• Monthly Blog Round-Up - October 2007
• Monthly Blog Round-Up - September 2007
• Monthly Blog Round-Up - August 2007

LogChat Podcast 1: Anton Chuvakin and Andrew Hay Talk Logs

"LogChat" Podcast is born! Everybody knows that all this world needs is a podcast devoted to logs, logging and log management (as well as SIEM, incident response and other closely related subjects).

And now you have it - through the sheer combined genius of Andrew Hay and myself, Anton Chuvakin.

Administrative items first:

1. We need a new name! We are not entirely happy with "LogChat" and, sadly, "LogTalk" is taken. Please suggest a name - if we pick yours, you get a free signed  copy of my "PCI Compliance" book.
2. We will post the transcript, not just the MP3 file - in a few days. If you have ideas for a good/inexpensive transcribing service, we are all ears. I will try Amazon Mechanical Turk first, but it might not be good enough for a technical podcast.
3. Please also suggest topics to cover as well - even though we are not likely to run out of ideas for a few years. Our first topic today is new log source integration - if it sounds boring...well...listen first/judge second :-)
4. We plan for this to be a monthly podcast. So, the next one will happen sometime early October.
5. Any other feedback is HUGELY useful. Is it too long? Too loud? Not enough jokes? Too few mentions of the "cloud"? Feedback please! Who knows...maybe there are more PCI books left in my secret stash and you too will earn that glorious prize for the most useful piece of feedback  :-)

And now, in all its, glory - the podcast: the link to MP3 is here [MP3].
UPDATE: RSS feed is here.

Enjoy the log chat!

Fun Project Honeynet Log Challenge: Log Mysteries

Project Honeynet just released its latest Forensic Challenge 5 - Log Mysteries. It is based on logs from a compromised virtual server and requires quite a bit of digging through messy log data.

The Challenge:
Analyze the attached sanitized_log.zip [A.C. – get the logs here] and answer the following questions:

1. Was the system compromised and when? How do you know that for sure? (5pts)
2. If the was compromised, what was the method used? (5pts)
3. Can you locate how many attackers failed? If some succeeded, how many were they? How many stopped attacking after the first success? (5pts)
4. What happened after the brute force attack? (5pts)
5. Locate the authentication logs, was a bruteforce attack performed? if yes how many? (5pts)
6. What is the timeline of significant events? How certain are you of the timing? (5pts)
7. Anything else that looks suspicious in the logs? Any misconfigurations? Other issues? (5pts)
8. Was an automatic tool used to perform the attack? if yes which one? (5pts)
9. What can you say about the attacker's goals and methods? (5pts)

Bonus. What would you have done to avoid this attack? (5pts)

Go get the challenge here and get to solving it – you have about a month. And, yes, there will be prizes too!

Finally, if you really want to make me happy (hehe...who’d want that? :-)), please invent a new approach while solving the challenge.

Possibly related posts:

• Everything tagged Project Honeynet