Wednesday, February 20, 2008

Top 11 Reasons to Analyze Your Logs

As promised, here is another "Top 11 Reasons" which is about log analysis. Don't just read your logs; analyze them. Why? Here are the reasons:

  1. Seen an obscure log message lately? Me too - in fact, everybody have. How do you know what it means (and logs usually do mean something) without analysis? At the very least, you need to bring additional context to know what some logs mean.
  2. Logs often measure in gigabytes and soon will in terabytes; log volume grows all the time - it passed a limit of what a human can read a long time ago, it then made simple filtering 'what logs to read' impossible as well: automated log analysis is the only choice.
  3. Do you peruse your logs in real time? This is simply absurd! However, automated real-time analysis is entirely possible (and some logs do crave for your attention ASAP!)
  4. Can you read multiple logs at the same time? Yes, kind of, if you print them out on multiple pages to correlate (yes, I've seen this done :-)). Is this efficient? God, no! Correlation across logs of different types is one of the most useful approaches to log analysis.
  5. A lot of insight hides in "sparse" logs, logs where one record rarely matters, but a large aggregate does (e.g. from one "connection allowed" firewall log to a scan pattern). Thus, the only way to extract that insight from a pool of data is through  algorithms (or, as some say, visualization)
  6. Ever did a manual log baselining? This is where you read the logs and learn which ones are normal for your environment. Wonna do it again? :-) Log baseline learning is a useful and simple log analysis technique, but humans can only do it for so much.
  7. OK, let's pick the important logs to review. Which one are those? The right answer is "we don't know, until we see them." Thus, to even figure out which logs to read, you need automated analysis.
  8. Log analysis for compliance? Why, yes! Compliance is NOT only about log storage (e.g. see PCI DSS). How to highlight compliance-relevant messages? How to see which messages will lead to a violation? How do you satisfy those "daily log review" requirements? Thru automated analysis, of course!
  9. Logs  allow you to profile your users, your data and your resources/assets. Really? Yes, really: such profiling can then tell you if those users behave in an unusual manner (in fact, the oldest log analysis systems worked like that). Such techniques may help reach the holy grail of log analysis: automatically tell you what matters for you!
  10. Ever tried to hire a log analysis expert? Those are few and far between. What if your junior analysts can suddenly analyze logs just as well? One log analysis system creator told me that his log data mining system enabled exactly that. Thus, saving a lot of money to his organization.
  11. Finally, can you predict future with your logs? I hope so! Research on predictive analytics is ongoing, but you can only do it with automated analysis tools, not with just your head alone (no matter how big :-)) ...

 Past top 11 reasons:

Technorati tags: , , ,

No comments:

Dr Anton Chuvakin