Monday, June 02, 2008

Cross-Device-Type Log Management vs Device-Specific Log Management

Now, I have to first admit that, in general, dealing with logs on a device-specific basis is a cruel joke. What I mean here is when you gather Windows logs in one place, Linux logs in another place, database logs in yet another place; all in different formats, all in different systems not connected to each others, all managed by different people who don't talk to each other (and sometimes hate each other). Yuck! Basically, this situation is "logs at their worst": all different, all disjointed and, as a result, all next to useless.

However, there are rare situations where you can choose device-specific log management approach (and still not look like a money- and time-wasting and idiot :-)). For example, you might be motivated by the fact that tools that can handle one specific type of log data (e.g. Windows-only, web server-only or Cisco PIX-only) are usually many times less expensive than cross-device log management tools. The table below clarifies it:

Use Case vs Approach No log consolidation - logs remain on systems they are produced Device-specific log consolidation and analysis Cross-device log consolidation and analysis from all log sources
Alerting based on log strings (keywords) that indicate security or operational problems Impossible or tremendously hard to manage across many systems Acceptable - alerts on each log type are handled by different teams Superior - all logs are available for analysis when the alert is triggered
Reviewing logs for troubleshooting server problems Acceptable - server logs are sufficient for Better - one can also look at logs from other similar servers Better - but same as device-specific log analysis since only one type of logs needs to be reviewed
Log review for compliance with PCI DSS Not acceptable - log management is mandated by Req 10 Impossible or very inefficient - as many types of log data needs to be collected and reviewed Optimal - all PCI relevant logs can be collected and reviewed in one system
Looking for records of a specific user activity Impossible or tremendously hard since hundreds of systems might need to be searched Inefficient - several different systems needs to be accessed to review all records of user's activities (and then data needs to be manually correlated) Optimal - one query gives all traces of the user activity
Log review for incident response or forensics investigation Impossible or tremendously hard since hundreds of systems might need to be searched for evidence Inefficient - several different systems needs to be searches for evidence and then data manually correlated Optimal - all log evidence can be found, reviewed and analyzed on one system, neither hundreds, nor several

Also, while looking at logging tools, one needs to make a distinction between tools that can collect all sorts of logs but only allow you to analyze one log type at a time (e.g. sawmill) vs tools that can collect all sorts of logs AND allow you to analyze all of them together (e.g. LogLogic). The former tools still fall under "device-specific log management" despite their ability to gather hundreds of different log types.

The bottom line: in most cases, cross-device, uniform log management provides huge value, especially if your motivation for log management is compliance or incident response.

Technorati tags: , ,

Dr Anton Chuvakin