Cross-Device-Type Log Management vs Device-Specific Log Management

Now, I have to first admit that, in general, dealing with logs on a device-specific basis is a cruel joke. What I mean here is when you gather Windows logs in one place, Linux logs in another place, database logs in yet another place; all in different formats, all in different systems not connected to each others, all managed by different people who don't talk to each other (and sometimes hate each other). Yuck! Basically, this situation is "logs at their worst": all different, all disjointed and, as a result, all next to useless.

However, there are rare situations where you can choose device-specific log management approach (and still not look like a money- and time-wasting and idiot :-)). For example, you might be motivated by the fact that tools that can handle one specific type of log data (e.g. Windows-only, web server-only or Cisco PIX-only) are usually many times less expensive than cross-device log management tools. The table below clarifies it:

Use Case vs Approach	No log consolidation - logs remain on systems they are produced	Device-specific log consolidation and analysis	Cross-device log consolidation and analysis from all log sources
Alerting based on log strings (keywords) that indicate security or operational problems	Impossible or tremendously hard to manage across many systems	Acceptable - alerts on each log type are handled by different teams	Superior - all logs are available for analysis when the alert is triggered
Reviewing logs for troubleshooting server problems	Acceptable - server logs are sufficient for	Better - one can also look at logs from other similar servers	Better - but same as device-specific log analysis since only one type of logs needs to be reviewed
Log review for compliance with PCI DSS	Not acceptable - log management is mandated by Req 10	Impossible or very inefficient - as many types of log data needs to be collected and reviewed	Optimal - all PCI relevant logs can be collected and reviewed in one system
Looking for records of a specific user activity	Impossible or tremendously hard since hundreds of systems might need to be searched	Inefficient - several different systems needs to be accessed to review all records of user's activities (and then data needs to be manually correlated)	Optimal - one query gives all traces of the user activity
Log review for incident response or forensics investigation	Impossible or tremendously hard since hundreds of systems might need to be searched for evidence	Inefficient - several different systems needs to be searches for evidence and then data manually correlated	Optimal - all log evidence can be found, reviewed and analyzed on one system, neither hundreds, nor several

Also, while looking at logging tools, one needs to make a distinction between tools that can collect all sorts of logs but only allow you to analyze one log type at a time (e.g. sawmill) vs tools that can collect all sorts of logs AND allow you to analyze all of them together (e.g. LogLogic). The former tools still fall under "device-specific log management" despite their ability to gather hundreds of different log types.

The bottom line: in most cases, cross-device, uniform log management provides huge value, especially if your motivation for log management is compliance or incident response.

Technorati tags: logs, log management, logging