Ideal Tool to Solve Real Problems ... of the Near Future?

Remember my write-up about an ideal log management tool?

Somebody asked me: "That's great that you have such a clear  vision of a future log management technology - but tell me first what future business problems will such 'ideal tool of the future' solve?"

First, I laughed and said: "Dude, look around, will ya? :-) There are plenty of log-related problems today which we are not even close to solving. We need to solve the problems of today first, before we can get to solving the future problems..."

So, what I consider to be the biggest log-related problems of today?

1. Not knowing what to log - whether  for compliance, tracking attackers or troubleshooting system problems. Remember all the comedy about "Tell me EXACTLY what to log for PCI?" If not, reread it!
2. Log volume  - there is too darn many log messages (seriously, 100,000 each second is a lot of log - but there is more at large companies!), and, which is worse, a lot of them are of unknown value to the users (might be useful, might not - but you never know in advance); thus, log clutter networks, systems and brains of security/system analysts.
3. Log diversity - logs all look different (at least while standards are being developed) and no single person have the skill set to understand  more than a few types. PIX admin groking SAP logs? No way!
4. In light of the above, just pure bad logs are also a major challenge - logs that miss a key piece of info (like the infamous "login failed" without the username...) or are useless in some other way are sadly common.
5. How about getting the logs from all the nooks and crannies where they are stuck  (think application logs here) - it is a problem if you want to achieve  (expand, rather) your operational awareness of applications.
6. Finally (not really, the list can go on and on), making sense of logs in  an automated fashion is still a #1 challenge  (IMHO) - we are getting better creating tools for humans to go thru logs (via reports and search), but log->conclusion process still requires a human, and a darn smart one.

Now, when you read the above think "end user", not "log management  vendor" challenges (I plan to post about these later). My idea of an ideal tool will seek to solve these and others.

Along the same line, this picture from 4th SANS Log Management Survey shows how people perceive the logging challenges:

as well as my logging challenges poll (analysis here):

Now, let's think of logging problems of the near future, say in 2 years.

But you'd have to wait for the next post for this :-)