Logging Poll #5 "Top Logging Challenges" Analysis

OK, this poll WAS fun! The raw results are here and below:

What do we learn from this? Sadly, this poll was less popular than I hoped, so the results are not as statistically significant. Still, we can draw some fun conclusions from the data.

First, what are the top challenges? It is with great regret :-) that I report that the #1 challenge is exactly the one I thought it would be: We collect logs but don't have time/resources to look at them. Yes, automated "analysis challenge" has only become more of a challenge as we deploy more tools that enable log collection on a massive scale (e.g. 75,000 logs/second). I dare to predict that we will finally have to tackle this one in the next year or two. In fact, this challenge rears it ugly head via another popular response, Lack of log analysis tools, which made Top 5 responses.

Second, even though I didn't have any predictions about the #2 entry, but I was surprised: No way to effectively search all logs is a  very close #2 (obviously, 1 vote is not statistically significant here). Indeed, log searching is an elusive little problem, especially when we want to do it fast and on a large pool of logs. Even though I think we need to search less and discover more, the need to search logs will be with us forever (and, no, I don't think you need a special product just to search logs, Raffy :-))

Third, I am happy to report that this poll shows that we finally broke the back of "the beast" of  not having logs. Responses that point at not having logs (e.g. Logging is not enabled, We don't know what logging we must enable,  etc) are not terribly popular (then again, maybe it is due to self-selection of my enlightened blog readers ...)

Fourth, infrastructure! Specifically, No infrastructure to manage the log volume we have is very popular as well (#4). This proves the point that I used to not take very seriously in the past (by mistake): when megabytes become gigabytes and those flow into terabytes, many things that used to trivial (e.g. moving logs from A to B, saving logs to disk, etc) become grand engineering challenges... Indeed, to manage high volume of logs you need a scalable log management solution (example :-))

Sixth, as I lamented, few care about log security (this counts as laments, I guess).  Secure storage of logs is only bothering a few people. One word: yet! ;-) As of today, stored log hashing + (sometimes!) log transport encryption + (rarely!) encrypted archives are the state of the art.

Next poll is coming up!

Technorati tags: logging, log management, polls