So, I was talking to this small log management vendor the other day and he confided to me that his product faces fierce competition in his target market (which is, important to note, small to medium companies with 10-100 systems): and this competition is apathy.
More specifically, his prospects either just blow him off by saying "pah, who needs this logging crap" or they profess their undying love of all things logging - and then still don't buy his product (which is priced, shall we say, "to go" :-))
Admittedly, these are the companies that form the core of today's botnets (thru sheer idiocy - and a generous helping of resource/skill shortage!) and enable RBN to deliver high-quality malicious services to criminal enterprises worldwide, which is no mean feat. Still, if you happen to have thoughts along the line of "who needs logs?" or "ah, logging? it will come later!", you really deserve a nice fat check from RBN and other malicious "hacking" syndicates since it is extremely likely that your overall attitude towards security is just as misguided (Did I just invent a metric called "logging as a litmus test of security program maturity"? :-))
But how to move from such ... what was before the Stone Age? ... Sharpened Stick Age? to modernity? Most companies go thru the following in regards to their logging:
- Deep log ignorance: "Logs? What are those?"
- Shallow log ignorance: "Later...later...later... #37 on the TODO list."
- Log collection: "We gather and store dead log data...cold."
- Log searching: "We will dig into the pile when we have to ... hopefully never!"
- Log analysis and reporting: "We know our logs - and what they mean"
(also see my post "Natural Flow of Log Management" for some specifics)
Of course, compliance (PCI DSS and others) help move people from 1. and 2. to 3., but - here comes the punchline! - people often get stuck at 3. or 4. and never progress to Logging Enlightenment of 5.
Yes, PCI DSS and other regulations mandate not just log collection, not just dead cold log storage, but also log review, (daily, in case of PCI DSS Requirement 10) but "review" happens to be the item that gets overlooked all too often.
I think the reason is that log analysis is still too hard and still not automated enough for an average organization. Yes, I did see some corporations that built their own log analysis systems that - surprise! - exceeded the best available from the vendors [at the time]. However, a typical company IT department would not have Ph.D. poring thru some text mining research papers in order to improve their home-grown log analysis AI. They expect the vendors to eat the logs, chew on them for a bit - and then spit out the answers.
Are we there yet? No, but we will be!!