FYI, this piece has been specially created for LogManagementCentral (original post), an awesome resource for all logging things. It is reposted here for posterity.
While many people have seen 10 things that your chef, real-estate agent, wedding planner or pilot won’t tell you, the world has not yet seen Top 10 things your log management vendor won't tell you. Finally, this gap is now closed.
1. “We talk analytics, but really, most of our customers use us for collection only.” While some products within SIEM and log management offer advanced analytics features, many of their customers are not truly ready for them. They need to start dealing with the basics – logging, log collection, log review before delving into advanced areas. Buying a product based on features you won’t use is a mistake. For example, see “Log Management Before SIEM”
2. “Our tool won’t make you PCI compliant. You’d have to do A LOT of things yourself – every day – to get and maintain compliance.” Sadly, many security solutions – and SIEM / log management are no exception – are sometimes sold as “compliance in a box.” You need to be aware that to stay PCI compliance you need to do more than purchase tools. For example, see “How to Stay PCI Compliant”
3. “No, you cannot buy an entire SOC in this small box.” Just as with compliance, you cannot buy an entire Security Operations Center in a box, big or small. However, some will try to sell you their SIEM as “SOC-in-a-box.” Running an effective SOC includes multiple processes and procedures which are just as necessary as a market-leading SIEM tool
4. “We are cloud-ready, because … mmmmm… well, we are ready for it!” Many vendors will tell you that their tools are cloud-ready – without really thinking what they mean. Effectively monitoring traditional and multi-tenant cloud environments distributed across regions and countries requires more than updated marketing materials. You need to carefully test the tool in your own hybrid environment before concluding that it is “cloud ready”
5. “Our SIEM is really just a renamed log management tool. But that’s all you probably need.” The confusion around SIEM and log management functionality rages on – it also allows some tools to be sold as SIEM without having any critical SIEM functionality such as correlation and real-time dashboards. Even though it might be all many customers need, it does not make such tool a SIEM tool. For additional reading, see this whitepaper.
6. “We can do everything with logs, but it might require some SMALL customizations. Our PS team is standing by!” More than a few SIEM vendors will promise support for every possible log – including logs they have never seen. However, fully integrating a new log source for reporting, correlation and visualization will always takes work and cannot be taken for granted.
7. “If you make a mistake with capacity planning, we’d be happy to sell you more log management than you really need.” Many organizations are having trouble estimating how much log data will be coming into their SIEM or log management tools. Both under as to making and overestimating are common. It is recommended that you spend about a week measuring log volumes across the systems that will be reporting to a SIEM.
8. “We think our tool is scalable, but we don’t really have production customers of your size. Our engineers believe that it might work.” Scalability claims are cheap and would often be made by SIEM and log management vendors. However, the only real proof that the tool will scale to your requirements is testing the tool in your environment. Thus, you should insist on performance testing during the pilot if there are any doubts.
9. “Out tool offers predictive security intelligence. No, we don’t know what it means either – and we can’t really predict it.” SIEM is one of the most over-hyped and over-marketed security technologies out there. The only way to get the tool that satisfies your requirements is too carefully spelled out those requirements and then test the tool yourself.
10. “We estimate our performance using really small log messages sizes.” Yes, our tools can do a million message an instant – but these are our special messages that we create in the lab. Nowadays, application logs and proliferation of XML-based logging has pushed the message sizes up to 1 kb or more from a traditional 200 byte logs from firewalls. Thus, you need to be wary of performance estimates based on such artificially short logs.
So what is your vendor NOT tellin’ ya?
