Just FYI, SANS got this new interesting whitepaper on benchmarking SIEM [PDF] as well as a related webcast, that tries to inject some objectivity into an esoteric subject of SIEM tool testing.
My fave quote? "This is the problem with benchmarking Security Information Event Management (SIEM) systems, which collect security events from one to thousands of devices, each with its own different log data format. If we take every conceivable environment into consideration, it is impossible to benchmark SIEM systems." [that is exactly why people are having some trouble with them]
On the other hand, the paper is way too "EPS-obsessed" to my taste. And EPS is so 90s :-) And some vendors count EPS before their "magic rule" 'drop event if condition = whatever' kicks in, which is kinda sad. And EPS won't help you compare the tool that just stores all the log records vs the one that applies complex analytics over live and stored log data. And if you pick a tool by EPS, you are guaranteed to select a tool that "does less" with the log data ...
