On "Data Retention of Event Logs for Compliance"

This - yeah, you guessed it! - fun report by Eric Ogden from Enterprise Strategy Group is called "Security Information Lifecycle: Data Retention of Event Logs for Compliance" Among other interesting bits, it has this point that "typically active Fortune 500 corporation [is] generating 250,000 events [or log records] per second"

Is it scary? It depends what scares you (and, of course, whether you are easily scareable :-))

* Does collecting all this data scares you? Actually, its not that scary as long as your log collection is distributed and thus does not cause any major bandwidth consumption in one network segment...
* Does storing all this data scares you? Actually, its pretty benign given a great combination of log high compressibility with cheap disk drives (even when sizes hit terabytes). Yes, we are talking about storing all this data on a disk, not tape (it will be clear why in the next item!)
* Does accessing all this data scares you? Aah, we hit a good one. Some of the solutions that claim to support the above rate only support it for collection+storage (which as the easy - or easier - ones), and if you want to actually access the data - its another story. It might involve a bit of waiting...
* Does making sense of all this data scares you? Well, this one is a bummer as well- it is pretty scary. But, it opens a whole universe of log analysis, which justified a later post... One thing I would like to note is that making sense of data should be more automated than in most current solutions: the less time the user spends thinking the better (after a lot of thinking was done by the software developers and log analysis researchers...)