SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?

One of the ugliest, painfulest, saddest issues with SIEM is resourcing. Yes, that SIEM appliance might set us back $75,000 in hard earned security budget dollars, but how much more will we have to spend in the next 3 years deploying, integrating, using, tuning, cursing, expanding the thing? How much manpower will the new operational procedures (example) cost us? And if we actually build a SOC or “a virtual SOC”, how much will we have to spend on an ongoing basis to get the value and benefits? In fact, how much will the coffee cost if we have to work 20 hours in a row recovering that crashed SIEM database partition?

These and other questions are super-important for every SIEM and log management project. And the time has come for me to reveal my secret knowledge of SIEM resourcing. OK, that’s a joke – it is not a secret, just a bunch of things that are often unpleasant for many SIEM buyers, users and sellers to hear.

So:

NEWSFLASH! SIEM costs money. “Free” SIEM (example) costs money too, BTW. Let’s try to delve into what those costs are. I will be not-quite-scientific in regards to real “hard money” costs (e.g. software license purchasing) and “soft costs” costs (e.g. staff time costs), but I will try to clearly mark each kind of SIEM cost below.

First, assumptions and limitations:

• This is NOT “SOC staffing” , but simply “running a SIEM” staffing. SOC implies more processes and more tasks and a broader mission.
• Assumes in-sourced, traditional “buy and run” SIEM; outsourced, co-managed/co-sourced cost model would look different.

Here is the rough cost model for some of the most common SIEM cost categories:

1. Initial “hard” costs 
   1. SIEM license costs: base price +per user, per node, per EPS, per CPU (and per CPU core), etc costs – however your favorite vendor charges
   2. For software SIEM, also hardware, OS, database costs for as many servers as you need
   3. If any, mandatory 3rd party software license costs (occasionally, agents, reporting tools, etc)
   4. If chosen, vendor or consultant deployment services costs. If not chosen, staff time for deployment will pop out in soft costs below!
   5. Vendor training or 3rd party training on logs, log management,  SIEM, etc
   6. Additional external storage (in most cases)
2. SIEM ongoing, operating “hard” costs
   1. Various SIEM vendor services: support (typically mandatory), ongoing professional services costs
   2. Personnel to operate a SIEM: from part of FTE (very small scale, few use cases for a SIEM) to 1 FTE (small appliance deployments) to many FTEs of various roles (+much more for SOC staffing if live monitoring is implemented). 0 FTE for SIEM = SIEM project FAIL with 100.00% probability.
3. Periodic or occasional “hard” costs
   1. Various SIEM vendor services: professional services, custom development work for device integration (some of these may go into soft costs if done internally – for advanced organization or those experienced with SIEM already)
   2. Periodic staff training on SIEM operation and tuning
   3. Specialty staffing: DBA, sysadmins, node sysadmins, in-house developers for custom connectors, Crystal Reports administrator (yuck!), etc – some of these might go into “soft” costs if “poaching” existing personnel time
   4. Deployment expansion costs: same as initial costs, but for additional systems, software, hardware, etc as you grow; these sneak up really fast if SIEM is licensed using many dimensions such as user+CPU+node+server+something else.
   5. External storage expansion costs – yes, you will buy more storage if your volume grows, and log retention time stays the same (e.g. 1 year for PCI DSS)

On the other hand, here are some of the “soft” costs, such as time expenditures by existing resources:

1. Initial “soft” costs 
   1. Deployment time for the SIEM project – allocate more time if deploying purely using internal personnel, not vendor or consultant
   2. Log source configuration and integration – this will likely take way more than simply installing the tool. This is what makes SIEM deployment projects go for months in complex, distributed organizations with many silos.
   3. Initial tuning, content creation  and adapting the tool to your environment  (however lightweight it may be)
   4. Training and other staff time costs to jumpstart the operation (Congratulations! You bought ta SIEM. Now you need to operate it…)
2. SIEM ongoing, operating “soft” costs
   1. Report review and other ongoing monitoring tasks – from 24/7 to daily to weekly
   2. Alert response and escalation; SIEM implies correlation and automated alerting
   3. Other “using SIEM” tasks such as reviewing the dashboards
   4. Uptime maintenance tasks i.e. caring for your SIEM as well as storage – backups, updates, minor troubleshooting, etc
3. Periodic or occasional “soft” costs
   1. SIEM rule tuning, reports creation, dashboard customization, new log source integration, other ongoing SIEM tasks
   2. Periodic training and related staff time costs
   3. Expansion: same as initial soft costs

As was suggested by a discussion on SIEMusers.org (shhh…the site is not ready for launch yet), it is useful to separate soft costs  that are mandatory FOR SIEM operation from those which commonly arise FROM SIEM operation. The most obvious example is incident response due to increased awareness of network and system activities, delivered by your SIEM.

”Soft” costs that commonly result from SIEM:

1. Added cost of incident response: more incidents are likely to be detected
2. Resulting incident remediation costs and even cost of new technologies deployed for preventing the discovered issues
3. Other department personnel time for dealing with issues revealed by SIEM – the soft costs do leak out of the security department to IT and even beyond (legal, HR, etc).

Anything big I missed that you experienced? BTW, in my experience, I have seen the total cost of a SIEM project (hard + soft) range from 10% of SIEM license costs (for shelfware SIEM “deployments”) to a mind-boggling 20x of license cost.

P.S. Finally, if you want to really annoy Anton, mention “SIEM ROI.” If you do that, I will send you to Gal Shpantser for a mandatory “why he avoids SIEM!” class

Possibly related posts:

• Log Management Tool Selection Checklist Out!
• "So, What Should I Want?" or How NOT to Pick a SIEM-III?
• On Choosing SIEM
• I Want to Buy Correlation” or How NOT to Pick a SIEM?
• The Myth of SIEM as "An Analyst-in-the-box" or How NOT to Pick a SIEM-II?
• Logging, Log Management and Log Review Maturity
• Log Management + SIEM = ?
• On SIEM Complexity
• SIEM Bloggables: SIEM Use Cases
• Whitepaper with detailed SIEM use cases (using a particular SIEM as an example)
• Log Management / SIEM Users: "Minimalist" vs "Analyst"
• All posts labeled SIEM

Guest Post (First Ever!): Branden Williams “‘Free’ or Commercial—YOU DECIDE!”

This is the first ever guest post on my blog. In it, my esteemed co-author of “PCI Compliance”, Branden Williams, analyzes the use of open source software for various projects.

Here is more information about Branden and his awesome blog:

• Site: BrandenWilliams.com
• Blog: BrandenWilliams.com/blog (RSS)
• Twitter: @BrandenWilliams

"Free" or Commercial—YOU DECIDE!

Open source software that is freely available for download and use is one of the greatest things about our technical community. The fact that at any given time I have a massive library of software available at my fingertips to accomplish any number of software tasks is nothing short of amazing! Then you tell me that if there is something I want to add to the software, I just jump in and do it? WOW!

Don’t let the rest of this post fool you, I am 100% pro open-source. In fact, I released more than one open source project over the years (though nothing recently of note). Open source has a place in research and the commercial world alike.

But can you just assume that open source software is FREE software?

One of the biggest misconceptions that I see in our industry is open source software is free.  Freely downloadable, 0pen source software is by no means free—remember you need smart ladies and gentlemen like yourselves to install, configure, and support it. That aside, there is absolutely no reason why open source software should not be used to meet security or compliance requirements.

Before settling on a particular solution (commercial or open source), security professionals should do a full cost analysis including some risk-based elements.  I know security people avoid doing this because we trust our guts more than we trust business tools, and it can be very time consuming. When you have to put out fires on an hourly basis, fiddling with a spreadsheet just doesn’t seem like a good use of your time.

Know this: going through the exercise will pay off in spades by showing the team when and where open source is strategic.

Before considering an open source software package, check with your legal team to see if your company has a position on any of the plethora of open source licenses under which software is typically licensed. For example, I work with a customer that strictly forbids GPLv2 software from being used (due to the requirements to contribute code improvements back to the larger community), but permits software licensed under the BSD license. Get a legal opinion from your legal counsel before your business comes to depend on a piece of software.

Once you have the green light on a set of licenses and find a software package that meets your requirements, it’s time to do your cost analysis. Open source software that is freely downloadable does have a cost greater than zero, yet that cost is often left out of the comparison (or incomplete) between commercial and open source software packages. Here are some things to consider:

• Do you have to acquire equipment for this software to run? Be sure to include network infrastructure to support it.
• How much of your time is required to keep it up to date? Estimate it, then use your salary plus bonus, and add anywhere from 15-25% for a benefit load. This will get you in the ballpark.
• Do you need to hire a staff to keep it up to date? Use the same calc above.
• Will someone else in your company have to support it? Same calc as above.
• Will you need a second tier support contract from the open source group to handle advanced support issues?

The base formula should look something like this:

Total Cost = (Total Man-Days * Estimated Daily Salary Costs) + Initial Hardware Cost + Hardware Upgrade Cost + Annual Support Contract.

Whereby:

• Total Man Days = the TOTAL number of man-days you will spend per year. If maintaining this software will take 10% of your time, then that would be 192 hours (based on a 1920 hours/year) or 24 days. If you have multiple staff classes, you will need to do the math in the parenthesis multiple times with the correct corresponding day rates and man-day effort.
• Estimated Daily Salary Costs = Your fully loaded daily rate. If someone made 70K/yr plus a 15K bonus, that’s 85K/yr target compensation, plus a 20% benefit load = 102K/year, divide that by 240 days per year and you get around 425/day. This and the previous would get you a support cost of 10K.
• Initial Hardware Cost = The capital you must spend to get hardware to support your project.
• Hardware Upgrade Cost = Your current hardware is probably on a 3 or 5 year lifecycle. Estimate costs of replacement and divide by the normal lifecycle to get an annualized cost.
• Annual Support Contract = The annual cost of second tier support from the group that writes the software.

NOW you have something to compare to your commercial-off-the-shelf vendor’s estimate. In more cases than some of us want to admit, freely downloadable, open source software can be more expensive than commercial software. That doesn’t mean you shouldn’t use it, or that it always negatively impacts your business. On the contrary, this exercise will help you document all of the costs and risks associated with deploying the package.

Besides, on a personal note… if it goes down at 4am on a Sunday, isn’t it nicer to scream at someone’s face and then go back to bed? :-)

==

Enjoy the guest post – feel free to check out my guest post at Branden’s blog too.

Watch Ma .. A Blog Fight!

Always a suckler for a good blog fight! A subject is a bit dumb (“Is security a cost center?”), but still, this one is fun to watch:

1. McAfee, who should know better, starts it: “Is information security compliance really a cost center?” - “No. Absolutely and unequivocally not. I am drawing the line in the sand.” Read the rest here, even though it gets t sound pretty darn stupid at times (example: “ … makes it obvious that it is better and more efficient to be compliant as a business” – uhu… go tell it to all the small businesses trying to avoid PCI DSS)
2. First, Hoff kicks them in the balls (in their comments, no less): “If security compliance isn’t a cost center, are you then suggesting it’s a profit generator? So on the balance sheet it shows up as a revenue generator or profit center?”
3. Next, enlightened-not-insane Mike Rothman dropkicks them in “Compliance is SO a cost center” – “OMG. I figured a big company like McAfee would have a drug testing policy, but evidently not. I want some of what this guy is on” and even “A "Compliance Driven Company" is the next Heartland or TJX”, “CEOs don't care about security or compliance” and – fun! – “And even better, they don't want to spend money on avoiding either of those cases because it's not going to happen to them. Seriously. They see the headlines, they ask some questions about whether they are "secure," the CSO lies to them, and they go back to their mahogany conference room and check on the sales numbers.” He then ends with “Like I said, Little Red needs to check what's in this guy's water bottle. It ain't water.“
4. Finally, Pete runs, jumps in the air and lands on the McAfee guy ("Security Insights Draws Security Incite"): “It is entirely misleading to suggest that "information security compliance" is NOT a cost center. That smacks of a misunderstanding of exactly what a cost center is.” He then again jumps and lands with “I have a HUGE problem with this statement: "...a good business leader needs no justification to do to the right thing." It is so laced with b.s. that the cows are lining up in the barn waiting their turn.”

Enjoy! This whole thing makes me so want to kick them too, but I think there is a law against kicking the dead horse or something :-)

Also, this sooo reminds me of the ROI wars last year.

I will add to it, if this grows.

Second ROI War

Another day, another security ROI blogwar.

• "Schneier on Security ROI": "It's a good idea in theory, but it's mostly bunk in practice." and "The term just doesn't make sense in this context."
• Richards adds to it: "Schneier Agrees: Security ROI is "Mostly Bunk""
• "Security ROI - The debate continues": "ROI ... reared its ugly head."
• "FOI, Failure of Investment": "Not that you asked, but IMHO: ROI and TCO are SWAG at best. And, they are rarely at their best." The secret weapon of ROI war - FOI - is thus unleashed!
• "Yet more evidence: your CISO needs an MBA" - a very good view of ROI as "comparison tool", nothing else.
• "Can you get ROI from reduced costs?" - a key missing piece for "ROI puzzle", so-called 'return in the form of savings' is finally (!?) clarified by Pete Lindstrom.
  
• This is also very relevant :-)

Overall, I love it when educated peoples' debate just falls waaaay down to the level of "I won't care what YOU call it as long as you don't care what I call it...." Yuck! :-)

All security ROI coverage is tagged here: http://delicious.com/anton18/ROI. The previous, "First ROI War", is summarized here.

On Geekonomics

I am sitting in hotel here in San Antonio, TX (I presented at TRISC 2008 today - it sure was fun!) reading "Geekonomics" (can't work - I have a bit of a flu), provided by my friends from Addison-Wesley.

And you know what? The darn thing is turning me into a software liability advocate (like Bruce Schneier) - I really need to resist that ... :-)

Seriously, I just read another 10 pages and I am already thinking "Some say that if we have software liability, we will lose open source... this is kinda bad ... but such is life" :-(

Somebody please save me this train of thought :-)

Again On Breaches and Stock Price

Richard "IDS is dead" Stiennon throws a bomb: "First, esoteric matters like IT security really do not matter to the overall performance of a retailer. Customers, employees, stakeholders, apparently don’t care. Second, no matter what the security industry says, you should not justify security spending based on potential impact of a data breach on your stock price. That theory is completely disproved by TJX."

Enraged? Think he is pushing it too far? Being illogical? Me too :-) I don't think TJX example just goes and "disproves" it; we don't really know how it works with breaches and stock prices (some say 4-8% down, some say none, some say 'major impact', whatever...)

He then clarifies: "But let me point out that TJX has attributed $200 million in direct costs to this breach. It is easy to surmise this is bigger than just about anyone’s security budget. In TJX’s case some well known security practices and a little security spending would have avoided this whole incident."

Overall, a fun read. Still, I think breach impact assessment and breach's impact on anything (much less the stock price...) is not really well-defined or understood yet ...

OMG, Security ROI Comes Back - And It is Mad As Hell :-)

OK, not really mad :-) In fact, pretty intelligent :-) But a new salvo has been fired in a "great security ROI war." Counter-salvos have been fired as well :-)

The salvo is the paper called “The Fallacy of Information Security ROI” by Jon Pols ("ISSA Journal", February 2008) where Jon argues against the ROI for security (since there is no money earned by security, just saving which are NOT the same thing); Jon proposes "security as insurance" model which, in all honesty, I am not too comfortable with (since security doesn't "pay you back" after the breach).

ROI proponents "hit hard" in return: 'One is Jos Pols who, in his recent article “The Fallacy of Information Security ROI” in the February 2008 issue of the ISSA Journal (membership required to access link resource), claims that one cannot have a return where there is no income. .' They next bring back the "return in the form of savings" (which many disagree with ...): 'this is an overly restrictive view of the meaning of the word “income.” The avoidance of potential losses redounds to the bottom line, as does revenue, so that a cost saving is a return on an investment.' Read the whole pro-ROI counter-point here.

Previous "ROI War" is cataloged here. A new one is upon us? Unholster your handguns, charge the lasers, enrage your attack hamsters - hurraaaaaaaah!!!!! :-)

On Breach Economics

Here is a fun excerpt from an even more fun discussion on securitymetrics list related to "pricing" security breaches, especially TJX. One line summary: we know nothing :-)