Let me start my [much delayed] coverage of RSA 2010 conference with the awesomeness of Metricon 4.5 (technically, a Mini-Metricon 4.5 :-)) where I spent my first RSA day (sacrificing the Cloud Security Alliance meeting that was reported to be packed).
Here is an agenda for the meeting with my comments:
08:45 - 10:05: Morning Session I - Chair: Jeremy Epstein
- Qualitative Tuning as Preparation for Quantitative Methods, Pete Lindstrom
This was one of the most fun presentations, focusing on expert opinion vs. fact/metric in security. Pete showed an interesting approach for assessing the opinions in order to come up with something that looks more like fact.
- Metrics for insights on the state of application security, Ashish Larivee
This was an interesting presentation of Veracode research of binary analysis (paper, some highlights). A few thing actually blew me away first, but, upon further consideration, started to look perfectly logical. For example, software industry is worse at developing secure software than financial service industry. It can be explained that FS folks develop only mission-critical software though. Still, this seems to prove that in some areas “if you want it done well, do it yourself and do NOT trust the professionals to do it” :-) In fact, commercial software overall fared worse [vulnerability-wise] than internally developed AND outsourced software. It also had longest remediation cycle, while open source had the shortest (for methodology details see their full report)
10:20 - 11:40: Morning Session II - Chair: Joe Magee- Translating the Narrative into Metrics: The Verizon Incident Sharing Framework,Alex Hutton and Wade Baker
Verizon VerIS was released via this presentation (release, exec summary, document [PDF]). VerIS “translates the incident narrative (the attacker did this, then that, then the other thing) into a data set” and thus allows the creation of such awesomeness as DBIR.
- Ontologies for Modeling Enterprise Level Security Metrics, Anoop Singhal
This presentation was a bit of a cruel joke. It carried unfortunate signs of being done by somebody who never ventured in the real world of security (for example, single number “asset value”, “risk = damageValue”, “security measures that reduce the frequency of attacks”, etc, etc, etc). And, what was even more embarrassing, it came after the stellar presentation by the Verizon team; I think I have seen the grimaces on their faces :-) And every time the NIST speaker mentioned “this was done on tax payer dime” or uttered the word “ontology”, I wanted to just reach for a ShmooBall. To make his material even more insulting, he was also a bad presenter. Yuck!
13:10 - 14:40: Afternoon Session I - Chair: Caroline Wong- Improving CVSS-based vulnerability prioritization with business context information, Christian Fruhwirth
This was a curious little preso that basically can be summarized in one phrase “using CVSS as it was intended by the original team – with Env scores – is valuable.” Even though there was one “cringe moment” when the speaker expected a normal distribution of vulnerability CVSS scores (pray tell me, why medium severity are more likely than low severity?)
- Security Metrics Field Research, Ramon Krikken
This presentation by a Burton …eh... Gartner… analyst Ramon Krikken was hugely insightful. They did some metrics research among their clients and came up with some surprising conclusion that shows metrics programs largely in the Stone Age (in fact, what was before the Stone Age? Ah, yes, Sharpened Stick Age! The maybe the metrics are in that age…). Here are some of the themes, but get the presentation materials when they are posted – very worthwhile. As expected, “compliance metrics are easy; security metrics are hard”, “assessments and audits matter”, “need to map to ” and “ONLY prevention of ‘business being stopped’ matters at many companies.” The research showed no focus on improvements, no peer benchmarking, etc. Regarding tools, MS Excel was by far the #1, couple of times RSA/Archer and SIEM.
15:10 - 16:30: Afternoon Session II - Chair: Ray Kaplan
- Metrics for Cloud Security, Lynn Terwoerds, Caroline Wong, Betsy Nichols
This panel announced that CSA is starting a cloud security metrics effort, which was in a VERY early stage. No material has been created yet.
- Identifying critical information security areas with a Threat Agent Risk Assessment, Matthew Rosenquist
I read the TARA paper back when it came out, but this presentation (and the discussion) was still VERY interesting. The main idea is that vulnerability or asset focused approach makes no sense since there are way too many vulnerabilities (presenter example was “data center is vulnerable to a meteor strike”) and thus the way to go is to focus on threat agents that are motivated to cause damage and that can realistically to do so. The logic thus becomes: threat agent –> vulnerability –> control –> what remains is the risk that needs to be dealt with somehow. But read the paper instead of this, Intel folks explain it much better :-)
So, as I said, Metricon was the most thought-provoking part of RSA for me! And I am not even mentioning the level of hallway discussions there…
