On Cloud Logging Standards, Unique IDs and Other Exciting Logging Matters

Two of my esteemed colleagues, Misha Govshtein of AlertLogic and Raffael Marty of Loggly had a bit of an argument over something fairly central to logging and log management, especially as it applies to the coming cloud wave. Let’s review what happened.

In 2010, AlertLogic  folks have submitted an IETF draft of what they called “Syslog Extension for Cloud Using Syslog Structured Data”. Draft is available here and AlertLogic team explanation of its mission and purpose can be found here and  here (unfortunately in MP3 form). The draft reads as if they are proposing a new cloud log standard since the very first sentence of the document is: “This document provides an open and extensible log format to be used  by any cloud entity or cloud application to log and trace activities  that occur in the cloud.”

Said draft has found its way to the CEE Editorial Board (via IETF list message) and has caused some interest and, dare I say, unrest. And some disagreements. Raffael Marty of Loggly has published his position on the draft here. Further exchange of opinions can be seen in comments here, as well as heard in the hallways of RSA 2011 conference.

What do I think of this? I think both of these renowned log literati are both right and wrong (at this point, somebody might say “Anton…you are such a consultant”… and I am )

Unquestionably, I believe that the idea of cloud logging having its very own special standard, completely disconnected from all other logs is misguided. Being disconnected from both the rest of the logging domain and current log standardization efforts (like CEE, XDAS, etc) only makes this idea more misguided. In essence, if you grab an example of a current bad application log, add “cloudiness” to it (more on this later) and then publish it as “cloud log standard”, you generate mostly hilarity and not value for the IT community. Logically, it goes like this:

1. Bad log + cloud ID = really bad cloud log.
2. Really bad cloud log + public IETF draft = really bad, standard cloud log, exposed in public
3. Really bad standard log in the cloud EXPOSED in public = stupidity
4. Stupidity –> funny blog posts from Anton, like, for example, this one.

This just reminds me of Chris Hoff saying “Cloud security suffers from the exact same siloed security telemetry problems as legacy operational models…except now it does it at scale.” In fact, here is an example from the draft:

Aug 16 13:34:18 [context aid="149683FC-8DF5-1004-E1A8-00000A000152"
   provider="example.com" rid="1:123"][transit client="172.16.1.82"]
   User authentication successful for 1:123

Would YOU like to spend your mornings analyzing logs like this? If you expose such examples in a purported standard draft, future generations of log analysts will hate you with a passion….

However!

I also happen to think that there are significant differences of logging from/at cloud computing platforms (whether SaaS, PaaS or IaaS) compared to BOTH traditional system logging AND distributed application logging. Cloud computing (as defined by NIST) has inherent multi-tenancy, elasticity, immediate provisioning and other fun properties, not found in traditional applications and platforms – whether distributed or not. All of these happen to affect accountability, auditability and transparency – all the goals logs serve – in a number of big ways. Thus, cloud computing must change how logging is done and it will change it. Specifically, adding a unique ID (“audit identifier which uniquely  identifies an external request for activity”) to logs in order to enable serves a useful purpose.

So, we must change logging for the cloud AND we must improve logging  everywhere through standard work. It will result in GOOD, USEFUL LOGS that ALSO WORK WELL IN THE CLOUD. The caveat? We need it sooner than CEE is finished and adopted on a broad scale. “CloudLog” effort contains useful ideas that need to be implemented in future logs produced by cloud framework components, but the method chosen (uncooked IETF draft choke full of bad log examples) deserves mostly ridicule…

Book Review “Cloud Security and Privacy”

Amazon just posted my review for “Cloud Security and Privacy” by Tim Mather, Subra Kumaraswamy and Shahed Latif.

It is reposted below for posterity – and my esteemed blog readers :-)

It goes without saying that I was very excited to pick up the first book on cloud security and privacy. Due to my Cloud Security Alliance (CSA) involvement, I was extremely interested in Tim’s take on the subject. The book is indeed a comprehensive treatise on everything cloud, and everything cloud security. The author team covers the topics based on IaaS/PaaS/SaaS (SPI) for infrastructure, platform, and software as a service model. They address stored data confidentiality, cloud provider operations, identity and access management in the cloud, availability management as well as privacy. My favorite chapter was of course the one on audit and compliance - chapter 8. Another fun chapter was chapter 12 on conclusions and the future of the cloud (which is, BTW, all but assured…).

One of the most important things I picked from the book was a very structured view on separation of security responsibilities between the cloud provider and the customer for all of the SPI scenarios. This alone probably justifies getting your own copy.

As far as technical contents, the book stays fairly high-level even though it touches on the details of SAML and other authentication protocols.

The only downside of the book is its extremely dry writing style. There are only a few examples and case studies. Following “just the facts” model sometimes might lead the reader towards losing interest, no matter how important the subject is – and this subject is pretty darn important. To put this in the context, I do read security books for fun, not only for work.

Enjoy the book!

Possibly related posts:

• Other book reviews on my blog
• My old book review site.

Open Source CLOUD SIEM, Anybody?

"It's too bad I am not building an open-source SIEM … but if I would" (extra credit: guess the inspiration for this line), I would actually not build a software security information and event management (SIEM) product.

If I were doing it (and I am not!),  I’d built an open-source-based “cloud SIEM” with a default option to have it hosted by the vendor (that is you) as well as with a possibility to have it hosted by the customer (that is, old school on-premise model). The latter option would give you a classic open source “product.”

Think about it! If you are a new company, you cannot afford to be in the appliance business. That leaves two choices: software and SaaS. If you want to sell software, you are probably insane, go lock yourself up :-) If you want to give away software and sell services, you are OK. But won’t SaaS be better? And you can combine it with on-prem model, if some folks insist.  My recent employment made me quite SaaS-obsessed, since I had a chance to observe an excellent SaaS operation from the inside. It was amazing how easy it is to provision trials as well deploy the service in production, track usage and improve customer experience.

But let’s take a step back first! A lot of folks try to understand the relationship between “open source” and “cloud everything” (discussions about their relationship here, here, here and obviously here). The main idea here is the following: if people have “trust issues” with cloud providers, what is a better “mistrust breaker” than showing the source code for your solution? “Wonna audit?” - “Knock yourself out!” Moreover, if people have “hidden costs issues” with open source software, what is a better way to mitigate  it but to offer to run it for them? No hidden costs, not even electricity…

So, if you are possessed by SIEM aspirations (and I am not!), head to the clouds. For starters, your tool will be easier to deploy and update. But what is much more important, you’d take a fare shot at solving scalability problems without being a database tuning uber-guru. Unlike early software SIEM vendors (FAIL!), you won’t have to sheepishly point in the direction of mammoth Sun boxes, you’d just fire up another EC2 instance (or a thousand).  Moreover, on the scalability front, you’d have a fare shot against established SIEM vendors: I often hear rumors that even today, in the age of “cheap hardware”, some large organizations with loads of log data simply cannot architect a SIEM to handle all their security log data. And cloud computing leads to affordable scalability!

Solving these performance problems will let you use the CPU resources for log data parsing, correlation, stored data analysis and all sorts of other fun stuff. Storage, whether raw text or parsed data, will be even easier. Bandwidth will be a bit tricky, but I am leaving it out of this piece for a particular reason … don’t want to spill too much (I am leaving “the collection challenge” out as well).

Finally, as cloud applications [as well as web applications, in general] grow in importance, the whole idea of “SIEM in the cloud for the cloud” won’t be as naive. As we know,  the need for auditing comes last (here), but when it does come, it will come in force and both SMBs and F2000s will have this problem.  And it never hurts to be better prepared for the death of on-prem apps, if we are to believe certain visionaries.

So, throw together some EC2 magic, some database code and a sleek, well-designed UI to delight the users – and you are ready to do battle with foes 10x your size…

Any thoughts?

Possibly related posts:

• Why No Open Source SIEM, EVER?
• “On Open Source in SIEM and Log Management”
• Various SIEM posts.

Obligatory “added everywhere” posts :-)

• I am not at Qualys anymore and looking for the next big security idea to work on! Meanwhile, I might be available for fun consulting projects related to PCI, log management or other fun security things.

BlackHat 2009 Day 2 – Fun Cloud Stuff

BlackHat 2009 is over, but sharing impressions from it is certainly not.

So, for the the remainder of day 1 went to “Weaponizing the Web” (which had good ideas on CSRF, see their tool here) and “Psychotronica” (which was great content totally killed by a sleep-inducing speaker – I left mid-talk. In fact, I am yawning even as I write about it…). And then I had a chance of seeing Linus get his pwnie…

Then I started Day 2 at Jeremiah and Trey talk, which was a lot of fun. Moreover, it was so much fun as to reach 100% entertainment, which is another way of saying that it was not useful for any practical purpose (apart from entertainment purpose mentioned above, of course :-)). In brief, it covered a whole bunch of fun “non-hacking hacking“ cases (such as compromise of a system to issue licenses to do logging in Brazilian jungle, which supposedly netted somebody a cool $800m). They touched (but, sadly, didn’t analyze) a few things such as what is a better focus: “super hacker strategy” (vs advanced targeted attacker on key systems) vs basic baseline (vs opportunists strategy on all systems) [“both” is what they hinted at, of course]. BTW, their deck is posted here, check it out!

Next was my cloud talk #1, “Clobbering the Cloud.” (UPDATE: full slide deck here) A lot of fun and useful things were discussed – and some impressive cloud “0wnage” was shown too. It started from a useful reminder that the whole permission for “testing the cloud” (whether via scanning or manual pentesting) issue is not resolved. Moreover, PaaS/IaaS made it that much worse, since you might have a permission from the cloud application vendor, but not from Amazon and then end up blacklisted (“Never allowed to buy from Amazon again” :-)). In addition, even issues like “Which version of the application/OS/environment are you testing?” are frequent, since SaaS provider might update their application at any time.

They briefly touched on “cloud compliance”, focusing on transparency of the cloud. Somehow they had an impression that nobody is putting regulated data in the cloud…mmmm… right :-) The also mentioned the subpoena risks of having your data obtained by this or that government without you even knowing. Their point was that trust matters A LOT in the cloud, but at the same time the “verify” part of ‘trust but verify’ often fails.

Here is a set of fun things discussed:

• Cool method for password brute-forcing with password reset links; after all, most if not cloud apps use some password recovery (email- or secret questions-based)
• A very interesting sifto tool (SaaS nikto) written as a Salesforce.com app, which then runs off a high-bandwidth link for free (the story also features a CAPTCHA with its text left in the same web page…)
• Also, a bunch of good ways to steal cloud resources: Amazon cloud instance of Windows license stealing, paid application theft (via DevPay), etc.
• Fun “cloud DoS” with exponential, virus-like growth of VM instances and users.
• Impressive use of trojaned images combined with a tool to make them popular and have them show up at the top of the list. Instant mass cloud 0wnage!

Overall, amazing Amazon IaaS rampage! Also, they showed some fun Apple MobileMe 0wnage as well.

What are my thoughts on this?

First, I’d bet that offensive cloud use (either using stolen benign cloud resources or native “built for evil by evil” clouds :-)) will beat defensive cloud use (like Mark Curphey’s security data analysis ideas) by a long shot. Before we harness cloud resources for security (such as for analytics, etc – we do harness them for scanning already), somebody will turn it against us in a big way. But then again, botnet use for password cracking (which is more “distributed computing” than “cloud computing”) is already there so, “evil cloud” stuff is starting to be a reality…

Second, something made me think that, personally, I’d always keep an offline backup (for BOTH data and processing capability!) for anything I’d put in the cloud. Notice how it compares to the past paranoid mantra “don’t store anything truly private on an Internet-connected PC” – nowadays it is “don’t store it ON the Internet” :-( What’s next, don’t announce it on Twitter? :-)

Third, people talk a lot about software liability and how hard/controversial it is. I had this thought that maybe cloud computing will be where it will start?

Finally, how’s that for a paradox?

a) Many folks say that: “cloud security" (loosely defined here) can be and needs to be awesome.

b) Everybody agrees that: web app security is horrible and will be horrible for a long time.

c) Obviously: cloud computing today is mostly web apps.

Huh? Isn’t the whole cloud security fun (now I know why some folks are so excited about it)?

Next, I went to Kostya’s “Cloudburst” talk; I didn’t follow VMWare security closely enough, but seeing another reliable Guest->Host escape is pretty cool. Sadly, too many people chose this room to catch up on some much needed sleep after a rough night, it seems.

Finally, Bruce Schneier did a very fun talk (yes, really!), which deserves its own post tomorrow.

Possibly related posts:

• BlackHat 2009 Day 1 – Laws of Vulnerabilities Panel

More On KindleGate

SANS just repored that the KindleGate is worse than just "1984":

"COPYRIGHT, PIRACY & DIGITAL RIGHTS MANAGEMENT
--Amazon Deletes Purchased Books From Kindle Users' Devices (July 17, 2009)
Kindle owners who had purchased [emphasis by Anton, explained below] electronic copies of George Orwell's Animal Farm and 1984 were no doubt surprised to find the books deleted from their devices last week. Apparently the company that added the editions of the books to the Kindle catalog did not have the rights to do so. Amazon credited affected users' accounts for the cost of the books. Amazon says that if it faces similar circumstances in the future, it will not delete books from users' devices. Comments in customer web forms indicate that certain editions of Harry Potter books and works by Ayn Rand had similarly disappeared. The Kindle terms-of-service agreement nowhere states that Amazon has the right to delete purchased content from users' devices. The irony of Orwell's books being deleted has not been lost on the public.
http://www.techweb.com/article/showArticle?articleID=218501227&section=News
http://www.nytimes.com/2009/07/18/technology/companies/18amazon.html"

But this teaches us something truly deeply funny and sad about the world we live in! Even SANS thinks that the readers "PURCHASED" ebooks; I am sure that said readers thought so too. That was their honest perception.

In reality, they never did purchase anything - they licensed it.

As a result, I suspect that the more stuff like "KindleGate" happens, the more the following perception (whether true or not!) will grow, strengthen and develop:

When you "BUY" digital content, you don't really BUY it - it is not really a PURCHASE.

THEREFORE

When you STEAL digital content, you don't really STEAL it - it is not really a CRIME.

Please, folks who enforce the rules the way Amazon just did, watch for that monster you are helping to create. It will destroy you!

UPDATE: fun discussion about clouds, DRM and the KindleGate here (read the comments too)

UPDATE: Kindlegate truly ends; "Amazon Settles Kindle Deletion Lawsuit For $150,000" ("Amazon.com has agreed to pay $150,000 to the student who sued the company for deleting his digitalcopy of George Orwell's 1984 from his Kindle e-book reading device.")

Vulnerability Scanning and Clouds/SaaS/IaaS/PaaS

Here is a very fun post called “Vulnerability Scanning and Clouds: An Attempt to Move the Dialog On…” I loved it so much, I will just quote my favorite parts here with a few comments. It starts like this:

“Much has been said about public IaaS providers that expressly forbid customers from running network scans against their cloud hosted infrastructure.”

In other words, they host your server, but you cannot check it for vulnerabilities at all.

“As has been noted before, a blanket ban on legitimate scanning activity by customers of their own infrastructure (whether outsourced or not) undermines security assurance processes and can make regulatory compliance impossible; e.g. PCI DSS mandates network vulnerability scanning as a control”

Use PaaS – lose PCI DSS compliance status. Nice! Sadly, the above interpretation is correct as, I suspect, the IaaS/PaaS provider is not allowed to scan your boxes either. So, nobody does. And then you get 0wned.

Next the post highlights that the fact that vulnerability management challenges are magnified by using PaaS/IaaS. For example:

“Scanning may trigger automated or manual actions by the provider. A common automated response from a provider is to apply traffic shaping to slow down the scan, or simply block the client IP address via an ACL update.  This can lead to false negatives; i.e. vulnerabilities present are not discovered as the scanner IP was automagically identified as a noisy vulnerability scanner and auto-throttled/blocked.”

He then highlight somewhat obvious, but still key point:

“Even if spinning up copies of “known good/secure” virtual machine (VM), you still need to scan them.”

Further:

“So the bad guys get to scan because they don’t care and yet the customer, who wants to do the “right thing”, is not allowed to.  Is that rational?”

The solution is “easy” – if you need scanning (and everybody does!) and you PaaS doesn’t allow it, don’t use that PaaS. But is this a solution? The post the  suggests “ScanAuth API” which will allow controlled scanning:

“Something like an “ScanAuth” (Scan Authorize) API call offered by cloud providers that a customer can call with parameters for conveying source IP address(es) that will perform the scanning, and optionally a subset of their Cloud hosted IP addresses, scan start time and/or duration. This request would be signed by the customers API secret/private key as per other privileged API calls.”

I am not sure about you, but this sounds like an awesome idea! The original post calls for the start of discussion, and I am happy to continue it. Finally, as of today, I don’t think we can rely on “other security tools”  (software assurance, secure coding, etc) to supplant the need for vulnerability scanning. if you deploy an OS in the cloud, you’d need to scan it.

BTW, similarly to network vulnerability scanning, the situation is actually worse for web app scanning? If you have “doubts” about your blog provider, can you hit it with Qualys WAS?

RSA 2009 Impressions, Part II or The Only Fun RSA Keynote

OK, so people make fun of RSA keynotes as being “content-free”, buzzword-heavy and overall annoying. I did that too. However, this year I had advance knowledge that one keynote will be very fun, insightful and “B.S.-free”!

So, I came a bit earlier and the previous keynoter (not sure who that was) was working hard proving that RSA keynotes suck by droning on and on about nothing. I just couldn’t wait for Philippe’s keynote to start – and then it did and proved even more fun and insightful than I thought. Here is what caught my attention in his keynote:

• First, “The Inconvenient Truth”: critical data is spread across devices / laptops / phones today; many of such devices are lost every day. Data->gone.
• Second, vulnerabilities are being a) exploited and b) not fixed (updated Laws research shows no change in half-life of a vulnerability – still at 30 days as 4 years ago)
• The above two lines should tell everybody (rephrased by me for increased drama): “cloud is not a threat to data governance, you are!”
• Deploying applications to deal with security problems seems to open more security issues. Thus, enterprise LOST the security battle since it is impossible to secure today's networks and applications. To top it off, business need systems, IT resources faster than ever: and they are impossible to secure even at the slower pace.
• I have heard the whole “$84 billion to maintain Outlook+Exchange per year” line before, but it still has shock value. That is what people pay for insecure apps that handle valuable data (=email) today.
• Answer? SaaS! If you sell software and somebody does it in the cloud, you will be replaced. Good bye!
• Good news: today’s expansion of SaaS is also another chance to “build security in”; we failed this for platforms and applications, now we can [try to] do it for SaaS.
• Also, SaaS allows for more control over data (analogy: old mainframe model) and for more usable-yet-effective security. Obviously, there are a lot of problems to solve (e.g. browsers with holes, authentication across apps, strict and enforceable SLAs, etc)
• Example: end to end secure email was attempted since the 80s (with proven 100% failure of adoption rate), but now a big cloud provider (e.g. Gmail) can do it easily.
• Final word: “in cloud we trust, but it is our job to verify it!”
  

Full keynote video is here (yes, it IS actually worth watching!) and a lot of media coverage is here, here, here ("Cloud: Resistance is futile"), etc.

Enjoy all RSA coverage here.

Possibly related posts:

• RSA 2009 Impressions, Part I or "PCI DSS is NOT a Pill Against 'Stupid'"

Finally, Somebody Said It Like It Is...

So, I was writing a long blog post (still to be finished) about how I read some stuff people write about "cloud security" and laugh. Why? They write about cloud security and cloud-based security services, but we DO it. Now somebody said it really well here:

"Cloud computing is all the rage now.

But Qualys, a fast-growing Redwood City-based network security firm, was a pioneer in offering computing applications and services over the Internet when it was founded in 1999."

Think about it ... you are writing write about it in 2008 and there is somebody who has been doing it since 1999.