Tuesday, December 21, 2021

Stealing More SRE Ideas for Your SOC

As we discussed in “Achieving Autonomic Security Operations: Reducing toil” (or it’s early version “Kill SOC Toil, Do SOC Eng”), your…


As we discussed in “Achieving Autonomic Security Operations: Reducing toil” (or it’s early version “Kill SOC Toil, Do SOC Eng”), your Security Operations Center (SOC) can learn a lot from what IT operations learned during the SRE revolution. In this post of the series, we plan to extract the lessons for your SOC centered on another SRE principle — evolving automation.

First, for many security operations teams automation in a SOC is about saving time by automating routine tasks. To me, this constitutes current conventional wisdom about automation in your SOC. However, in this post, we want to reveal a broader truth about automation in your security operations activities, drawing the lessons from the field of site reliability engineering (SRE). Naturally, I am not an SRE, but I feel that my analyst experience makes me qualified to translate or “port” the findings from their domain (SRE) to ours (SOC) — and to make new discoveries in this process too.

Security operations materials often point out that automation is a force multiplier, not magic. The SRE book says the same: “For SRE, automation is a force multiplier, not a panacea.”

However, the book also adds that “multiplying force does not naturally change the accuracy of where that force is applied.” This reminds us that automating a broken process often makes it more broken, but also that automating something that isn’t game-changing or systemic for a SOC would make you slightly better, if that.

By the way, this is why the most common starter SOAR playbook is about phishing, a major time-suck of many aspiring SOCs (I’ve heard one spent 40% of analyst time on phishing response and that was after the email security gateway did its work).

So people often point out that the value of automation is about saving time. However, both security operations center practitioners and SREs agree — consistency is also a big part of such value (“What exactly is the value of automation? Consistency!”). In fact, “automation provides more than just time saving, so it’s worth implementing in more cases than a simple time-expended versus time-saved calculation might suggest.” Think about it — it’s not only about saving time, scaling (“scale is an obvious motivation for automation”), but also consistency of what gets done whenever it needs to be done. By the way, how to make the security processes consistent yet allow for creativity, such as threat hunting? We will explore this in the next SOC paper in January.

Speed does come up a lot in SRE discussions of automation, after all “humans don’t usually react as fast as machines.” In the past, I largely implied (even in 2009) that sub-second speed matters little in security, especially in the day and age of 200+ day response timelines. Guess what? With ransomware, speed does matter. If you detect it via a ransom note, it won’t matter how good your SOC was …

To summarize, the main lesson from SRE is that “the factors of consistency, quickness, and reliability dominate most conversations about the trade-offs of performing automation.” These lessons work well when starting to make your SOC scale faster than the threats.

Further, I picked up a particular new insight from the SRE book, namely that automation separates the operation from an operator (“Decoupling operator from operation is very powerful.”). Why is it good? Glad you asked: “once you have encapsulated some task in automation, anyone can execute the task.” What does this solve? Some of the talent shortage problems in your SOC! This again gives us a chance to scale faster than the growth of threats and assets.

Here is another very useful reminder for your SOC from the world of SRE: “automatic systems also provide a platform.” What does it mean? That script you wrote is not a platform, even if it automates something. The way I think about it, the platform is a programmable entity, a base to develop other cool things. This means you have a chance to go for a more systematic automation of your current and future SOC activities.

Also, the SRE world delivers a very fun, slightly paradoxical, consequence: “A platform also centralizes mistakes. In other words, a bug fixed in the code will be fixed there once and forever” Think about it for a second! This is not about SOC being a great place to come and make mistakes … this is about the fact that you go to ONE place to look for mistakes, rather than chase them over 50 tools and 200 regional offices. Centralizing mistakes is awesome — and a new thought for me (and, I am assuming for many SOC practitioners as well).

Finally, “automation as a platform” leads us to metrics: “a platform can export metrics about its performance, or otherwise allow you to discover details about your process you didn’t know previously.” As you can guess, this delivers sizable — and positive! — implications for your SOC, given how hard security is to measure in general.

To my surprise, our SRE colleagues also pointed out a few negatives of automations. Now everybody likes to point out that problems with automation stem from automatic systems causing damage. This can happen both in the operations realm and of course in our beloved domain of cyber. Google SRE book describes beautifully horrible examples where many production systems at an, ahem, major tech company were deleted by automation, reimaged straight to demagnetized dust with enviable scale and effectiveness …

Now, what are the lessons? Here is a new idea as well: “Automation needs to be careful about relying on implicit “safety” signals.” What does that mean in a SOC? Well, a classic example would be blocking access based on badness, without checking for business criticality. We imply that it is safe to block access, but do we have an explicit “this machine is OK to auto-block” list? This is safe to shut down? This is safe to block access to? Using explicit safety signals for automation is a useful insight for me.

I have learned other challenges that are relevant to the world of security operations, which frankly I haven’t thought about. Many SOAR users complain that when the security tools change, EDR vendors change, APIs, logs change and other technologies evolve, their SOAR systems don’t always follow quickly enough. This is a well-known problem in the world of SRE: automation “being maintained separately from the core system therefore suffers from “bit rot,” i.e., not changing when the underlying systems change.”

Another lesson that we are starting to see in many security operations centers is that those automations that are infrequent, such as playbooks run upon seeing rare attack indicators are difficult to test. “Automation that is crucial but only executed at infrequent intervals and therefore difficult to test is often particularly fragile because of the extended feedback cycle.” It is easy to refine an efficient playbook that runs 10 times a day, but it’s much harder to run and refine a playbook that is supposed to help for a particular type of an advanced attack and may run twice a year, if that. How do we fix that? With more automation — test automation and simulations in this case.

Another great idea for your SOC is hiding deep inside the book. This has been characteristic of many leading security operations centers, and it has been discussed in many detection engineering articles, but it is definitely NOT common at many mainstream SOCs: “The most functional tools are usually written by those who use them.” This is why in our ASO workshops we explain that “SOC analysts” and “detection engineers” must go … and become one, or at least work together closely. “DevOps” your SOC!

We promised to discuss not just how to automate, but the evolution of automation. Here the news from the world of SRE is the most exciting. They chart a path of arriving to the autonomic system (that does not need extraneous automation) by starting from a manual approach and then evolving to automation. Here is what the book says:

  1. “Operator-triggered manual action (no automation)
  2. Operator-written, system-specific automation
  3. Externally maintained generic automation
  4. Internally maintained, system-specific automation
  5. Autonomous systems that need no human intervention”

While some of the above are not obviously related to SOC, let me try this:

SRE to SOC translations

That last step is interesting for sure. There is a lot of fun, thought-provoking stuff in SRE thinking related to “autonomous systems.” For example, they say that “software-based automation is superior to manual operation in most circumstances, better than either option is a higher-level system design requiring neither of them — an autonomous system.“ They further explain that non-autonomous “where automation replaces manual actions, and the manual actions are presumed to be always performable and available just as they were before.”

Finally, this is a security blog and so I was meaning to end it on a depressing note (compliance!). But then it turned out that SREs do “black humor” almost as well: “If we are engineering processes and solutions that are not automatable, we continue having to staff humans to maintain the system. If we have to staff humans to do the work, we are feeding the machines with the blood, sweat, and tears of human beings.” How is that for SRE noir?

Next we plan to dive into the SLOs of SREs and see what we can learn to make the SOC better!

A fully cooked version of this will be published on a Google Cloud blog, but I’d love some feedback and comments from either security or SRE side…

Thanks to Iman Ghanizada for ideas and brainstorming.

Related posts:


Originally published at Medium.

Monday, December 20, 2021

Anton’s Security Blog Quarterly Q4 2021

Sometimes great old blog posts are hard to find (especially on Medium) , so I decided to do a periodic list blog with my favorite posts of…


Sometimes great old blog posts are hard to find (especially on Medium) , so I decided to do a periodic list blog with my favorite posts of the past quarter or so.

Here is the next one. The posts below are ranked by lifetime views. This covers both Anton on Security and my posts from Google Cloud blog, and our Cloud Security Podcast too (subscribe).

Top 5 most popular posts of all times:

  1. “Security Correlation Then and Now: A Sad Truth About SIEM”
  2. “Can We Have “Detection as Code”?”
  3. “New Paper: “Future of the SOC: SOC People — Skills, Not Tiers”
  4. “Beware: Clown-grade SOCs Still Abound””
  5. “Revisiting the Visibility Triad for 2020”

Top 5 posts with the most Medium fans:

  1. “Security Correlation Then and Now: A Sad Truth About SIEM”
  2. “Beware: Clown-grade SOCs Still Abound”
  3. “Can We Have “Detection as Code”?”
  4. “Why Is Threat Detection Hard?”
  5. “A SOC Tried To Detect Threats in the Cloud … You Won’t Believe What Happened Next”

Top 5 Cloud Security Podcast by Google episodes:

  1. Episode 1“Confidentially Speaking”
  2. Episode 2 “Data Security in the Cloud”
  3. Episode 17 “Modern Threat Detection at Google”
  4. Episode 8 “Zero Trust: Fast Forward from 2010 to 2021”
  5. Episode 27 “The Mysteries of Detection Engineering: Revealed!”

Random fun new posts:

  1. “SOC Technology Failures — Do They Matter?”
  2. “Kill SOC Toil, Do SOC Eng”
  3. “Anton and The Great XDR Debate, Part 1”

Fun posts by topic.

Security operations / detection & response:

Data security:

Cloud security:

Enjoy!

Previous posts in this series:


Originally published at Medium.

Friday, December 10, 2021

SOC Technology Failures — Do They Matter?

Most failed Security Operations Centers (SOCs) that I’ve seen have not failed due to a technology failure. Lack of executive commitment…


img src: https://flic.kr/p/dwWHw5

Most failed Security Operations Centers (SOCs) that I’ve seen have not failed due to a technology failure. Lack of executive commitment, process breakdowns, ineffective workforces (often a result from poor management and lack of commitment … again) and talent shortages have killed more SOCs than any and all technology failures.

Example SOC Troubles from some presentation :-)

As we are working on the next SOC paper jointly with Deloitte (paper 1, paper 2, paper 3 coming out really soon), we came across the need to review some of the current technology challenges in the SOC. Hence this blog was born.

BTW, if somebody wakes me up at 3:00 a.m. and says “Anton, what is the top reason why a security operation center may fail?” I would name the loss of executive commitment. I have seen too many SOCs that decayed over time as management lost interest in their excellence, then in their performance and finally in their existence… Some of the noted breaches in the last decade can be traced to a SOC that was developed, refined and improved and then left to deteriorate (naïve outsourcing often played a role of a final nail in the coffin). On the flipside, there is nothing better to revive the presence of the SOC other than a major security incident — the trick is to keep the momentum going for years afterwards…

But I digress. Let’s stick to mostly technology focused failures. An astute reader will notice that in the list below, some of the purported technology failures are really process failures in disguise. I apologize for this in advance :-)

Scaling failures — Often when you get to POC an impressive new tool, the ability to scale is not truly understood and battle-tested until you onboard the tool into your workflow. There can be several dimensions to scaling from the ability to consume data, process data, store data, and make sense of all your unique data types at the scale you need it to be. Even the way you use the tool may cause unseen bottlenecks and affect your ability to scale. It’s far too often that vendors showcase a product’s abilities in it’s best and perfect use case, without little regard to scale. In other instances, the tool may have strong engineering behind it and truly have an ability to scale to many use cases, but a terrible user experience that makes everyone dread using it with the data volumes at hand (think drop-down menus with 2300 entries…) Or, the tool works, but only as long as you down-scope your collection efforts to below your actual security needs. Finally, the tool may “scale physically, but not economically” i.e. it will run at scale you need, but nobody can realistically afford it …

Tool deployed and then not operationalized sounds like a process failure, or a people failure. I lamented on this back in 2012, and this affliction has not truly subsided. However, why are some tools sitting unused in those boxes while others develop an active and passionate user bases? You don’t think it can be about the tool at all? Perhaps the tool vendor made some incorrect assumptions about how their technology is really used in the real world? Or, overestimated it’s value a bit for a particular type of client? Or a client may have thought that deploying the tool and turning it on was “self-service” — only to realize that they should have paid for the consulting partner to make it work.

In other cases, the tool “does runs, but does not work” (my name for failure to design the tool to be usable in real-world environments). There are also piles of tools that are deployed and used in production — but only 10% of the capabilities are utilized. Some technologies seem valuable, but are such a burden to maintain and use in real life that it is practically impossible. SOC should not spend time / resources managing such technologies. If your key SOC technology (whether you call it a SIEM or not) is that hard to manage, toss it, buy SaaS-based technology.

All in all, SOCs that have to manage too many tools suffer. A pile of boxes (yes, and cloud services) that need care and feeding, tuning, refining can overwhelm even a large team. Buy what you would use, and use what brings value!

Shiny new tool syndrome is still rampant in some SOCs. A new CISO comes in, tries to champion the implementation of a new tool, the CISO is gone after a short amount of time — like most CISOs, and then a new CISO comes in and tries it all over again. And tools multiply, getting less shiny day by day. Or, the SOC team champions a build-first strategy in areas where it is better to buy — as they figure out that building scalable solutions with open source tooling becomes way more challenging than initially thought. Yes, DIY SOC tools fail as well.

Data collection failures still plague many SOCs. Now, again, one can also blame this on people and processes (especially, those people in IT who just didn’t give us the data). However, in many cases it is in fact the tools (such as when a pre-cloud security monitoring tool is aimed at the cloud). Data collection does not stop at getting the data, because things like the lack of uniform data model, challenges with getting value out of raw data that is not enriched belong in the same category.

One sided visibility stack is definitely a tool challenge as well. A SOC that only uses a SIEM, only uses an EDR, or (if you are crazy) only uses an NDR is missing out. As I noted in the SOC visibility triad discussion (2020 refresh), there is a decent chance that in the near a future a SOC that uses a 2015-style triad of SIEM+NDR+EDR is also missing out, such as on the application security telemetry, as organizations develop more security use cases for observability data.

Somewhat related, old tools that don’t cover new environments sounds like a tool challenge, and the above cloud example works here as well. Frankly, many traditional SOCs suffer with cloud, with containers and with other modern IT technologies and environments (old example). You don’t need to buy the whole lot (CWPP, CSPM, CASB, SSPM, CNAPP, etc), but you do need to be mindful of public cloud visibility gaps in your SOC.

Along the similar line, tools that promise “a single pane of glass” usually don’t deliver that. Don’t wait for a vendor to invent a “single pane of glass”, it either won’t come, or it will come out of the box “pre-broken.” I am not sure if the mesh is the answer either. Note that integrated tools are hard to adopt if you already have siloed tools that are at least partially successful (will you ). Would you buy an XDR that includes an EDR if you are happy with your different EDR?

Automation is sometimes more work than value — even with SOAR. Again, this may be seen as a people challenge (“hey, you just don’t have enough security developers in your SOC ….oh wait … you have none”). In real life, automation is of course a benefit, but it does require work to deliver. Indeed, this reads like another people/process challenge, but perhaps tools can eventually deliver real — not excessively optimistic — low code/no code security automation?

Thanks to Iman Ghanizada for the ideas and contributions to this post!

Related posts:


Originally published at Medium.

Thursday, October 21, 2021

Do You Trust Your SIEM?

My admittedly epic (but dated) post “Security Correlation Then and Now: A Sad Truth About SIEM” mentioned the issue of TRUST as it applies…


My admittedly epic (but dated) post “Security Correlation Then and Now: A Sad Truth About SIEM” mentioned the issue of TRUST as it applies to SIEM. Specifically, as a bit of a throwaway comment, I said “people write stupid string-matching and regex-based content because they trust it. They do not — en masse — trust the event taxonomies if their lives and breach detections depend on it.”

This post is an exploration of that theme.

Where trust hides when you are using a SIEM-like tool, especially the cloud-based one?

In my view, skewed by looking at both the internals and usage of SIEM products for almost two decades, there is a lot of trust hides inside the SIEM

Let’s walk through the tenure chain

  1. If you follow SIEM vendor recommendations on how to configure logging, you trust the vendor to provide the correct settings for your use cases
  2. You then trust the SIEM collector (whether an agent, an API pull script or a syslog sink) to collect the logs intact, not drop data over 1400 character limit, not time out, not get overrun by volume
  3. You also trust SIEM to collect the logs more or less in sequence
  4. Before full parsing starts, you are expected the SIEM to understand the time stamps correction so the logs show up in search, consoles, etc
  5. Ah, and you also trust your SIEM to tell you that logs are no longer flowing
  6. You trust the SIEM vendor to create the right schemas and data structures to put extracted data to (sure, “schema on read” make this easier, but this trust still lurks)
  7. Next, you trust SIEM to extract the fields from the logs and assign the data to the correct structural elements (“trust that parsing works”)
  8. If there is a normalization (a unified schema for many log types), you trust the SIEM to not drop the fields extracted from logs; you trust that the structured data represents the raw data in a useful manner and supports your use cases
  9. If there is a taxonomy (wow, much 2002 SIEM!), you trust the SIEM to map the events to the correct category and not to confuse “password guessing” with “logon failure” or whatever
  10. You then trust that the detection logic (rules) is written correctly so that nobody mistyped “context.asset.vulnerability.severity” as “asset.context.vulnerability.severity” in a rule they wrote.
  11. In parallel, you trust that the storage of raw and structured data is sound, and that indexing does not miss any data you collected.
  12. If there is ML … well, let’s not even go there. Because there is a lot of trust hiding here, right in that ML unicorn hideout. Suffices to say, any ML-based operation inside the product implies a bit of trust.
  13. Naturally, you also trust that search, reporting and visualization interfaces represent the data correctly.
  14. With cloud SIEM, you trust the vendor not to lose, disclose, confuse or otherwise corrupt the data they store for you. There is also trust that the vendor won’t use the data for anything “off-label” and won’t let anybody else do it.

In light of the above looooooong list, I can see why some people choose glorified grep over an advanced product that does a lot of work for them. If a SIEM product violated their trust, they are less inclined to trust the next vendor. And even less, the next.

BTW, how do we fix it? Transparency! Radical transparency in parsers and detection code, data schemas, transparency in collector configuration, retention policies, etc. Back in 2015, I noted that people are having trouble trusting non-deterministic security. The same arguments apply to any/all black boxes, even if they are full of rules and not ML.

Related blog posts:


Originally published at Medium.

Thursday, September 16, 2021

Anton’s Security Blog Quarterly Q3 2021

Sometimes great old blog posts are hard to find (especially on Medium), so I decided to do a periodic list blog with my favorite posts over…


Sometimes great old blog posts are hard to find (especially on Medium), so I decided to do a periodic list blog with my favorite posts over the past quarter.

Here is the next one. The posts below are ranked by lifetime views. This covers both Anton on Security and my posts from Google Cloud blog, and now our Cloud Security Podcast too!

Top 5 most popular posts of all times:

  1. “Security Correlation Then and Now: A Sad Truth About SIEM”
  2. “Can We Have “Detection as Code”?”
  3. “New Paper: “Future of the SOC: SOC People — Skills, Not Tiers”
  4. “Beware: Clown-grade SOCs Still Abound””
  5. “New Paper: “Future of the SOC: Forces shaping modern security operations””

Top 5 posts with the most Medium fans:

  1. “Security Correlation Then and Now: A Sad Truth About SIEM”
  2. “Beware: Clown-grade SOCs Still Abound”
  3. “Can We Have “Detection as Code”?”
  4. “Why Is Threat Detection Hard?”
  5. “A SOC Tried To Detect Threats in the Cloud … You Won’t Believe What Happened Next”

Top 5 Cloud Security Podcast by Google episodes:

  1. Episode 1“Confidentially Speaking”
  2. Episode 17 “Modern Threat Detection at Google”
  3. Episode 8 “Zero Trust: Fast Forward from 2010 to 2021”
  4. Episode 2 “Data Security in the Cloud”
  5. Episode 27 “The Mysteries of Detection Engineering: Revealed!”

Now, posts by topic.

Security operations / detection & response:

Data security:

Cloud security:

Random recommended resource: our new paper on 10X SOC is here: “New Paper: “Autonomic Security Operations — 10X Transformation of the Security Operations Center””

Enjoy!

Previous posts in this series:


Originally published at Medium.

Anton’s Security Blog Quarterly Q3 2021

Sometimes great old blog posts are hard to find (especially on Medium), so I decided to do a periodic list blog with my favorite posts over…


Sometimes great old blog posts are hard to find (especially on Medium), so I decided to do a periodic list blog with my favorite posts over the past quarter.

Here is the next one. The posts below are ranked by lifetime views. This covers both Anton on Security and my posts from Google Cloud blog, and now our Cloud Security Podcast too!

Top 5 most popular posts of all times:

  1. “Security Correlation Then and Now: A Sad Truth About SIEM”
  2. “Can We Have “Detection as Code”?”
  3. “New Paper: “Future of the SOC: SOC People — Skills, Not Tiers”
  4. “Beware: Clown-grade SOCs Still Abound””
  5. “New Paper: “Future of the SOC: Forces shaping modern security operations””

Top 5 posts with the most Medium fans:

  1. “Security Correlation Then and Now: A Sad Truth About SIEM”
  2. “Beware: Clown-grade SOCs Still Abound”
  3. “Can We Have “Detection as Code”?”
  4. “Why Is Threat Detection Hard?”
  5. “A SOC Tried To Detect Threats in the Cloud … You Won’t Believe What Happened Next”

Top 5 Cloud Security Podcast by Google episodes:

  1. Episode 1“Confidentially Speaking”
  2. Episode 17 “Modern Threat Detection at Google”
  3. Episode 8 “Zero Trust: Fast Forward from 2010 to 2021”
  4. Episode 2 “Data Security in the Cloud”
  5. Episode 27 “The Mysteries of Detection Engineering: Revealed!”

Now, posts by topic.

Security operations / detection & response:

Data security:

Cloud security:

Random recommended resource: our new paper on 10X SOC is here: “New Paper: “Autonomic Security Operations — 10X Transformation of the Security Operations Center””

Enjoy!

Previous posts in this series:


Originally published at Medium.

Friday, September 03, 2021

Anton and The Great XDR Debate, Part 2

As you recall from “Anton and The Great XDR Debate, Part 1”, there are several conflicting definitions of XDR today. As you also recall, I…


As you recall from “Anton and The Great XDR Debate, Part 1”, there are several conflicting definitions of XDR today. As you also recall, I never really voted for any of the choices in the post.

While some of you dismiss XDR as the work of excessively excitable marketing people (hey … some vendor launched “XDR prevention”, no way, right?), perhaps there is a way to think about it from a different perspective.

What if we don’t look at XDR from either EDR or SIEM angle, but we look at it from first principles? Namely, what kind of detection and response toolset would you like to have in your life?

Before we go further, I wanted to share a personal story about my first encounter with XDR. When I was an analyst, many vendors showed me their tools and some claimed “XDR.” In most cases, my instinctive reaction was to argue with them, because I very clearly saw “SIEM” (or pieces of SIEM) in what they showed me …

Admittedly, my thinking has been colored by SIEM since 2002 when I joined my first SIEM vendor (a SIM vendor, to be precise). I did tend to treat every technology that analyzes log files and perhaps other similar telemetry as a SIEM. In light of this, it was very hard for me to see anything as “real” XDR and coincidentally see any merit in XDR idea as such. Note that it also made it very easy for me to first latch onto the “XDR is improved EDR” route, which I later abandoned …

However, why don’t we reset the perspective?! For this, we need to imagine the world without SIEM. Imagine this technology was never born (as SIM and SEM) back in the murky — but not cloudy — 1990s…

What then? Anybody would be free to invent a technology to analyze security telemetry (logs, endpoint traces, traffic) and call it whatever they want. Perhaps somebody will have invented CLA (Compliance Log Analysis) or SEC (Security Event Correlation) or perhaps SDS (Security Data Science) or SILS (Security and IT Log Search)?

Now, in this alternative world, what if you set out to invent the best technology to analyze various types of telemetry for threat detection and response (rather than to reinvent SIEM)? Let’s say you want to collect data from existing logs, collect traffic metadata and also make your own telemetry on the endpoint. Then you want to analyze the data, come up with conclusions, guide humans to proper security outcomes and occasionally automate the responses. You also want the tool to offer “out of the box” value as well as customizations/programmability. Finally, why don’t we call this technology we set out to invent “XDR”, just for fun?!

Imagine if we don’t have to relieve SIEM history and be biased by SIEM’s past as log retention and compliance reporting of the mid-2000s, as well as alert management technology of the early 2000s. Imagine we are doing the work today for today’s use cases and today’s threats. Imagine what you would build if you don’t have to fear the analysts poking fingers at you and saying “Ha, what a SIEM you got here, buddy?!”

What would we do differently? Wait for Part 3 for some of the answers.

Related blog posts:


Originally published at Medium.

Monday, August 30, 2021

Kill SOC Toil, Do SOC Eng

As you are reading our recent paper “Autonomic Security Operations — 10X Transformation of the Security Operations Center”, some of you…


As you are reading our recent paper “Autonomic Security Operations — 10X Transformation of the Security Operations Center”, some of you may think “Hey, marketing inserted that 10X thing in there.”

Well, 10X thinking is, in fact, an ancient tradition here at Google. We think that it is definitely possible to apply “10X thinking” to many areas of security (at the same link, they say that sometimes it is “easier to make something 10 times better than it is to make it 10 percent better”). However, our beloved domain of cyber is full of skeptics and cynics, as well as well-meaning people who just can’t take the exaggerations anymore…

With this post, I wanted to explore one particular area of 10X possibility. This area is “toil”, an SRE term that is crisply defined in Chapter 5 of Google SRE book. If you read the above short and fun chapter, and then look back at your SOC, you will realize that 100% of what a typical SOC analyst does on a daily basis fits the definition of toil.

Here in the post, we will present two components of the definition that are the juiciest, in my opinion.

Toil is the kind of work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows.
“If your service remains in the same state after you have finished a task, the task was probably toil.”

Does this remind you of SOC analyst work? Well, it is an exact match, no need to write any regexes here…

Now, some of you may say at this point: but Anton, SOC work is inherently like this. Attackers come, alerts trigger, we clear them, adjust, tune, response, rinse, repeat. If our IT remains in “the same state” after this, it is good, not bad, right?

Well, I bet the sysadmins and IT operations people of the 1990s thought the same when responding to availability incidents: “but our work is inherently like that”, and they were proven wrong by the SREs.

So, let’s talk about how we can make your SOC behave more the way good SRE teams do. But before we go there: where is that 10X?

Well, if you have increase in attacks, increase in assets under protection or increase in environment complexity, your “toil-based” SOC will need to grow linearly with all those changes. To get to 2X the attacks or to 2X increased scope (such as cloud added to your SOC coverage), you need 2X the people, and sometimes also 2X budget to spend on tools.

However, if we really transform the SOC based on the principles we discuss, your effort increase may range from nothing to minimal. Hence, you WILL achieve 10X effectiveness in real life, not on a marketing glossy. The evolution of security operations in general and SOCs in particular is heavily dependent on a drive towards an engineering-first mindset while operating modern, more secure systems at large scale. So, you can’t “ops” your way to SOC success, but you can “dev” your way there, just like we do at Google!

So, how can we put these and other SRE lessons to work in your SOC?

First, educate your team on how SRE philosophies can be implemented in SOC. Find opportunities to do team-building exercises and empower your team to define this cultural transformation. Driving a cultural shift requires an inspired, motivated, and disciplined team — as well as specific skills in this area.

Next, seek to minimize your ops time to 50%, gradually. Try spending the remaining 50% on improving systems and detections with an “automate-first”, engineering mindset. BTW, engineering here is NOT the same as writing code: “Engineering work is novel and intrinsically requires human judgment. It produces a permanent improvement in your service, and is guided by a strategy.“

“Commit to eliminate a bit of toil each week with some good engineering” in your SOC. Here are some SOC examples: tweak that rule that produces non-actionables alerts, write a SOAR playbook to auto-close some alerts, script the test for log collection running optimally, etc, etc.

One route to go is hiring security automation engineers who have operations experience, or have the ability to ramp up quickly. The right person can set the tone for leading your whole team through evolution to “SRE-inspired” SOC.

We think that the largest current and future challenges in Security Operations can be solved with this approach. Otherwise, 30+ years of SOC work and we’re still facing the age-old challenges we had in the past (believe it or not, “too many [IDS] alerts” was a SOC challenge in 2002!).

Huge thanks to Iman Ghanizada for his contributions to this post.

Related blog posts:


Originally published at Medium.

Tuesday, August 24, 2021

To rephrase, XDR is a modern detection platform that basically avoids the 20 years of SIEM…


To rephrase, XDR is a modern detection platform that basically avoids the 20 years of SIEM misfortune :-)


Originally published at Medium.

Friday, August 06, 2021

Anton and The Great XDR Debate, Part 1

I know you may hate me for this, but I‘ve been finally tempted into the Great XDR Debate.


I know you may hate me for this, but I‘ve been finally tempted into the Great XDR Debate.

Here, if you want TL;DR, my position on XDR today is “wait and see” (boring, huh?). Unlike some of my esteemed former colleagues, I don’t really have a horse in the race.

First, a very brief bit of history. The origin of the term XDR (Extended Detection and Response) is disputed. Wikipedia (entry, reviewed 8/6/2021) has us believe that Palo Alto invented the term “in 2018.” Josh Zelonis points out that he in fact invented the term. My Googling for its earliest use didn’t yield any revelations.

Today, I see several visions of XDR that are somewhat conflicting. So, let me outline them the way I understand them.

  • “XDR as improved EDR” or “EDR+” vision; on the analyst side, we have Forrester with illustrious Allie Mellen (example, FAQ) and on the vendor side we have many EDR vendors (example, example). This is definitely a defensible view of XDR as EDR with more data collection outside of the endpoint. Thus defined, XDR can nicely coexist with SIEM, but may also collide with it later on.
  • “XDR as ‘UTM’ for D&R” view considers XDR to be a combo toolset (likely from a single vendor); Gartner, for example, says XDR is “vendor-specific” and “natively integrates multiple security products into a cohesive security operations system.” This is also a defensible view of XDR as “bundled D&R toolset.” Here, we have XDR on a rapid collision course with SIEM. Smart SIEM vendors are coopting it.
  • “XDR as EDR + NDR” with some SOAR added and SIEM not added (example). This view is also defensible, and it seeks to dethrone SIEM from its central spot in many SOCs. This vision of XDR can nicely coexist with SIEM, but may also collide with it as SIEMs collect more endpoint and network data.
  • “XDR = SIEM” line of thinking considers XDR to be essentially a SIEM 3.0; it avoids the debate of XDR vs SIEM by stating that XDR is in fact SIEM rebranded.
  • Some other combination of security technologies in this area (XDR = SIEM + EDR example, find others on your own…). This is all over place and, frankly, does not deserve my analysis.
  • “XDR as a senseless marketing term” (example) or “random security technology rebranded”; no comment, make your own conclusions.

So, some points of agreement:

  • XDR is cloud-native. There is no on-premises XDR, and if you think you have one, sorry, you were lied to…
  • XDR is about detection. There is some debate over how much response needs to be there, and what it even means (workflow? investigations? action? playbooks?), but detection is there for sure.
  • XDR may be related to EDR, but the nature of the relation is under debate.
  • XDR may collide with SIEM, and these technologies may merge (just like SIEM and UEBA did)

As a minor aside, somehow I never got to get myself to care deeply about “open” vs “native” XDR. If we don’t agree on what XDR is, this is not the time to debate variations and subspecies of it…

And here is my favorite (Really?! No, not really…) list of XDR vs SIEM comparisons, just for fun:

There you have it! Not bad for a Friday afternoon? :-)

Related blog posts:


Originally published at Medium.

Friday, July 30, 2021

How to Make Threat Detection Better?

I keep coming to the same topic over and over — why are we still bad at detecting threats?


I keep coming to the same topic over and over — why are we still bad at detecting threats?

I’ve lamented on this a few times, either touching on general difficulties with detection, its uncertainty or highlighting the fragile detections people write. I also noted the critical role of context in threat detection, which seems to imply that the best detections are written on-site by each team, and not by the vendors in their comfy little labs …

Here, I want to continue the conversation on detection quality. Also, I want to look for some ideas that can help everybody. How do we get the mainstream companies to improve their detection quality? What does it mean to have “good” detections? How do we get to more and better detections? More curiously, how do we get to better detections that are also developed rapidly (to avoid the “good/fast/cheap — pick any two” syndrome that led to many fragile IP- or hash-based detections)?

Now, some people will say that there are NO good answers. Hence, their only answer is to have organizations outsource detection to a quality MDR or an MSSP. Sure, that would solve the problem for them (It would, right? Right?!), but it would really just shift the bigger picture problem resolution to their MSSP. Specifically, how would many MSSP teams write good detections without access to the best of the best detection authoring talent? Worse, sometimes said MSSPs will try to “mass produce” detection rules hence losing local, on-site detection context. Even more worse, sometimes you can spot an MSSP that simply relies on commercial SIEM stock detection rules. Economies of scale here may translate into missed attackers …

Also, there will be a category of people that will say that the answer is …. drumroll … machine learning. However, guess what makes detection difficult for many security scenarios? The fact that we do not have reliable data on attackers and attacker behavior. Also, guess what is required for machine learning to work well? Just the data we just noted the lack of. No way, right?! So, ML unicorn cavalry is not coming.

Well, instead of just lamenting and highlighting the challenges, what can we do?

Let’s try to decompose the good detection into elements. What do you need to write a good detection? (Note that we kinda sorta punted the definition of “a good detection” to a later post…)

  • You need some knowledge of the attacker; specifically, you need elements of such knowledge that you can latch onto in your detections. For example, if you somehow know what the attacker is thinking about, you can not necessarily write rules on that …
  • Naturally, knowing the attacker goal is useful, because this may point at the types of telemetry you’d use to detect them and from what parts of the environment (Database logs? Application traces? Cloud authentication logs?)
  • In addition to attacker knowledge, you need some useful telemetry that you can apply that knowledge to. Note that this equally applies to both rule-based and ML -based detections, but you will likely need more telemetry if you were to develop your own ML detections.
  • Knowledge that separates good from bad, attacker knowledge alone won’t cut it as there are legitimate activities that look the same in some regards. Unless you know “good” well, your detections will become “false positive” machines.
  • Naturally, some way to express or declare detections, a language (like say YARA-L or Sigma) or a method at least. Further, it goes without saying that you would need some kind of tool to actually implement that method.
  • Finally, it is fair to say that you would need some test data or a test environment, or at least a testing approach, to verify that what you created actually does what you wanted to do (and does not do a lot of other unneeded things).
  • Anything else I missed?

Now, reading the above surely does not mean that you will write good detections, but IMHO you will have a fair shot at that. Frankly, this is much more nuanced IRL.

For example, during a recent SANS webcast we went through a bunch of rules and derived these additional lessons.

Recent SANS Webinar on Detection

By the way, this post ended up being more of “an incomplete thought” then a solid framework, but I hope you forgive me. More to come!

Related blog posts:


Originally published at Medium.

Wednesday, July 21, 2021

New Paper: “Autonomic Security Operations — 10X Transformation of the Security Operations Center”

It is with much excitement that we announce a new paper about transforming your security operations; it is published under the Office of…


It is with much excitement that we announce a new paper about transforming your security operations; it is published under the Office of the CISO at Google Cloud.

This work is focused on our vision as well as our lessons in building effective security operations for the future. We spent a lot of time thinking about what to call the new model. We ultimately settled on the name “Autonomic Security Operations” for the vision (note that the previous contender was “10X SOC”, which one do you like more?)

Now, when we say “autonomic” here, we do not mean “without people” or “fully automated.” All of us here agree that it is quite impossible. When we think of autonomic security operations, we think of an approach that leverages automation, but also focuses on being agile, adaptive to rapidly changing environments, and the one that utilizes people with dramatically increased effectiveness.

While I want you to read our paper, let’s now think about two interesting and useful ideas from it.

One question that I’ve been trying to explore, such as in this podcast, is: we do know that DevOps and SRE approaches revolutionized how IT is managed and practiced. I’ve been thinking about what can possibly be a similar revolution in security and, more specifically, in security operations.

The answers, of course, are obvious. What is the “O” in SOC? Operations. What happens to operations in the DevOps model? It becomes fused with development, thus toil decreases, while automation increases. Less toil, but more code.

Naturally, we see the same start to slowly happen in good SOCs or detection teams (if you insist on being “SOCless”). In the SOC context, this is about detection engineering, detection as code and related concepts.

Just as naturally, and just as it happened in IT first, many people don’t like these changes. Remember the comments to my blog post about how they don’t want anything “as code” because this means they have to code? I’m pretty sure that Windows NT system administrators of the 1990s also did not want to become part of DevOps…

Next, what about the other part of the SOC, namely the “C”? Will SOC remain at the center of anything? Will it always be a crowded room with fancy monitors?

As we watched current events affect security operations, we learned that SOC as a big room, full of people, may in fact disappear. However, I’ve long argued that SOC is first a team that focuses on detecting threats and helping respond to them and so it will not vanish anytime soon. What if the future SOC would be a center in the same sense as “a center of excellence”, but not a central physical location anymore?

So, what happens if a SOC both becomes infused with DevOps approaches and stops being a centralized silo?

As you learn from our new paper on autonomic security operations, we are presenting this very vision of future security operations. Using the lessons from DevOps and SRE approaches, we propose that a security operations center would become less about operations and would become less of a center.

These changes are critical, if not to say truly necessary, to deal with the business, IT landscape and threats of the future. Below — and of course in the paper — you will see many examples of how we describe such transformed SOC and how you can take steps towards making your security operation center work that way.

So, please see a tiny subset of my favorite quotes from our paper:

  • Autonomic Security Operations is a combination of philosophies, practices, and tools that improve an organization’s ability to withstand security attacks through an adaptive, agile, and highly automated approach to threat management. “
  • “An unfortunate common theme of many cloud transformations is that the SOC requirements get deprioritized when organizations have tight timelines and budgets to drive their teams to the cloud. The reason being, most SOC teams are too busy fighting fires and don’t have the spare cycles to focus on adapting their use cases to cloud workloads and modernizing their own infrastructure.”
  • “To be 10 times more effective with the people component, your SOC cannot achieve this by increasing the personnel by a factor of 10. As of today, both threats and technology resources that need effective security are increasing at a much faster pace than people entering the workforce. […] It is absolutely impossible for most organizations to 10x their headcount in a SOC.”
  • ”At Google and across other industry-leading security operations teams, the role of an analyst is not simply to manage cases and perform tier-1 level work. Analysts are engineers, architects, project managers, and are empowered to be leaders of their subject matter focus. At such a SOC, the concept of Level 1 to Level 3 analysts is a thing of the past, rather, you should organize teams based on aligning skills to the use cases that fall under their purview.”
  • The SOC can only truly be 10X and transformative if it also has strong influence over the upstream elements of the security lifecycle. You can make a significant impact on the amount of alerts that get into your SOC if your team has a strong integration with your DevOps practice. A deep understanding of how infrastructure and applications are securely built, deployed, and managed across your organization paired with your ability to influence this design can only improve your ability to catch attackers at their earliest onset, or even better, prevent them from getting in entirely.”

But really, go and read the paper here. We have a lot more value there!

Also, read about the related solution launch here.

Related blog posts on SOC:


Originally published at Medium.

Monday, June 21, 2021

Anton’s Security Blog Quarterly Q2 2021

Sometimes great old blog posts are hard to find (especially on Medium …), so I decided to do a periodic list blog with my favorite posts…


Sometimes great old blog posts are hard to find (especially on Medium …), so I decided to do a periodic list blog with my favorite posts of the past quarter or so.

Here is my third. The posts below are ranked by lifetime views. This covers both Anton on Security and my posts from Google Cloud blog, and now our Cloud Security Podcast too!

Top 3 most popular posts of all times:

Top 5 posts with the most Medium fans:

Now, posts by topic.

Security operations / detection & response:

Data security:

Cloud security:

Top Cloud Security Podcast episodes:

BTW, the new website for our podcast is here (subscribe, please!)

Enjoy!

Previous posts in this series:


Originally published at Medium.

Dr Anton Chuvakin