Tuesday, October 24, 2023

Build for Detection Engineering, and Alerting Will Improve (Part 3) [Medium Backup]

 This blog series was written jointly with Amine Besson, Principal Cyber Engineer, Behemoth CyberDefence and one more anonymous collaborator.

In this blog (#3 in the series), we will start to define and refine our detection engineering machinery to avoid the problems covered in Parts 1 and 2.

Adopting detection engineering practices should have a roadmap and eventually become a program, effectively re-balancing where efforts go in a SOC by investing in high quality detection creation (and detection content lifecycle, of course).

Put simply, if you spend more time building better detections, then you spend less time triaging bad alerts. Simple, eh? If it were simple everybody would do it!

Embracing leaner, consistent, purpose-driven detection workflows is key, and you may want to assert where you land on these key areas:

⚒️ Breakdown and Backlog: Build a continuous roll of issues corresponding to threats to analyze, and detection requirements to implement. What you are doing next for detection content should be clear in most cases, and yes, this is security, so there will be nasty surprises. Eventually, the only unpredictable tasks would be the genuine rare surprises — your routine detection work would not surprise you.

🌊 Smoothen yer workflow : Remove people interfaces that don’t work, define minimal ceremonies, and put content reviews in the right places. Shorten approval times for releases, ensure detection quality reviews are followed up. If dealing with a MSSP/MDR, make sure to lay the governance structure for building custom content jointly with you (JointOps, not finger-pointing).

☣️ Embrace threat-driven approach: Study adversary tradecraft in detail before making educated calls on what to detect, and where/how. Starting from available telemetry data will more often than not be prone to bias, inefficiency and mistakes.

πŸ’‘ Embed Intel: A CTI enclave in a SOC will often provide higher ROI than dealing with a separate team, as they understanding SOC needs better, and plug in directly into their processes (this is full of nuance, so YMMV)

⚡ Lower Intel-to-Rule KPIs : Quantify how long it takes to go from intel input (ItR for Intel to rule, ha-ha, we just made up a new acronym! Take that, Gartner! ;-)) to an actual detection; with as much granularity as possible. An good ItR metric would be to transform a high risk threat intel into working detections in hours or in a few days.

πŸ‘€ Visibility over assumptions : If you can not answer accurately within minutes what your detection coverage is and which shortcomings it has — you likely need to start parsing your detection library and threat modeling into qualitative metrics and make that data transparent to the DE team.

πŸš€ Release Soon, Release Often! : There are new threat variants every week, and detection engineering scales directly on the quality of the intelligence input. This is where modern software engineering practices come handy.

πŸ”₯ Quantify, Measure, Orient Operations : Define what healthy operations look like : what FP rate is acceptable for new detections ? What turnover time for tuning should be aimed at ? Where are quick wins, where are detection gaps ? Where is capacity spent, should it be reassigned to more urgent priorities ? Where are process bottlenecks ?

🦾 Automate the hard — and boring — part : Everything produced during R&D should generate rich and structured knowledge bases, metadata, and metrics. Version your detection library, and roll it out with CI/CD toolings. BTW, this advice alone is worth the price of this blog!

πŸ’Ž Where does ATT&CK fit in the DE picture ?

While a great model to categorize adversary tradecraft, and a necessary tool in the detection engineering arsenal, ATT&CK is often overused as a palliative measure to map detections and generate coverage maps — by skipping the detail (“Do you cover T1548? — Ehhh… YES?”)

This does not accurately represent SOC detection performance, since techniques can be fairly broad (since it is the purpose of the taxonomy to normalize specificities), and rule quantity doesn’t equate quality (not when they drown analysts in False Positives).

Detection hints from ATT&CK are also rather generic, since a Technique is itself a concept which clusters different procedures together. Thus, while ATT&CK can give a direction of what a SOC needs to develop, it doesn’t give a way to achieve detection objectives; which is the detection engineer core concern.

The journey to Detection Engineering maturity is hard, but you should now have a clearer perspective to smoothen the journey toward building better detections.

But it all starts with quality input : in our next blog post, we’ll look at in more detail what exactly a Detection Engineering team needs from Threat Intelligence to be fully informed, and propose collaborative models. Stay tuned!

Related blog posts:

Focus Threat Intel Capabilities at Detection Engineering (Part 4) [Medium Backup 10/24/2023]

 

This blog series was written jointly with Amine Besson, Principal Cyber Engineer, Behemoth CyberDefence and one more anonymous collaborator.

In this blog (#4 in the series), we will start to talk about the elephant in the room: how intel becomes detections (and, no, it is not trivial)

Detection Engineers are often picky with the intelligence they receive … and for good reasons. Incomplete, too high-level or overly specific data leads to long analysis time, bias and ultimately inconsistent detection quality and detection coverage gaps. On the other hand, the intel team may process so much intelligence data that it can be a real challenge to understand what the detection team is even expecting. So, in many cases there is an “intelligence — detection gap”, of sorts. This is our topic in this part of the blog series.

To start, in the Detection Engineering Maturity Matrix by Kyle Bailey, intel is referenced on the upper maturity level of the “Threat Operations” Category — and historically this has held true. In some cases, Detection Engineering (DE) teams do not get valuable threat intel at all, and have to do their own threat research as a result (if this is not like that in your organization, congratulations!). But only working in tandem allows us to reach best-in-class results for detection!

Today, let’s explore what’s wrong (for detection, specifically!) with much intelligence data provided, what’s good intel (for DE purposes), and how we can work better together (but do look at the previous posts Part 1, Part 2, Part 3 even if this one is most interesting for you…)

Intelligence is Often Lacking in TI

A frequent observation in CTI and DE team relationships is that DE is asking CTI what they should focus on, what threat they should address first, and CTI cannot deliver precise answers beyond a bundle of ATT&CK techniques (which are largely the same for the large majority of threat actors and organizations). As we’ve seen in our previous blog post, ATT&CK is not sufficient for more technical DE needs and for detailed planning of the priority detection engineering activities.

In some lucky cases, useful intel is often forwarded directly without processing. Similarly, if the IR team is effective and mature, they will send useful data back to CTI and DE teams. In this case, naturally, it’ll always be after the fact (i.e. after an incident), where the DE target is to develop ahead of an incident.

These observations beg the question — what is good intel for detection engineering purposes? For the most part, we can derive some common bad practices:

  • πŸ’… Just ATT&CK techniques: just giving a set of technique ID to a DE is not valuable intel (“Attackers used T1548“ — “Great! What do we do with it!”) : it may take weeks in best cases to properly develop detections for the scope of a single technique on one platform, and technique-level detections are very hard to reach without resorting to “false positive”-prone approaches. Realistically, most of the time, only well-known procedures will be covered.
  • 🧐 Generalist “intel”: everyone knows brute force is a bad thing — but what did the actor do exactly that we could detect? DE teams need technical threat descriptions. There is a large set of ways to brute force something and an ever larger set of ways to detect the resulting activities.
  • πŸ’” Hard to parse with detection in mind: Nicely documented CTI reports have a bad tendency of containing the entire attack path and a lot of other exciting yet irrelevant for detection details. For DE teams, that now means breaking down the report into a dozen singular threat elements that each needs to be evaluated and processed into detection.
  • πŸ‚ Lack of variety: Yes, Email and Windows are exploited left and right. But we also have Cloud, SaaS, Remote Workers and other unique threat vectors to look into. If your company uses multiple platforms, but your intel team feeds your DE team with Windows stuff only, attackers will get in and you won’t know it.
  • ⚖️ No value add, just copy from the feed: CTI ultimate purpose in an organization is not to compile external content (some of questionable provenance), but to analyze threats through the prism of your business operations and IT systems in place. “Detect threats that matter” is easy to say, but really hard to do!
  • 🫠 Doesn’t understand the threat: An intel analyst should deeply understand the threat they communicate to DE to accelerate further processing, and ensure enough information is delivered. DE is a process, where any threat can be processed into detection, but only as long as we understand the issue at hand. Ideally, bi-directional CTI-DE comms are established…
  • 🐒 Sloooow: DE aims at moving at the speed of intelligence, thus it needs intel which is closely following the latest developments. Intel done for research can be slow, but intel done for detection cannot be.

🧬IOCs only get you so far

CTI teams spend a considerable amount of money building an infrastructure that captures, stores, and distributes IOCs to various intel consumers (including whatever detection platforms you use, SIEM, EDR, etc). For detection engineers, handling those IOC to scan incoming logs, search historical ones or enrich existing alerts is a very efficient catch-all. However, most threats nowadays are highly effective at avoiding simple markers and naive rules.

Modern malware, domain generation algorithms and other approaches introduce fuzziness in the behavior monitoring systems will see, and require different approaches to be effectively detectable. Starting from studying the exact procedures seen in the wild, we can model the common behaviors into low to high level detection objectives that remain highly targeted (and thus without resorting to trying to detect an entire ATT&CK technique, which may be costly and lengthy compared to the detailed intel received).

For example, a registry key used for persistence may change continuously and not be a great way to detect, but the presence of certain patterns in the registry value (certain non-latin characters, mentions or prefixes), very indicative of a certain actor may be great detection objective to successfully and quickly detect an entire campaign. But coming up with this takes skill, time and good intel!

For CTI to deliver a higher value-add to creating new detections, it must have a stronger focus into compiling intelligence frequently, and delivering the correct level of technical knowledge to detection engineers. Level of abstractions matters more (“they brute forced” is too much abstraction, but “they tried this password 37 times” is definitely too little)

😀 Massive PDFs Ain’t Intelligent!

CTI teams are encouraged by their TIP (threat intelligence platform, which sources and catalogs third-party threat reports) to transform multiple PDF into a single one, presumably so that it would be easier for humans to read. This is perhaps a vestige or non-cyber intelligence, which was often compiled in long paper reports.

Such aggregated PDFs are, frankly, an annoyance for most detection use cases! In fact, for good DE teams, this is a superb method to make them go mad, as they parse the report in weeks (yes, weeks) into individual objects, then research each one of them to identify detection opportunities. It discourages prioritizing particular aspects of the report, since the entire analysis blocks a DE, which then needs to present and act upon a large body of work, instead of neatly packaged work items. Consider that intel is only “intelligent” if it is processed in a way that doesn’t require further manual work and more analysis to understand.

Structuring expert teams

Detection Engineering is such a novel function that the reality is simply that many CTI teams are unprepared for the additional work needed to support such a function that intends to build on intelligence, rather than simply read it.

As a result, many DE teams today actually work very independently, combining threat intelligence and threat modeling into their processes, where in a target operating model, both of these inputs should be expected from threat intelligence analysts. In many cases, when a CTI team exists, they can’t scale to the DE challenge (and, yes, there are beautiful examples of effective CTI-DE collaboration at scale).

Creating new services in the CTI functions thus requires thinking about the organization structure, and how to make teams integrate. Next in the series? An operations model! Very fun!

Related blog posts:

How to Banish Heroes from Your SOC? [Medium Backup 10/12/2023]

 This blog was born from two parents: my never-finished blog on why relying on heroism in a Security Operations Center (SOC) is bad and Phil Venables “superb+” blog titles “Delivering Security at Scale: From Artisanal to Industrial.”

BTW, what is heroism? Isn’t that a good thing? Well, an ancient SRE deck defines “IT heroism” as relying on “individuals taking upon themselves to make up for a systemic problem.” As those who have seen the inside of a SOC can attest, this is, ahem, not entirely uncommon in many Security Operations Centers.

If you recall our Autonomic Security Operations (ASO) vision, we advocate for automation, consistent processes and systematic, and engineering-led approach to problems. Yet in real life heroes are very much needed at many SOCs for their routine operation. This is the essence of our conundrum: human heroism is usually good, but a system that relies on heroes for routine operation is bad.

Here is a great quote from another domain that explains this even better:

The need for heroism is revealing the fact that you haven’t scaled your organization’s processes to effectively withstand the brunt of the unexpected, leaving it on individuals to bear.” (source)

Is your SOC such a system? If yes, how to change it?

First, where might this show up in your SOC?

  • Heroic alert triage where analysts stay late, extend their shifts, accept escalations at all hours, etc (likely the most common example, frankly)
  • Heroic rule writing where rules and content gets created, instead of a detection engineering practice you have a detection firefighting crew…
  • Heroic remediation is the classic “wait, wait, I can fix it” syndrome that, statistically speaking, very rarely leads to a good solution.
  • Another classic: working long hours to resolve an incident alone.
  • Frequently coming up with creative one-off solutions to wide-ranging systemic problems.

What do you want instead? Well, you want an industrial system! What is it? Here, Phil explains it better than I can:

source: Phil’s blog https://www.philvenables.com/post/delivering-security-at-scale-from-artisanal-to-industrial

Now, let’s see if we can quickly contextualize it for SOC

source: I just made it :-)

Notice that the heroism makes many appearances in Phil’s “artisanal” side of the table. ”Dependent on individual artisans [read: heroes] to sustain work”, “Organization success is like spinning plates, if the people don’t show up there’s immediate and catastrophic failure“, “Hard to replicate” all carry the unmistakable mark of an IT hero…

OK, gimme some good news! How to fix it?

Trigger warning: this is going to be scary.

Ready?

source: privately shared

Now for the painful, painful truth: “It’s better to let a process break and uncover a systemic issue (like the need for better tooling or an adjustment of priorities), than to have individuals try to make up for the problem.“

You want more? Sorry, all I got is this ;-) Definitely more thinking and learning is required.

Now a question: have you successfully industrialized or “de-hero-ized” your SOC? Have you used our ASO ideas? What are the lessons? Insights? Key hurdles?

Related blogs:

Monday, July 08, 2019

This Blog Is Here For Historical Interest Only!

This is a reminder/disclaimer: this blog is here for historical interest only. Its history really ended on August 1, 2011 with this post.


Naturally, you already know all this if you follow me on Twitter ...

There you have it!

Friday, August 31, 2018

Monthly Blog Round-Up – August 2018

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts based on last
month’s visitor data  (excluding other monthly or annual round-ups):
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010 – much ancient!) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our research on developing security monitoring use cases here – and that we updated for 2018. A lot more SIEM use case discussion is coming, here is a new post for 2018 SIEM use cases.
  2. Simple Log Review Checklist Released!” is often at the top of this list – this rapidly aging checklist is still a useful tool for many people. “On Free Log Management Tools” (also aged quite a bit by now) is a companion to the checklist (updated version)
  3. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009 (oh, wow, ancient history!). Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  4. Updated With Community Feedback SANS Top 7 Essential Log Reports DRAFT2” is about top log reports project of 2008-2013, I think these are still very useful in response to “what reports will give me the best insight from my logs?”
  5. Again, my classic PCI DSS Log Review series is still very popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book  – note that this series is even mentioned in some PCI Council materials.
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 7X of the traffic of this blog]: 

Critical reference posts:
Current research on SIEM:
Current research on SOC:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Other posts in this endless series:

Wednesday, August 01, 2018

Monthly Blog Round-Up – July 2018

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts based on last
month’s visitor data  (excluding other monthly or annual round-ups):
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010 – much ancient!) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our research on developing security monitoring use cases here – and that we updated for 2018. A lot more SIEM use case discussion is coming, here is a new post for 2018 SIEM use cases.
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009 (oh, wow, ancient history!). Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this rapidly aging checklist is still a useful tool for many people. “On Free Log Management Tools” (also aged quite a bit by now) is a companion to the checklist (updated version)
  4. Updated With Community Feedback SANS Top 7 Essential Log Reports DRAFT2” is about top log reports project of 2008-2013, I think these are still very useful in response to “what reports will give me the best insight from my logs?”
  5. Again, my classic PCI DSS Log Review series is still very popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book  – note that this series is even mentioned in some PCI Council materials.
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 7X of the traffic of this blog]: 

Critical reference posts:
Current research on SIEM:
Current research on SOC:
Recent research on testing security:
Recent research on threat detection “starter kit”
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Other posts in this endless series:

Friday, June 01, 2018

Monthly Blog Round-Up – May 2018

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts based on last
month’s visitor data  (excluding other monthly or annual round-ups):
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010 – much ancient!) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our research on developing security monitoring use cases here – and that we UPDATED FOR 2018.
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009 (oh, wow, ancient history!). Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this rapidly aging checklist is still a useful tool for many people. “On Free Log Management Tools” (also aged quite a bit by now) is a companion to the checklist (updated version)
  4. Updated With Community Feedback SANS Top 7 Essential Log Reports DRAFT2” is about top log reports project of 2008-2013, I think these are still very useful in response to “what reports will give me the best insight from my logs?”
  5. Again, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book  – note that this series is even mentioned in some PCI Council materials.
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 7X of the traffic of this blog]: 

Critical reference posts:
Current research on SIEM, SOC, etc

Just finished research on testing security:
Just finished research on threat detection “starter kit”
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Other posts in this endless series:

Tuesday, May 01, 2018

Monthly Blog Round-Up – April 2018

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts based on last
month’s visitor data  (excluding other monthly or annual round-ups):
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our research on developing security monitoring use cases here – and we just UPDATED IT FOR 2018.
  2. Simple Log Review Checklist Released!” is often at the top of this list – this rapidly aging checklist is still a useful tool for many people. “On Free Log Management Tools” (also aged quite a bit by now) is a companion to the checklist (updated version)
  3. Updated With Community Feedback SANS Top 7 Essential Log Reports DRAFT2” is about top log reports project of 2008-2013, I think these are still very useful in response to “what reports will give me the best insight from my logs?”
  4. Again, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book  – note that this series is even mentioned in some PCI Council materials. 
  5. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009 (oh, wow, ancient history!). Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 7X of the traffic of this blog]: 

Critical reference posts:
Current research on testing security:
Current research on threat detection “starter kit”
Just finished research on SOAR:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Other posts in this endless series:

Dr Anton Chuvakin