As we learned, SIEM still matters in 2023.
As we learned, SIEM still matters in 2023.
So, we went through “Debating SIEM in 2023, Part 1”, now let’s debate a bit more. At this point, everybody who didn’t…medium.com
But since one winter day in 2002, when I wrote my first correlation rule for a now-defunct “SIM” product (probably “if 10 auth_failures, followed by 1 auth_success on the same destination, alert” or perhaps “exploit followed by outbound connection from the same system, alert” , but I truly don’t remember which one was first), I have been bothered with a question of what I am actually doing when I am creating SIEM content — while sitting on the vendor side.
- Am I writing a piece of detection code that will run in the customer environment to detect threats that matter to them? Put another way, am I in the prod code business?
- Or am I writing a piece of example code that customers can use to learn how to write good rules, perhaps by modifying it and changing to fit their needs? Put another way, am I in the education business?
(please “the-why-not-both crowd”, be patient for a moment…)
Let’s try to understand this conundrum. We will use a peculiar instrument for understanding this: FALSE POSITIVES (FP).
Again, there are two schools of thought here.
One school assumes that an SIEM vendor (or, frankly, any other detection-related vendor) ships a product — a platform plus content — and the content included in the box is an essential part of a finished product. These detections should produce useful alerts in many or most customer networks. Thus, FP = a product bug.
Another school of thought indicates that the vendor ships a platform and some sample code that customers are supposed to modify or learn from. These detections should teach clients to detect. Thus, FP = a reminder that content needs to be adapted by the client to work for said client.
Notice here that the attitudes towards FPs sort of mimic the attitudes towards detection content? And this we can test for, at least using surveys. For example:
(source)
(source)
One can hypothesize and attribute the higher “we will tune” response from Twitter due to a more technical audience there. Averaged over both, let’s say ~30% think that “false positive” is a bug, while ~55% think they need to tune. Note also that changing the question to false positives reduces the ability to vote for “why not both” hence producing, hopefully, cleaner data on the problem.
This is good! To be honest, I did not have any set expectations for what the outcome should be apart from a vague opinion that a “false positive” from using unchanged, canned SIEM content is often not a bug.
Now, some of you came and wisely said “It depends.” Because of course it does!
Specifically, some would say it depends on the nature of detection content. Perhaps two types of content should exist. Type 1 should just work, and Type 2 should be used as a template. You probably won’t modify the detection rule that just matches a known malicious IP address (what, put another IP in there?) or a hostname (just like few would want to change anti-malware signatures). EDR, funny enough, is also in the “it depends” land, yet I see a lot fewer people modifying EDR content compared to SIEM content.
Or, they would say it depends on the nature of a ‘false positive’. Is this a truly broken detection logic (like a typo in a regex) or a nuanced environment problem? “Report the former and tune the latter” seems like sensible approach.
Or, they would say it depends on what you mean by “tuning”: limit the scope of rule applicability, add conditions (light tuning) vs changing the rule logic (heavy tuning)? Perhaps we need a separate discussion on this as “just tune it” is occasionally used as a throwaway comment by people…
Yet another group would bring up MDR and say that the attitudes for MSSP/MDR (service) are dramatically different from SIEM (product). In my mind, you will absolutely report an FP as a bug to an MDR, because ultimately this is why you pay them. Now, would the answer change for a regular software SIEM vs a modern SaaS SIEM? It is a lot easier to report an FP to a SaaS SIEM and the vendor in this case has a much better chance of helping you because they have all of the data (so “it depends on a SIEM”, perhaps?)
Now for the even more fun part!
To me, this SIEM content and false positives debate is a micro instantiation of a much bigger debate: the paradox between consuming security and engineering security.
For example, our podcast episode 117 revealed that “engineering-led” approach to security is superior for scaling and other reasons, yet clearly there are people who don’t want to engineer security, they just want to be secure. They essentially want to consume security, rather than engineer it.
There’s a large and some would say growing group of people who actually believe that security should become more like software engineering and that more people should be engineering security rather than consuming it. These are likely the people with robust detection engineering programs who may not even rely on vendor-made content that much… so this debate may feel silly for them.
This came out a bit all over the place, so conclusions?
- It depends :-) Some “false positives” really are bugs in your SIEM while others are “natural” and unavoidable, so you report the former and tune the latter.
- The degree of what constitutes “tuning” is not always clear and additional discussion is needed here.
- If you buy a SIEM today, you will need to tune / refine / adjust detection rules and likely create your own. No way around it.
- Still, most do need to push your SIEM vendors to delivering more and better detection content out of the box (duh!).
Related posts:
- “Detection as Code? No, Detection as COOKING!”
- “How to Measure Threat Detection Quality for an Organization?”
- “Detection Coverage and Detection-in-Depth”
- “Data Driven Detection Engineering” by Julius Vehent
- “Detection in Depth” by SpecterOps
- “A Research-Driven process applied to Threat Detection Engineering Inputs” by Alex Teixeira
Originally published at Medium.
