“The cloud compromise factors and outcomes observed in Q2 2023 were largely similar to previous quarters and consistent with the last 12 months of reporting. […] weak credentials continue to represent the largest compromise factor where many observed instances were a result of attackers brute forcing default accounts, Secure Shell (SSH), and the Remote Desktop Protocol (RDP)” [A.C. — as usual, shocking but not surprising. Perhaps the surprise is that it is NOT changing over 2–3 years of ‘clouding’…]
“In the Q2 2022 Threat Horizons Report, we highlight that a disproportionate percentage of attackers opportunistically use coin mining across Cloud products and alter their tactics to evade discovery. This is consistent with this quarter’s findings, as this is the most observed outcome from compromises.” [A.C. -another ‘resilient’ finding, most cloud attackers just cryptomine]
“This quarter our teams observed a 8.5% increase in vulnerable software compromises led primarily by PostgreSQL being the most exploited.” [A.C. — an interesting choice, perhaps some of the instances got ransomed too? Also, I sense this is related to credentials above…]
“SaaS providers were also targeted earlier in the year by suspected financially-motivated DPRK actors in order to gain access to downstream victims.” [A.C. — this is interesting, SaaS as a stepping stone! Is SaaS — likely SaaS credentials, frankly, your weakest link?]
Finally, here is some interesting data focused on healthcare cloud compromises
Let’s review three organizational models of integrating an existing threat intelligence (CTI) team with the detection engineering function for optimum detection work.
Operating Model 1: CTI Feeds SOC / Detection Engineering
Some organizations have a clearly defined and separate CTI team, which supplies information to different teams, functions and recipients. Detection engineering (DE), whether inside or outside the SOC (here, specifically this point may not matter), is just one of the recipients.
When there are organizational reasons why this setup cannot be radically changed, you should define a concrete interface: requirements, expectations, procedures for threat sharing (including rush or priority ones), artifacts sharing, cadence and some other common processes.
CTI Feeds SOC / DE
Ultimately, this is the setup that is most affected by all the classic “silo” problems, “us vs them”, finger pointing, cross-blaming, etc. It can work, and it has worked for people, but it sure isn’t the best.
Operating Model 2: CTI Feeds Mini-CTI Inside SOC / DE
A more effective approach is to embed a CTI analyst(s) into the Detection Engineering team. Working shoulder-to-shoulder allows shorter turnaround times and a better understanding of processes on both ends, while keeping the CTI team separate and able to serve other stakeholders.
CTI Feeds Mini-CTI Inside DE
In other regards, this is similar to model 1, but it likely will work better for most organizations (well, most orgs mature enough to understand all this stuff and who have the actual SOC, TI, etc teams)
Operating Model 3 : CTI — DE Peer Team Model
An even better way is what some call a Cyber Fusion Center (because, clearly, we don’t have enough “cyber” in our lives and we miss the cold fusion idea…), which has been a term recently popularized by MSSPs/MDRs and some threat intelligence vendors. To us, this is a model of peer teams that work together for common mission.
CTI — DE-IR Peer Team Model
This model restructures the CTI, SOC, Hunting (sometimes also a special team) and CSIRT / IR functions in a single cross functional organization, with defined, but highly collaborative functions serving shared goals and targets. It allows teams to build complex workflows directly, with a single voice, without the trouble of interfacing with separate teams owning different backlogs and agendas. Further, ideally these teams needs to be fluid and rotate people thus obliterating siloes (a TI guy may do a bit of DE, an IR could do TH, etc.)
Naturally, if you are building a shiny new SOC, we’d recommend looking into integrating this fusion center collaborative concept in your architecture: breaking down (better, not creating) silos is always easier to do when planning, than with existing organizations.
Better CTI for Fun and Profit … and Detection
As we discussed, threat intelligence is key to Detection Engineering, however as we have seen its input is sometimes vague, hard to parse, or just insufficient. Frankly, we’ve seen cases where the CTI team is genuinely at a loss when dealing with DE teams, as it doesn’t always understand what is expected.
DE teams expect from CTI not reports (especially not overstuffed 27 page PDF reports ), presentations or news articles, but handing over a concrete, well-described knowledge item (like a wiki page) that they can easily understand and that enables kicking off R&D easier. Ideally, it should be tagged to make the overall knowledge base easy to search, and tracked as an issue in a project management system to build a relevant backlog of threats to detect. These last requirements also make metrics and reporting on detection quality much easier.
Delivering those prioritized, technical threat knowledge items should be a core service of CTI teams focused on helping the DE team. Other functions like IOC compilations, high-level reports and threat landscape analysis are still helpful, but will not deliver what detection engineers need to succeed in their mission.
From the DE perspective, the best intel input is highly cohesive (focuses on describing a particular adversary behavior) and loosely coupled (isn’t so high level that it combines multiple classes of attack in one behavior). This can be seen as the best raw materials to cook tasty finished detections.
Some key markers of helpful intel that will lead to better detection engineering :
Focused on described a single threat, in a particular domain
Specific to technologies, protocols, OS, device type.
Threat impact is evaluated and so the threat can be prioritized into a rolling backlog
Technical, procedure level information without losing track of overarching TTPs and commonalities across variants.
Displays some unique characteristics which are helpful to isolate invariable behaviors
Threat descriptions that at least hints at detection opportunities
To make it even better, such information should:
Show relevance to organization IT estate and crown jewels
Doesn’t assume deep threat actor knowledge in DE teamand explains concepts in clear English
Not a PDF or email, but a knowledge base item; doesn’t need a second parsing by a DE into more usable knowledge items
Delivered at set frequency (frequent but not overwhelming intel flow means that the team has reached the correct level of granularity and cadence)
In this way, when the DE and CTI are able to work together in a common flow, it improves development and delivery of new detections (both in time-to-ship and quality), evaluation of the actual detection coverage, and boosts the teams competence in understanding cyber threats. It also builds team morale, as there is now a single direction and single prioritized backlog instead of divergent perspective in what detections to build first, and how to develop them best (this is kinda a big deal!).
You should now be very familiar with the challenges that detection engineers face with the threat intel they receive, and be able to take away some key improvement points to improve cohesion in your particular organization structure. Or, if you don’t have a TI/CTI function at all, keys to build one!
In our next blog post we’ll explore the other side of the coin: assuming you get nice intelligence, how to break it into detections?
This post is our first installment in the “Threats into Detections — The DNA of Detection Engineering” series, where we explore opportunities and shortcomings in the brand new world of Detection Engineering.
Detection Engineering Defined
As many of you already know, detection engineering is the process of building, refining and managing detection content (rules, content, code, logic — whichever word is most suitable for you). It is a relatively new discipline, but it is rapidly gaining importance as the threat landscape becomes increasingly complex and (for top-tier threat actors) more targeted to each environment.
It rewrites the classic SOC-building handbook by putting an emphasis on detection quality from the start, and dedicates capacity directly to the content engineering concern (as we say multiple times in our Autonomic Security Operations materials). Detection engineers also work closely with other security teams, such as threat intelligence and incident response, to ensure that their detections are developed quickly and work well.
It is a challenging field that requires a deep understanding of both security and software engineering. On the security side, detection engineers need to be able to identify and understand the latest threats and attack techniques. They also need to be able to develop and maintain detection rules and signatures that can accurately identify these threats. On the software engineering side, detection engineers need to be able to develop and maintain detection systems that are scalable, reliable, and efficient. They also need to be able to tune and optimize these systems to reduce false positives and negatives.
There are many broad challenges that detection engineers face, including:
Messy threat landscape: The tactics and tools that attackers use are constantly changing, making it difficult to keep up with the threats. Understanding attack chains performed by threat actors is complex work that needs to be performed both quickly and reliably (false negatives and/or false positives can ruin the program). Naturally, attackers also have an interest in not being detected, at least in some cases.
The need for speed: Detection systems need to be able to identify threats quickly in order to minimize the damage they cause — new detection content needs to be rolled out ASAP once threat intelligence is received. All this should work in the context of increasing volumes of telemetry data, without incurring the engineer burnout.
The complexity of data and systems: Detection systems need to be able to process large amounts of data from a variety of sources, including network traffic, cloud services and endpoint data. When logs are not parsed properly, or doesn’t contain required data, detections lean on the poorer side: garbage in, garbage out. Reliable and fast detection engineering for IT “layer cake” from mainframes to containers, present in many environments, is not getting easier. The age of “I just need to know a few popular Windows event IDs to detect” is long over.
With capable threat actors mastering both LOL and custom malware, and with the realization that a lean, reproducible, and efficient workflow is required to go from threat to detections, we have seen more signs of the evolution from generic security analysts who operate on canned or lightly tuned detection content to more dedicated roles like threat hunters and detection engineers. We are also seeing further signs of the common L1/L2/L3 concept progressively dissolving, which moves the role of the detection engineer to the front of the detection battle.
Things As They Stand
At many companies, Detection Engineers started to be differentiated as a separate role 2–3 years ago (and, yes, we have seen organizations that had similar roles for a decade or more), and while they’re now in demand, they are still somewhat of a rare breed.
In some organizations, an L3 analyst tends to be designated to do the same activities as detection engineers, but also have the double (triple if they also hunt) duty to perform investigations, or hunt threats without always plugging into new detections. This is less optimized than having a fully dedicated capacity, but is indicative of the push toward conscious detection content development (it is perhaps better for an overburdened L3 to do this compared to … nobody, at least as a transition stage)
On the other hand, threat intelligence teams (outside of SOC) have had a hard time scaling to detection engineering needs. This leaves detection engineers often having to do the bulk of the threat research required to build detection backlog items and often even having to select intelligence sources themselves. Ideally, this process would be collaborative, to allow a continuous and smooth handover of actionable CTI data to detection engineering teams.
In the next part, we will cover scaling the intelligence to detection content pipeline.
This post is our second installment in the “Threats into Detections — The DNA of Detection Engineering” series, where we explore the challenges of detection engineering in more detail — and where threat intelligence plays (and where some hope appears … but you need to wait for Part 3 for this!)
Contrary to what some may think, a detection and response (D&R) success is more about the processes and people than about the SIEM. As one of the authors used to say during his tenure at Gartner, “SOC is first a team, then a process and finally a technology stack.” (and just repeated this at mWISE 2023) And here is another: “A great team with an average SIEM will run circles around the average team with a great SIEM”
SIEMs, or whatever equivalent term you may prefer (A security data lake perhaps? But please no XDR… we are civilized people here), are essentially large scale telemetry analysis engines, running detection content over data stores and streams of data. The signals they produce are often voluminous without on-site tuning and context, and won’t bring value in isolation and without the necessary process stack.
It is the complex cyber defenders’ knowledge injected at every step of the rule creation and alert (and then incident) response process that is the real value-add of a SOC capability. Note that some of the rules/content may be created by the tool vendor while the rest is created by the customer.
So, yes, process is very important here, yet under the shiny new name of TDIR (Threat Detection and Incident Response), lies essentially a creaky process stack riddles by inefficiencies and toil:
Inconsistent internal documentation — and this is putting it generously, enough SOC teams run on tribal knowledge, and even an internal Wiki would be a huge improvement for them
Staggered and chaotic project management — SOC project management that is hard to understand and improve, that doesn’t happen smoothly, with release/delivery process that is completely irregular and traceability is often lost midway through the operational noise.
No blueprint to do things consistently — before we talk automation, let’s talk consistency. And this is hard with ad hoc process that is reinvented every time…
No automation to do things consistently and quickly — once the process is clear, how do we automate it? The answer often is “well, we don’t”, anyhow see the item just above…
Long onboarding of new log sources — while the 1990s are over, the organizations where a SOC needs to shove paper forms deep inside some beastly IT organizations to enable a new log source have not vanished yet.
Low awareness of removed or failed log sources — SOCs with low awareness of removed or failed log sources are at risk of missing critical security events and failed — worse, quietly failed — detections.
Large inertia to develop new detection content, low agility — if you make an annual process into quarterly, but what you need is a daily response, have you actually improved things?
Inscrutable and unmaintainable detection content — if the detection was not developed in a structured and meaningful way, then both alert triage and further refinement of detection code will ..ahem … suffer (this wins the Understatement of the Year award)
Technical bias, starting from available data rather than threats — this is sadly very common at less-mature SOCs. “What data do we collect?” tends to predate “what do we actually want to do?” despite “output-driven SIEM” concept having been invented before 2012 (to be honest, I stole the idea from a Vigilant consultant back in 2012).
While IT around your SOC may live in the “future” world of SRE, DevOps, GitOps and large scale automation, releasing new detections to the live environment is, surprisingly, often heavy on humans, full of toil and friction.
Not only is it often lacking sophistication (copy pasting from a sheet into a GUI), but it is also not tracked or versioned in many cases — which makes ongoing improvement challenging at best.
Some teams made good progress toward automation by using detection as-code, but adoption is still minimal. And apart from a handful of truly leading teams, it is often limited to deploying vendor-provided rules or code from public repositories (ahem, “detection as code written by strangers on the internet”, if you’d like…). As a result, it then poses a real challenge of reconciling internal and external rule tracking.
An astute reader will also point out that the very process of “machining” raw threat signals into polished production detections is very artisanal in most cases; but don’t despair, we will address this in the next parts of this series! This would be fun!
Apart from that, much of the process of creating new detections has two key problems:
Often it starts from available data, and not from relevant threats.
Prioritization is still very much a gut feeling affair based on assumption, individual perspective and analysis bias.
Instead, there should be a rolling evaluation of relevant and incoming threats, crossed with current capabilities. In other words, measuring detection coverage (how well we detect in our environment against the overall known threat landscape) which allows us to build a rolling backlog of threats to detect, identify logging / telemetry gaps and key improvement points to steer detection content development. This will turn an arts and crafts detection project into an industrial detection pipeline.
💸 How about relying on vendors ?
What about avoiding all the above trickery, and relying on a wise third party for all your detection content? Well, not only does external detection content quality vary drastically from provider to provider, such dependency can occasionally become counter-productive.
In theory, the MSSPs / MDRs with a pristine ecosystem to research, build and release should have solid detections for most clients conventional threats, but the good ones are far and few between. They often cannot spend their development time to create custom detections (“economies of scale” argument that is often used to justify an MSSP actually prevents that). Instead, some build broad, generic detection logic, and then spend their — and customer! — time on tuning out False Positives on the live environments of their clients.
From the end-client perspective, there is neither a guarantee nor complete understanding that the process is going measurably well (especially, and I insist here, in regards to False Negatives) and that it actually leads to increase in client detection coverage. In this regard, some would say that MSSPs / MDRs with regard to detections and detection coverage compete in a market of lemons .
For internal or co-managed SOCs (where a small internal team works with a MSSP), relying excessively on externally sourced rules also has the occasional side-effect of lowering the actual understanding of both the threat and the detection implementation, further encouraging a downward spiral.
When working the other way around, handing over in-house detections to the provider for alert response, there are often slowness and protests as it interferes in their processes and they are (legitimately) concerned that their analysts won’t understand how to process the incidents since they lack on-site context and tribal knowledge. This plays a real part in the industry where a 70% False Positive rate is rather common, by over-relying on response capacity to tune out noisy rules (or, a beautiful SOAR at the output of an ugly SIEM, now a classic!), rather than having a defined development lifecycle where lowering FPs is a priority.
With all this being said, integrating vendor-made (either ready-made rules that come with SIEM tools or outsourcing their implementation) into creating detections is perfectly viable. However, this is true only as long as you have a strong grasp of the end-to-end process, and understand technical objectives very well. Only then will the external rules fit in your environment without adding burden…
In this blog (#3 in the series), we will start to define and refine our detection engineering machinery to avoid the problems covered in Parts 1 and 2.
Adopting detection engineering practices should have a roadmap and eventually become a program, effectively re-balancing where efforts go in a SOC by investing in high quality detection creation (and detection content lifecycle, of course).
Put simply, if you spend more time building better detections, then you spend less time triaging bad alerts. Simple, eh? If it were simple everybody would do it!
Embracing leaner, consistent, purpose-driven detection workflows is key, and you may want to assert where you land on these key areas:
⚒️ Breakdown and Backlog: Build a continuous roll of issues corresponding to threats to analyze, and detection requirements to implement. What you are doing next for detection content should be clear in most cases, and yes, this is security, so there will be nasty surprises. Eventually, the only unpredictable tasks would be the genuine rare surprises — your routine detection work would not surprise you.
🌊 Smoothen yer workflow : Remove people interfaces that don’t work, define minimal ceremonies, and put content reviews in the right places. Shorten approval times for releases, ensure detection quality reviews are followed up. If dealing with a MSSP/MDR, make sure to lay the governance structure for building custom content jointly with you (JointOps, not finger-pointing).
☣️ Embrace threat-driven approach: Study adversary tradecraft in detail before making educated calls on what to detect, and where/how. Starting from available telemetry data will more often than not be prone to bias, inefficiency and mistakes.
💡 Embed Intel: A CTI enclave in a SOC will often provide higher ROI than dealing with a separate team, as they understanding SOC needs better, and plug in directly into their processes (this is full of nuance, so YMMV)
⚡ Lower Intel-to-Rule KPIs : Quantify how long it takes to go from intel input (ItR for Intel to rule, ha-ha, we just made up a new acronym! Take that, Gartner! ;-)) to an actual detection; with as much granularity as possible. An good ItR metric would be to transform a high risk threat intel into working detections in hours or in a few days.
👀 Visibility over assumptions : If you can not answer accurately within minutes what your detection coverage is and which shortcomings it has — you likely need to start parsing your detection library and threat modeling into qualitative metrics and make that data transparent to the DE team.
🚀 Release Soon, Release Often! : There are new threat variants every week, and detection engineering scales directly on the quality of the intelligence input. This is where modern software engineering practices come handy.
🔥 Quantify, Measure, Orient Operations : Define what healthy operations look like : what FP rate is acceptable for new detections ? What turnover time for tuning should be aimed at ? Where are quick wins, where are detection gaps ? Where is capacity spent, should it be reassigned to more urgent priorities ? Where are process bottlenecks ?
🦾 Automate the hard — and boring — part : Everything produced during R&D should generate rich and structured knowledge bases, metadata, and metrics. Version your detection library, and roll it out with CI/CD toolings. BTW, this advice alone is worth the price of this blog!
💎 Where does ATT&CK fit in the DE picture ?
While a great model to categorize adversary tradecraft, and a necessary tool in the detection engineering arsenal, ATT&CK is often overused as a palliative measure to map detections and generate coverage maps — by skipping the detail (“Do you cover T1548? — Ehhh… YES?”)
This does not accurately represent SOC detection performance, since techniques can be fairly broad (since it is the purpose of the taxonomy to normalize specificities), and rule quantity doesn’t equate quality (not when they drown analysts in False Positives).
Detection hints from ATT&CK are also rather generic, since a Technique is itself a concept which clusters different procedures together. Thus, while ATT&CK can give a direction of what a SOC needs to develop, it doesn’t give a way to achieve detection objectives; which is the detection engineer core concern.
The journey to Detection Engineering maturity is hard, but you should now have a clearer perspective to smoothen the journey toward building better detections.
But it all starts with quality input : in our next blog post, we’ll look at in more detail what exactly a Detection Engineering team needs from Threat Intelligence to be fully informed, and propose collaborative models. Stay tuned!
