Please believe me that I was meaning to write this post for my old blog, somewhere in March. It is a total coincidence that I work for…
Please believe me that I was meaning to write this post for my old blog, somewhere in March. It is a total coincidence that I work for Google GCP / Chronicle now that happens to be really good at this very task…
So you recall my recent post about TI matching to security telemetry like logs in near real-time? I did say that most threat intelligence (TI, also called “threat data” in this post) comes from past observations of badness. In fact, the whole model of value for threat intelligence is that even though such badness is past for the initial observer, it is a likely future for the intel recipient / consumer.
Hence it is more useful to match TI to past observations in your environment — such as your logs and other telemetry collected over time and not only after the threat data arrives at your door. Naturally, it is even better to match TI to both current/future and past telemetry.
For example, if the initial badness (say, an activity by a particular threat actor) was observed in June 2019, resulting intel was delivered to a client in August 2019 and this is when matching vs their logs started, how do you detect a compromise by the same actor that happened in July 2019? Exactly — with TI retro-matching! The word “retro” here does not mean the 1990s, BTW :-)
See my crude visual that proves, hands-down, that I should stick to writing text:
In this post I wanted to talk about specifically matching TI to past telemetry. To rind, “past” here is defined as “events that occurred before this threat data landed in your lap.”
Now, in real life, most organizations do real — time matching (i.e. they match logs as they arrive to the TI they already have) and do not do any historical matching. Most vendor products from literally firewalls to antivirus as well as “advanced” security tools make real-time matching as easy as flipping one switch or setting a checkbox to ON.
This starts the process of checking the incoming logs versus the existing repository of threat data. You can immediately see that hilarity ensues when you try to match 1,000,000 indicators (the question why the hell you have so many seemingly relevant threat data points is left for the reader to ponder) to 100,000 events per second.
Now, logs arrive as the activities are observed, and threat data feeds arrive on a schedule set by the TI provider(s) or your TIP configuration. So there is no universal law that says matching needs to only look forward — and in fact, that is counter to a large part of TI’s value discussed above.
The historical matching problem is both easier and harder. It is easier since you face little pressure to match in the now. And, obviously, it is harder since you need to match vs larger pool of data and you need to do it repeatedly after new threat data arrives.
However, the question many have is: is this actually that important? Let me convince you that you want to explore threat intel retro-matching:
Much wider net cast for signs of badness in your environment
Increased the likelihood of actually seeing observables in their original context and relevant time windows
Improved chance to uncover advanced threats as better intelligence on their behavior becomes available
Ok, and why should some take a pass?
Very obviously, historical matches are not real time detections, since the threat was logged a while ago and not moments ago. If you are uncomfortable with that, well, take a pass.
You probably need a separate system with different performance and scalability requirements — if you cannot afford one, you should not attempt it
If using poor quality intel, the number of false patches may overwhelm you
You have no process (and no desire to create one) for dealing with persistent compromise at all; for you, a historical compromise is just not actionable.
Operationally, how often do you need to match TI beyond merely the logs that arrive after the threat signal? Personally, I want it done every time new intelligence arrives or soon thereafter. Practically, even a daily scan won’t be a bad thing since you may discover evidence of an intrusion that happened weeks ago.
How far back to dig? I want to go back a year, just for the heck of it (and because a year is a good retention number). It is very hard to justify, say, 30 days vs 90 days. On the other hand, you want to try and match this up to “average dwell time” (nowadays this number hangs in the 100–200 day range depending who you ask). Hence, you probably won’t go wrong with a year. However, keep in mind that environments do change and sometimes investigating an incident that happened a year ago brings up many different challenges (this discussion is left for future posts, if you care).
Now, this again raises the same old question: are TI matches finished/cooked detections or merely hunting clues. Personally, I’d treat all historical TI matches as hints or hunting clues, not as cooked detections or wake-me-at-3AM alerts. After a match, you still need to do the work to unravel the entire incident scope…
We are trying a per user license, per host is how it all started and EPS was see as an improvement. We like per user as more predictable. And we can make it work for us.
Eventually, we’d post it on a corporate blog, somewhere.
====== START GUEST POST =====
Introduction
All too often, we read sensational headlines that emphasize the identity of an attacker. Frequently, these same headlines parallel an organization’s CIO/CSO/Leadership first reactions when they themselves become aware of an intrusion or a breach: Who stole our data? Where did they come from? This mindset, the idea that “who” is the most important component to understanding a sequence of events, is not only dangerous, but distracts from the myriad of problems that require timely examination.
Let’s discuss why the first impulse to answer “who” when triaging and responding to incidents, is not the correct path (in most cases). We will begin by exploring the false premise that the “who” equates to the “how”. Then, we will proceed to discussing digestible, actionable intelligence and its role in informing defenders. We then will address- perhaps- the most critical issue facing every organization: resource allocation. Finally, we will explore the circumstances in which attribution can genuinely make a difference in your overall security posture.
Who != How
Attackers are constantly evolving to defeat the latest and greatest #product. As an organization, understanding what attacks may look like and how they can affect your business is of foremost import to continued success.
I believe the best way to explore the imbalance between “who” and “how” is to explore multiple realistic scenarios that enterprises deal with daily:
Business Email Compromise
Let’s first consider the following scenario: the CFO of your organization has fallen victim to a well crafted request to transfer funds to a third party aka: Business Email Compromise (BEC). Will answering “who phished us” address any of the following issues::
1: Provide any sort of information to better protect and defend against similar attacks
2: Assist with the recovery of funds, trade secrets, or other company property
No. The identity of your attacker doesn’t help you mitigate impact. The technical details (where the money was transferred to, accounts used) communicated in a timely fashion are essential to recovering funds. Guidance from the FBI indicates the most important consideration is rapid response to events. Frittering away time on attribution instead of analyzing the attack itself is a surefire way to lose your cash. When time is of the essence, opt for technical analysis: focus on the details!
To learn from the event above, the most important consideration is “how do we prevent this from happening again”. Falling victim to the same attacks multiple times is embarrassing and damaging to a company’s brand — over the time erosion of trust is more than just loss of equity up front. : This is ultimately indicative of a fundamental failure in the security apparatus to appropriately assess, analyze, and mitigate risks.
Ransomware
[A.C. — now, you may think ‘hey, ransomware generally attributes itself because you are contacted by its operator’ … wait … wait … not so simple!]
2019 has been a banner year for opportunistic ransomware deployments. Rather than the “spray and pray” malspam mechanisms of yesteryear (2016–2018), semi-manual deployment to soft victims of opportunity has become the case du jour. This so-called “big-game hunting” is often facilitated by remote access protocols such as Remote Desktop Protocols (RDP). Externally accessible RDP is effectively rolling out the red carpet to opportunistic intruders.
Does your organization care who specifically deploys these techniques; which “Spider” group deploys which malware?
No.
Addressing this risk from a perspective of “how does X facilitate y”, in this case: how do externally accessible management protocols enable ransomware deployment, is a far more critical consideration for defenders to undertake.
The hallmark of a successful attacker is a combination of opportunity and patience. Focusing analytic power on tangible issues such as visibility rather than attribution, pays dividends when it comes time to respond to an incident. The problem, then, is how do we, as enterprise defenders, allocate our time. We’ll discuss this in the following section.
Resource Allocation
Hardly a week goes by without a headline lamenting the “[Cyber]Security Talent Shortage” and while opinions seem to differ as to the cause and solutions to this issue, the net result is fairly simple: blue teams are chronically undermanned, underfunded, and under supported. This leads teams to use additional technologies to compensate (vendor X will solve problem Y), which may further accrue technical and knowledge debt. The end result? Resources are finite.
Time spent emphasizing attribution is time not spent on proactive introspection of your enterprise’s assets. The majority of enterprises do not have enough well-instrumented visibility to utilize even basic atomic indicators, aka threat data. Not enough thought or time is given to learning from an institution’s prior intrusions and compromises:
Stop allowing the same mistakes to happen by using previous defensive failures to understand the “what” and “how” fully.
Acknowledge that being “popped” twice (or more) in the same way is a failure.
Combine the above with proactive visibility into the entirety of the organization’s footprint to allow for rapid response and remediation: visibility is more important than prevention.
To emphasize the point, we conducted a survey of information security executives from February to June of 2018 in which we inquired, “Do you believe you have enough visibility into your organization’s technical assets to identify potential compromises?” The answer will probably not shock any practitioners or leaders.
Informal survey by Chronicle
The results of this question speak for themselves. You can’t protect what you can’t see.
Time is the most expensive (and limited) asset of a security team. Cycles spent on attribution concerns do not yield readily consumable results. Iterate on visibility, ask questions of your networks and endpoints, and learn from previous mistakes and architectural quirks.
Threat hunting is informed by attribution insofar as it is useful for clustering (applied labels): but good threat hunters will understand and distill the fundamental, actionable intelligence and use this data to proactively identify potential compromise. To threat hunters, “who” is just another label that helps organize TTPs. This leads us right into actionable intelligence.
Digestible, Actionable Intelligence
Operationalization of intelligence is derived from contextual application of indicators. Attribution is a useful label or classification to define a common language when sharing IOCs and TTPs. Mapping behaviors to responses and mitigations is more important than mapping rote identities; a critical idea to consider to this end: This set of behavior A is bad, here is how we handle A (consider the Ransomware example).
Most organizations do a poor job of using the massive wealth of threat data available to them. They do not use their own environments to validate threat data, but rather rely on the sources of said data to dictate prevalence. A mentor of mine once said, “Trust, but verify.” [A.C. — it is also a well-known Russian proverb…] This holds especially true for cyber intelligence.
Robust overall defensive posture and proactive threat assessment via hunting contribute more value than attacker identification. Identification should be considered a post mortem activity, a component of an “after action report” or lessons learned phase.
Attribution Guides Response
In mature organizations with defined defensive practices, well constructed threat hunt and intelligence teams, and adequate visibility attribution can be valuable:
Attribution as a component of understanding and assessing potential adversaries can allow organizations to prioritize their defensive efforts.
Proactive defensive planning can be facilitated by OSINT research into threat actor groups relevant to an organization or from repeated, related intrusion attempts. Evaluating mitigations and controls in the face of “real” TTPs is essential.
Understanding related sets of behavior can help shortcut response, especially in the face of repeated, related intrusions. Example: Alerts for stage 3 tool from adversary X should indicate to your team that there is a gap in visibility or a change in behavior.
Labels, distinctive groupings of adversary behaviors, allow for more streamlined threat sharing among peers; enabling collective, community security. This is especially prevalent among businesses in similar verticals.
Understanding threat actor motivations when your organization is concerned can help illuminate potential visibility gaps.
Approaching investigations from a perspective of “how would threat actor X do this” is a useful construct when approaching threat hunting.
Attribution makes a difference in organizations that are already well equipped to reliably and effectively answer questions of “How” and “What”. Adding “Who” generates additional context. Prioritizing “How” and “What” is essential to get to this stage of organizational defensive maturity.
To that end, Daniel Clemens (of Shadowdragon) shared the following with me while I was researching this topic:
“We have seen many of our more mature customers take on the role of long term attribution on the actors that are top threats to their business, enabling a large reduction in fraud as well as a catalyst for meaningful intelligence sharing that organically grows within specific industries”
The emphasis on operational maturity cannot be overstated. Achieving the outcomes highlighted above is enabled by defined, iterative blue team practices that are effectively prioritized, manned, and funded.
Closing Thoughts
Basic cyber hygiene is a frequent point of emphasis when considering the state of information security. Organizations must closely examine their overall security programs to accurately and honestly assess the state of their maturity. Overreaching when the basics aren’t covered results in a critical mis-allocation of resources that can result in catastrophic, but preventable failures. Attribution isn’t a silver bullet; it is a tool in the adaptive arsenal of [advanced] defenders. As in mathematics, order of operations matters, get a handle on the what and how before pursuing questions of who.
Acknowledgements
I would like to thank Pat Litke who was instrumental in helping to outline and brainstorm this idea (and who would’ve been my co-speaker if this topic had been accepted to the conference it was submitted to).
I would also like to thank Anton Chuvakin and my peers at Chronicle for assisting with edits and ideas.
If you want to use this data, a citation would be cool!
Additional Reading
The following research was useful to me when considering this problem set. I encourage those interested in this topic to begin with some of the articles below:
