Friday, February 28, 2020

Chronicle Road to Detection: YARA-L Language — Part 3 of 3

Note: this blog series now exists in a cleaned-up full whitepaper form at https://hubs.ly/H0pvD_30


Note: this blog series now exists in a cleaned-up full whitepaper form at https://hubs.ly/H0pvD_30

We call our approach YARA-L because it is inspired by YARA — invented by Google VirusTotal for malware analysis and applied to logs (hence the “L”) and other security telemetry inside the Chronicle platform.

What is important is that YARA-L is a language to express detections, not merely to query the data in order to eventually use said data for detection. In essence, it is a threat detection language, not a data query language! It is designed by and for security analysts, admittedly with some help from malware reverse engineers (who created the original YARA).

Building to become “the standard” or even “a standard” for expressing detection is a monumental undertaking. Understanding the success of YARA is essential in developing a strategy for nurturing the success of our detection language, YARA-L, broadly and Chronicle rule engine specifically.

How will YARA-L be used with Chronicle [not all of the below are real today, to be sure]?

  1. Define real-time detection rules
  2. Define historical detection rules of arbitrary complexity
  3. Perform advanced (hunting-style) searches

Admittedly, the cases 2 and 3 are similar, but 2. implies a scheduled rule run leading to an alert while 3 is about interactive exploration of data by a human analyst.

YARA — L Scenarios

What are some of the scenarios where we think this approach really shines?

  1. You hear about a new malware that uses a particular registry key for persistence
  2. You analyze a new threat and realize that it uses a Powershell launching with a hidden window parameter
  3. You have a need to expand coverage of a particular MITRE ATT&CK technique in your environment
  4. You want to cast a wide net of looking for types of newly popular suspicious activities such as access to browser cookie and credential files
  5. You need to watch for specific activities that are necessary for the attacker to take in order to steal particularly valuable data

YARA-L shines in all those scenarios and while other approaches may help with some of them, it is hard to find an approach that works universally.

One, a traditional SIEM product with a UI-based rule configuration lacks flexibility and hence the coverage of use cases, coupled with typically slower performance. Two, a search-based tool will require a layer of additional data analysis to deliver the detections. Three, a programming language-based approach (say express detection rules in Python or Go as some elite organizations do) would be a massive overkill for non-programming SOC analysis and other security professionals. We feel that YARA-L is the “just right” approach for the threat detection problem today.

Detection needs continue to shift and grow as organizations realize legacy product solutions are not the sole answer to security. As investments are made into human analysts, threat responders and hunters, we have the opportunity to introduce YARA-L as both a microscope and wide angle lens to defenders’ toolkits. We must support an investigative mindset that allows for ready on-boarding and application of threat intelligence, analyst insights, and investigative techniques.

Thus far, the collective industry has failed to develop a unified solution and remains unable — in most cases and/or without extreme reliance on uniquely talented people — to effectively use procedural based detectors derived from the study of attacker TTPs. YARA-L is an opportunity to throw our hat into the ring while also providing Chronicle customers a powerful mechanism to detect threats and intuitively pursue and iterate upon security investigations.

YARA-L Example

Example MITRE ATT&CK Technique T1054 Blocking of Indicators

profile mitre_T1054_Possible_Indicator_Blocking

{

meta:

author = “Chronicle”

description = “Detection for registry change used to bypass Event Tracing for Windows (ETW)”

reference = “https://attack.mitre.org/techniques/T1054/"

version = “1.0”

created = “2019–12–13”

function:

func RegChange()

if ( (udm.metadata.event_type == “REGISTRY_MODIFICATION” or udm.metadata.event_type == “REGISTRY_CREATION” ) and

re.regex(udm.target.Registry.registry_key, “HKLM\SYSTEM\\CurrentControlSet\\Control\\WMI\\Autologger\\AUTOLOGGER_NAME.*”) )

then

return true

end

return false

end

condition:

if ( RegChange() ) then

outcome.match()

end

}

P.S. I can be convinced to drop more examples, if you are curious.

(written together with Brandon Levene)


Originally published at Medium.

Tuesday, February 25, 2020

Chronicle: One Year Later — Anton’s Reflections

RSA 2020 is a good time to reflect upon trends and evolution of our industry, and the players. This post is about our own evolution.


Chronicle: One Year Later — Anton’s Reflections

RSA 2020 is a good time to reflect upon trends and evolution of our industry, and the players. This post is about our own evolution.

Chronicle launched its security analytics platform at RSA 2019. While I was not there at launch, I feel that perhaps I can do the reflecting anyway, even if some of it would be as an outsider, and now as an insider.

I learned of Chronicle a bit before RSA 2019, when I was first briefed on the technology and introduced to the team. It was definitely a case of “love at first sight.” As I stated in my original post in June 2019, “what I see in Chronicle is an amazing technology platform, transparent business model, stellar team and huge potential for changing how we do security.” Now, as I am writing this in February 2020, much of the above is still here, though admittedly not all. However, the changes that happened since that day brought new and exciting opportunities to change the world of security in more ways than before.

So, after that initial meeting, I was observing Chronicle progress from my analyst vantage point. Let me reveal a secret here: Chronicle launch at RSA 2019 was the one for the history books. I don’t mean that as a marketing exaggeration, but in terms of the measurable impact on the industry.

Here, let me quantify this. If you’re an analyst, a new vendor launch gets reflected in your inbound client inquiry flow. In essence, do clients ask about the new company and their technology? In many cases, successful vendors appear in inbound inquiry activity within 1 to 2 years, when enough analyst firm clients hear about them and want to know if this new technology applies to their environments.

Chronicle was different. Clients’ questions flooded the inquiry inboxes within days of the launch, and the flood continued for months. Dozens of organizations were asking about Chronicle and that created a perfect storm of interest. Later on, I learned that the inbound pipeline generated by Chronicle looked like the one that a healthy 3 year-old security start-up may have.

Next, my life changed in June when I left the analyst ivory tower for the real world by joining Chronicle … and then the same month it changed again. Chronicle was acquired by Google.

Admittedly, not everybody was happy (no link, sorry!). As with every acquisition, there was a period of uncertainty and stress. Some code that should have been written perhaps wasn’t. However, a new exciting mission has emerged from this period of uncertainty!

To be sure, the new mission does not have a cool catchphrase like “Give Good The Advantage,” but it has the breadth beyond threat detection/response and the depth of Google experiences and resources behind it.

First, some great things stayed pretty much the same. Many customers and partners continue to love the Chronicle product. Note that they do so both because we save them money compared to alternatives (due to our per-user pricing model and one year data retention) and also because we help solve real problems they have today.

We are expanding to Europe, Australia and Asia this year. New partners are coming online too — if you are an MSSP or an MDR with an outdated or expensive back-end platform (and yes, even “free” can be expensive if done wrong), come use Chronicle.

We can still search any/all structured telemetry data we have in 0.25 seconds. We can still match threat indicators to massive log piles. The UI keeps getting better and new data sources are being integrated all the time.

Second, some things changed for the better. We are now part of the security unit of Google Cloud, with more fun products on the way. These products will leverage the assets only Google has — our scale, our experience, our engineering resources, our leading web browser, etc. Some of the products are already out, while others are being built as we speak.

The new unit has responsibilities for security of our cloud, security of customer resources in our cloud and — with Chronicle and other products — for security beyond our cloud.

Chronicle platform itself is getting a huge value boost due to our detection engine that runs YARA-L, a new detection language. The language is aimed at changing how we — and hopefully later others — do detection engineering and detect new threats. Our rules scale massively, run over enriched data, cover a broad range of security telemetry (beyond logs) and enable detection of advanced threats with minimal threat analysis and with no need to train any algorithm. These detection features are backed by a clear event schema that works across logs, endpoint telemetry and traffic data. More posts on YARA-L are coming this week.

Other things are not finished yet, but they are very exciting — and may or may not involve machine learning...

To conclude, the future is bright! Admittedly, it’s a different future that most of us envisioned, but it is bright. Google can and will change how organizations practice cyber security and will enable more organizations to secure themselves with the same tools and approaches Google has successfully applied to its own security...


Originally published at Medium.

Chronicle: One Year Later — Anton’s Reflections

RSA 2020 is a good time to reflect upon trends and evolution of our industry, and the players. This post is about our own evolution.


Chronicle: One Year Later — Anton’s Reflections

RSA 2020 is a good time to reflect upon trends and evolution of our industry, and the players. This post is about our own evolution.

Chronicle launched its security analytics platform at RSA 2019. While I was not there at launch, I feel that perhaps I can do the reflecting anyway, even if some of it would be as an outsider, and now as an insider.

I learned of Chronicle a bit before RSA 2019, when I was first briefed on the technology and introduced to the team. It was definitely a case of “love at first sight.” As I stated in my original post in June 2019, “what I see in Chronicle is an amazing technology platform, transparent business model, stellar team and huge potential for changing how we do security.” Now, as I am writing this in February 2020, much of the above is still here, though admittedly not all. However, the changes that happened since that day brought new and exciting opportunities to change the world of security in more ways than before.

So, after that initial meeting, I was observing Chronicle progress from my analyst vantage point. Let me reveal a secret here: Chronicle launch at RSA 2019 was the one for the history books. I don’t mean that as a marketing exaggeration, but in terms of the measurable impact on the industry.

Here, let me quantify this. If you’re an analyst, a new vendor launch gets reflected in your inbound client inquiry flow. In essence, do clients ask about the new company and their technology? In many cases, successful vendors appear in inbound inquiry activity within 1 to 2 years, when enough analyst firm clients hear about them and want to know if this new technology applies to their environments.

Chronicle was different. Clients’ questions flooded the inquiry inboxes within days of the launch, and the flood continued for months. Dozens of organizations were asking about Chronicle and that created a perfect storm of interest. Later on, I learned that the inbound pipeline generated by Chronicle looked like the one that a healthy 3 year-old security start-up may have.

Next, my life changed in June when I left the analyst ivory tower for the real world by joining Chronicle … and then the same month it changed again. Chronicle was acquired by Google.

Admittedly, not everybody was happy (no link, sorry!). As with every acquisition, there was a period of uncertainty and stress. Some code that should have been written perhaps wasn’t. However, a new exciting mission has emerged from this period of uncertainty!

To be sure, the new mission does not have a cool catchphrase like “Give Good The Advantage,” but it has the breadth beyond threat detection/response and the depth of Google experiences and resources behind it.

First, some great things stayed pretty much the same. Many customers and partners continue to love the Chronicle product. Note that they do so both because we save them money compared to alternatives (due to our per-user pricing model and one year data retention) and also because we help solve real problems they have today.

We are expanding to Europe, Australia and Asia this year. New partners are coming online too — if you are an MSSP or an MDR with an outdated or expensive back-end platform (and yes, even “free” can be expensive if done wrong), come use Chronicle.

We can still search any/all structured telemetry data we have in 0.25 seconds. We can still match threat indicators to massive log piles. The UI keeps getting better and new data sources are being integrated all the time.

Second, some things changed for the better. We are now part of the security unit of Google Cloud, with more fun products on the way. These products will leverage the assets only Google has — our scale, our experience, our engineering resources, our leading web browser, etc. Some of the products are already out, while others are being built as we speak.

The new unit has responsibilities for security of our cloud, security of customer resources in our cloud and — with Chronicle and other products — for security beyond our cloud.

Chronicle platform itself is getting a huge value boost due to our detection engine that runs YARA-L, a new detection language. The language is aimed at changing how we — and hopefully later others — do detection engineering and detect new threats. Our rules scale massively, run over enriched data, cover a broad range of security telemetry (beyond logs) and enable detection of advanced threats with minimal threat analysis and with no need to train any algorithm. These detection features are backed by a clear event schema that works across logs, endpoint telemetry and traffic data. More posts on YARA-L are coming this week.

Other things are not finished yet, but they are very exciting — and may or may not involve machine learning...

To conclude, the future is bright! Admittedly, it’s a different future that most of us envisioned, but it is bright. Google can and will change how organizations practice cyber security and will enable more organizations to secure themselves with the same tools and approaches Google has successfully applied to its own security...


Originally published at Medium.

Chronicle: One Year Later — Anton’s Reflections

RSA 2020 is a good time to reflect upon trends and evolution of our industry, and the players. This post is about our own evolution.


Chronicle: One Year Later — Anton’s Reflections

RSA 2020 is a good time to reflect upon trends and evolution of our industry, and the players. This post is about our own evolution.

Chronicle launched its security analytics platform at RSA 2019. While I was not there at launch, I feel that perhaps I can do the reflecting anyway, even if some of it would be as an outsider, and now as an insider.

I learned of Chronicle a bit before RSA 2019, when I was first briefed on the technology and introduced to the team. It was definitely a case of “love at first sight.” As I stated in my original post in June 2019, “what I see in Chronicle is an amazing technology platform, transparent business model, stellar team and huge potential for changing how we do security.” Now, as I am writing this in February 2020, much of the above is still here, though admittedly not all. However, the changes that happened since that day brought new and exciting opportunities to change the world of security in more ways than before.

So, after that initial meeting, I was observing Chronicle progress from my analyst vantage point. Let me reveal a secret here: Chronicle launch at RSA 2019 was the one for the history books. I don’t mean that as a marketing exaggeration, but in terms of the measurable impact on the industry.

Here, let me quantify this. If you’re an analyst, a new vendor launch gets reflected in your inbound client inquiry flow. In essence, do clients ask about the new company and their technology? In many cases, successful vendors appear in inbound inquiry activity within 1 to 2 years, when enough analyst firm clients hear about them and want to know if this new technology applies to their environments.

Chronicle was different. Clients’ questions flooded the inquiry inboxes within days of the launch, and the flood continued for months. Dozens of organizations were asking about Chronicle and that created a perfect storm of interest. Later on, I learned that the inbound pipeline generated by Chronicle looked like the one that a healthy 3 year-old security start-up may have.

Next, my life changed in June when I left the analyst ivory tower for the real world by joining Chronicle … and then the same month it changed again. Chronicle was acquired by Google.

Admittedly, not everybody was happy (no link, sorry!). As with every acquisition, there was a period of uncertainty and stress. Some code that should have been written perhaps wasn’t. However, a new exciting mission has emerged from this period of uncertainty!

To be sure, the new mission does not have a cool catchphrase like “Give Good The Advantage,” but it has the breadth beyond threat detection/response and the depth of Google experiences and resources behind it.

First, some great things stayed pretty much the same. Many customers and partners continue to love the Chronicle product. Note that they do so both because we save them money compared to alternatives (due to our per-user pricing model and one year data retention) and also because we help solve real problems they have today.

We are expanding to Europe, Australia and Asia this year. New partners are coming online too — if you are an MSSP or an MDR with an outdated or expensive back-end platform (and yes, even “free” can be expensive if done wrong), come use Chronicle.

We can still search any/all structured telemetry data we have in 0.25 seconds. We can still match threat indicators to massive log piles. The UI keeps getting better and new data sources are being integrated all the time.

Second, some things changed for the better. We are now part of the security unit of Google Cloud, with more fun products on the way. These products will leverage the assets only Google has — our scale, our experience, our engineering resources, our leading web browser, etc. Some of the products are already out, while others are being built as we speak.

The new unit has responsibilities for security of our cloud, security of customer resources in our cloud and — with Chronicle and other products — for security beyond our cloud.

Chronicle platform itself is getting a huge value boost due to our detection engine that runs YARA-L, a new detection language. The language is aimed at changing how we — and hopefully later others — do detection engineering and detect new threats. Our rules scale massively, run over enriched data, cover a broad range of security telemetry (beyond logs) and enable detection of advanced threats with minimal threat analysis and with no need to train any algorithm. These detection features are backed by a clear event schema that works across logs, endpoint telemetry and traffic data. More posts on YARA-L are coming this week.

Other things are not finished yet, but they are very exciting — and may or may not involve machine learning...

To conclude, the future is bright! Admittedly, it’s a different future that most of us envisioned, but it is bright. Google can and will change how organizations practice cyber security and will enable more organizations to secure themselves with the same tools and approaches Google has successfully applied to its own security...


Originally published at Medium.

Friday, February 21, 2020

Thanks a lot for this comment, BTW.


Thanks a lot for this comment, BTW. Still, I think we [here we stands for “you and me” :-)] kinda stopped a bit short of per asset / per network dimension of coverage. We did do control types, but not what assets/systems/data the controls cover, right?


Originally published at Medium.

Thursday, February 20, 2020

Detection Coverage and Detection-in-Depth

For some time, I’ve been also fascinated with the concept of detection-in- depth and a somewhat related concept of optimal detection…


For some time, I’ve been also fascinated with the concept of detection-in- depth and a somewhat related concept of optimal detection coverage.

This fascination was born out of a particular type of analyst inquiry I used to get: if I have SIEM, do I also need an EDR? If I have SIEM with sysmon and Zeek data, do I also need an NTA tool? If I have anti-malware tool with an EDR module and a separate NTA tool, do I also need a SIEM?

These all map to a problem of detection layers and detection coverage, hence “detection in depth.” While many people ramble about “defense in depth, ” I feel that much fewer actually implement and practice layers of detection controls in their environments. As often happens in security, talkers talk … while some doers don’t do — they just buy more tools and then have them sit unused :-)

You can also look at this as a logical evolution of defense in depth. The whole premise was that you were preventing things with layers of controls. Now, the premise is you should be able to see things from various angles and layers. Things missed by some layers will be caught by other layers. This should be true even if — especially if — the attacker actively countered some layers of visibility (e.g. turned off logging, avoiding writing to disk, provided their own network link, etc). Apart from this, observing the same activity from different layers may lead to additional security insights (security fusion FTW!), even if no layer is compromised.

As an analyst, I was involved in several attempts to bring clarity to this question, starting from the 2014 “attack chain” paper (still a decent read, BTW) and then in our famous paper on starting your detection and response efforts (and perhaps here too). Even my SOC nuclear triad idea (which recently resurfaced as Anton’s security visibility triad) is pointing in the same direction, if vaguely. It basically says that you should collect and analyze logs, network data and endpoint data — but this advice is way too generic for this discussion.

So, what about a more operational guidance focused on specific systems and environments? This is still not easily available anywhere …

So, basically this is about TWO closely-related things:

  1. What telemetry you need for good detection in general?
  2. What telemetry you need for good detection on each asset given the telemetry you already collect elsewhere?

The above-linked documents (some are behind the paywall — sorry, the analysts need to eat) seek to answer 1. (and IMHO do a decent job there), but none really attempt to answer 2. with any depth (note that I am casually avoiding the question “detection of what?” by punting this to an old threat assessment discussion).

As extra credit, what other useful question did I just punt? Data quality / fidelity / consistency. If you have a chance to, say, get firewalls logs, netflow or full pcap of the same network link, which would you choose? Windows authentication events logs or EDR agent captures of authentication activity? This would also be a topic for another day.

Before I continue, I want to point out that most organizations don’t start with “detection in depth” (they also likely didn’t start with “defense in depth”, either). They usually start with one approach, picked based on all sorts of non-security considerations.

So, let’s unroll the main question:

1. For many organizations, the starting state is often about choosing ONE detection technology (it may be even whatever your managed provider has) and deploying it to cover a subset of assets:

2. Next comes the expected “???” — many expect AI magic here, naturally. But I’d prefer some useful answers. Or, at least useful questions.

3. And then some organizations arrive at a glamorous end-state where you have comprehensive visibility over systems, networks, applications, data, etc; this likely involves all of the above (logs + network + endpoint), but also coverage over mobile devices, cloud, serverless, IoT, <insert the list of cool new technologies then remove blockchain from the list :-)>

And now here is my next question: how do we plan for a reasonably effective state 2. marked by a mysterious “???” above? Or, how to do we do better than a random mix of detection technologies BUT without reaching a security detection panopticon nirvana of “all security telemetry, all the time.” Similarly, how do we avoid detection gaps that really matter?

Next, let’s dispatch with some popular wrong solutions: a popular EDR vendor once tried to convince me that “because their EDR is oh-so-sooooooo good, you really just need an EDR, and you are ‘all covered’.” Essentially, their view was that EDR is the final answer to our detection and response challenges, an answers for the states 1, 2 and 3 above. Now, I love the endpoint visibility provided by a good EDR tool, but routers? IoT devices? Personal PCs and mobile devices? EDR and SIEM / logs work well together, representing two (of the three) sides of visibility.

Similarly, “just capture traffic” because “traffic is the truth” is equally silly. Again, I love network visibility, but SSL / TLS? Highly distributed networks? Attacker-provided “rogue” GSM devices? Cloud environments with no traffic capture? The pattern is obvious: ONE LAYER is not the answer. Because it won’t be detection in depth then, will it?

Now, lets get back on track: what are some of the ideas for solving this:

  1. Good old killchain teaches us to try to try to capture telemetry down the likely attacker path (e.g from client PCs to DMZ to cloud environments) and wherever a stage of a killchain may happen. Killchain makes logical sense when considering operations from an attacker point of view. But mapping it to a defender’s capabilities and visibility layers is not that straightforward.
  2. Ever-more-popular ATT&CK framework further gives you a list of “stuff” that you then need to “cover” by one or more source of telemetry (admittedly, very skewed to endpoint data sources i.e. a place where the attacker ends up). However, you then need to work to decide on the layers to deploy on your own.
  3. Ad hoc: perhaps I want to get 2–3 chances to detect every threat in my environment (the specific threat list would come from some threat assessment thinking). What layers do I need for that?

At this point, I feel that this post will be more of “an incomplete thought” post, and just like an elephant in an old Russian joke, it will end suddenly… But yes, this is what I have so far. Thoughts?

P.S. My next couple of blog posts will be about some new stuff we are releasing, BTW…


Originally published at Medium.

Tuesday, February 04, 2020

Dr Anton Chuvakin