One of the bigger headaches I think we can all agree on in the Cyber Security business is the overuse of buzzwords, and the overlapping mutations of what they mean, depending on who’s saying them. Threat Hunting has certainly become one of those phrases. So what is threat hunting, really? Well, depending on who you ask… (i.e., an Enterprise, a Vendor, or perhaps both), that answer can certainly vary. And the interesting thing is none of them may actually be wrong, despite having somewhat different answers to the same question. There is certainly a wide collection of different tools, skill sets, approaches, and processes to help identify things currently happening or that could happen within the network. What is an effective hunting process for one organization may be a waste of time for another, depending on each company’s understanding of what threats they might face. Man-hours spent hunting are typically most beneficial for large organizations, that may be targeted by the cybercriminal community on a regular basis, but that’s not to say that regular hunts for small/medium-sized enterprises can’t benefit from and identify threats by doing the same.
So let’s start with what is (at least somewhat of) a traditional definition of threat hunting. Threat Hunting can be defined as a focused and intensive human/machine-assisted process aimed to identify the possibility of something malicious happening within the network or likely about to happen; this is based on abnormal network behavior, artifacts, or identification via active threat research. A good example of this would be:
- A large bank has team members in which part of their job is to consume threat reports related to activity targeting their vertical and other companies that match their Enterprise profile. > A new threat report is published from an intel provider describing a new variant of malware that has been catastrophic at similar organizations. This report would ideally contain information around the process tree, registry key, etc. to help the threat hunters not just hunt for detection of the associated IOCs but dig deeper to identify patterns that match the behavior of the malware across the network, like abnormal PowerShell execution or account behavior. Essentially, one could generate an assumption that if other similar banks have already seen this, “we” are either currently being affected or are about to be and go on the hunt to confirm or deny that hypothesis.
But that’s just one basic example. The “possibly malicious” aspect mentioned above can come from a variety of hunting methods, which again can all be “correct” in their own right, depending on the threats, skills, and tools specific to an enterprise. These methods can be driven by the following:
- Intelligence-led hunting (IOC, actor, TTP), as mentioned above
- Hunting for specific unpatched vulnerabilities and if they have been exploited
- Hunting based on abnormal account activity
- Hunting based on abnormal machine behavior (i.e., deviations from the network baseline)
- Identifying imminent threats based on keyword monitoring of Deep Web and Dark Web forum activity
- Hunting based on endpoint activity collected by Sysmon or EDR agents (or both)
- Additional note: Considering that associated malware hashes, IPs, and domains can change often, the “IOC” type of information provided by Sysmon and EDR agents such as file paths, registry keys, and parent/child processes become particularly useful. The caveat here is that given the size of this data, long-term retention can be a problem. For a great example involving Petya, check out this blog post here.
One thing that is interesting about this topic is that when discussing it, you may hear something along the lines of “Threat hunting is not detection/remediation, and detection/remediation is not threat hunting.” This is 100% accurate, but the two processes certainly rub shoulders and compliment each other to a certain degree. An example of this could be a high-priority detection from a known critical asset to a known APT IP (which could be provided by a third party, open-source, etc.). Now, of course, this a detection and not part of a Threat Hunt, but if that one asset is communicating to that IP, what else might that APT have been up to? This is where a Hunter could pivot into an Actor profile, and gather/extract all additional information about their behaviors and additional infrastructure, such as TTP’s, IOC’s, etc., to execute a retrospective search within their SIEM/Endpoint tools to identify: a) what other machines might have been mapping to these behaviors; b) what other machines may have direct identification (associated IOC’s); c) what known critical assets were involved; and d) how long this activity has been occurring in the network.
The unfortunate thing about threat intelligence is that it is massive; in order to realistically operationalize it within the network, we have to make sacrifices based on contextual information, like how recent it is, the perceived severity by vendor A vs. vendor B, its associations, et cetera, et cetera. While we all want finished intel to seamlessly flow through our network, this is rarely the case. While you may see a piece of this APT activity that launches your investigation from that initial IP connection, chances are you don’t have all of the associated information for that APT being actively correlated. After your proactive hunting/gathering of “new” associated information to that APT, this can then be actively leveraged to find new threats or defend proactively against what would be the next phase of an APT’s attempts to maintain persistence. Unlike a lot of other tools/processes that tout automation as a central selling point to its usefulness, Threat Hunting is a different approach that requires manual effort to find the things that traditional tools are missing, and that’s something that we just have to live with (for now). As with any successful hunt, this, of course, will be dependent on the Threat Hunter’s skills and access to the right tools.
The above use case is one of the reasons Anomali developed “Anomali Match,” a dedicated appliance that is built to take a massive threat intelligence dataset (including threat reports, TTPs, and actor information) and correlate this against a compressed set of network activity to provide both retrospective search capability and lower-level intelligence correlation that may not be relevant until scenarios like the above arise.
Regardless of my definition of threat hunting, what do I need to be effective?
A lot of folks have a lot of opinions on this subject, but regardless of what your definition of threat hunting is, an enterprise needs to have the right components in place for it to be a possibility.
- Identify the right people with the right knowledge. Hunting is a resource-intensive process. The team executing the hunt should have an intimate knowledge of the network configuration, the endpoint/SIEM tool interfaces, and the user access policy and, perhaps more than anything, a deep knowledge of the prevailing operating systems in use. Most organizations do not have the resources for a dedicated hunt team on a permanent basis, so your hunters will likely be wearing a couple of hats. The folks that know the native processes installed on an organization’s “gold image” endpoint deployment is a good place to start.
- Dedicate time for the team to hunt. As mentioned before, this process is a human-driven process and will never net any results if there isn’t any time dedicated to it. Set aside daily or weekly hunt hours blocked off on your skilled hunters’ calendars.
- Know your critical assets. Sometimes it’s surprising to hear that the security teams of large organizations don’t actually know where the “crown jewels” are located, but it’s not an uncommon occurrence. If your team doesn’t have this mapped out, take the time to identify the MAC addresses, hostnames, etc., of the machines that hold the employee, customer, web application, and intellectual property data. This means that Threat Hunters can easily key on hunting against assets as part of their schedule.
- Invest in the right tools – The team needs a beach that holds hidden treasures in order for the metal detector to have a purpose. Without proper implementation of SIEM (or SIEM-like solution), Endpoint, and Threat Intelligence, it's impossible for them to find any treasure in an empty room.
- Invest in third-party intelligence – Know the threats specific to your vertical and to your brand by investing in ISAC’s, Deep/Dark web providers where applicable, and other intel vendors with a means to collect/curate/integrate large datasets of threat intelligence, like Anomali ThreatStream.
- Collect internal intelligence – Document all information and artifacts collected from successful hunts so that you don’t have to hunt the same thing twice from proper implementation of defensive measures, or at the very least, make it a lot easier to identify and remediate the second time around.
- Know your vulnerability landscape – Know what CVE’s are currently unpatched in your network; know their associated severity level; know what is actively being exploited by the actors that target you so that the Hunt team has yet another logical launching point. The “knowing” part here can come from manual analyst research or third party solutions like vulnerability scanners and threat intel companies.
- Have a clear plan before beginning. Don’t get caught up with trying to block/quarantine or setup correlation on each suspicious entity as you find them, develop a clear hypothesis of what could be happening, and tailor your hunting activities to confirm that hypothesis.
In conclusion, our security tools are never going to alert you 100% of the issues lurking in the network, which is why the term Threat Hunting exists in the first place. Attackers depend on you to rely on your signature-based detection, so every organization needs to dedicate skills, tools, and time to the hunting process to be a truly end-to-end security organization. Each organization’s process may be slightly (or widely) different from another, and that doesn’t mean it’s “wrong,” so long as that process is laser-focused on hunting for threats that were otherwise not seen or known about by traditional tools.
If you'd like to check out more on Threat Hunting, download our whitepapers: The Gamer Theory of Threat Hunting and SANS 2018 Threat Hunting Survey Results.