Threat Detection

Home » Threat Detection

The Biggest Companies in the World Are Failing at Basic Detection

in Incident Response, Malware, PacketSled, Threat Detection by Matt Harrigan Comments are off
We received a really interesting email from our upstream Internet provider yesterday – AT&T, indicating that we were effectively infected with Tinba malware. For those of you who don’t remember it, Tinba is 4 year old Windows specific Malware based on the old Blackhole exploit kit that does all kinds of fun stuff like steal facebook credentials, attempt to access your online banking accounts, etc. The email was as follows: att-incident Technical and operational observations are as follows:
  • First, we don’t possess any Windows machines capable of executing this Malware, with the exception of some VMs that are turned off unless otherwise needed.
  • Second, all of those machines have adequate endpoint protection on them, such that they would have averted this ancient attack, should it have been, you know, actually happening.
Regarding the email we received, if you haven’t already found the funny part, I’d like to point that out first. Obfuscating the IP address of the site that (according to them) indicates an infection is extraordinarily unhelpful, almost to the point where I feel like they might be trying to live up to some weird obtuse legacy of what it was like to order circuits in the 1990s. Thankfully, they took the time to include the domain name, which, you know…resolves to the obfuscated IP address listed directly above it. The unfortunate part about this situation is that they simply didn’t have the proper tools to make a quick determination that this was in fact, not a threat at all, and that we were running some tests on this network against legacy threat intel. Of course, we knew that, and we could easily validate that such testing was occurring during the timeframe that they mention with a simple query in the investigator as follows:
Querying specific IPs during specific timeframes using natural language

Querying specific IPs during specific timeframes using natural language

We get one session record returned:
PacketSled Investigator

PacketSled Investigator

And immediately determine that this “threat” originated from a Mac running Wget:
Hey look, not a Windows machine. Not even a real browser.

Hey look, not a Windows machine. Not even a real browser.

Nevertheless, we received this email, meaning that someone in the AT&T SOC had to investigate this non-incident, put in a ticket, send the email and manage the eventual closure of that ticket. This scenario is extremely commonplace. The number of SOC personnel, incident responders, and general infosec professionals that routinely chase non-incidents due to only having partial data is extremely perplexing. The data is in the packets. Making it accessible to analysts and automated processes such that it can be intelligently used is the goal. With something this simple getting incorrectly bubbled up to the top, we are clearly a long way off from that goal.

An Analysis of SPAM Across the Entire Organization in 5 Minutes

in PacketSled, Threat Detection by Matt Harrigan Comments are off
One of the neat things we get to do at PacketSled is work with our customers to address a very large variety of problems that don’t always immediately manifest themselves as obvious security incidents. This last week we had a customer experiencing an unusually large volume of email spam, some of which had malware payloads, some of which had adware payloads, and some of which was, well, just spam. The customer’s question was very interesting: “Do any of the spam senders have anything in common?” To answer this question, we turn to a core feature of PacketSled’s natural language search called “clustering.” Clustering enables a security analyst or incident responder to see the relationship between normally otherwise disparate data that traverses their network. Because PacketSled creates a semantic record of every conversation on the network by extracting all the meaningful attributes from its observed flows, we can establish the relationship between two or more otherwise seemingly disparate activities on any network. For example, clustering filenames on destination ip addresses will show us the nature of the files that our users are downloading, irrespective of protocol (HTTP, FTP, dropbox, etc). This feature is immensely helpful during breach analysis and incident response to determine the “who, what, when, where, how” in an incident. In this case, we’re trying to define why the spam filter isn’t working, and what the resolution should be. The obvious places to look for commonalities are in the characteristics of the email itself. First, we issue the search “cluster subject on [receiver_email]” and we return a list of all email subjects, followed by a count of those subjects. Clicking on that result shows us the recipients that received any email with that subject in it. redacted-cluster-on-no-geo (note: the recipients and subjects of legitimate emails have been redacted) As we examine the data, we start to notice that there seems to be a heavy correlation between any email that originates outside of the recipient’s country (in this case, non-US traffic), and the content of the emails. To validate this, we simply add the modifiers src_geo != lo (for local), and src_geo != US (to eliminate any emails that originated from the recipient’s geo, and we add the country of origin to the cluster query (src_geo). redacted-cluster-on-src_ip This is where the results get -very interesting-. There is not a single email in the entirety of 24 hours that originates outside of the customer’s native geography which is a legitimate email. Subjects range from telepathy classes to blue pill sales. A simple right click on the IP address in the visualization, and we can lookup the originator of over 90% of this traffic: Screen Shot 2015-02-18 at 3.13.55 PM Seems that LeaseWeb’s customers are our primary offenders. In this case, our customer had a legacy rule deployed in their message gateway (that is managed by a third party) to allow anything from that class A, irrespective of content. Nobody at the organization is quite sure why, but they are sure that this would have taken several days, if not weeks to hunt down, and exposed most of their users to large amounts of unnecessary risk. PacketSled enabled this customer to resolve this issue in record time, keeping the customer safe and productive.

Detecting ShellShock Using PacketSled

in Articles, PacketSled, Threat Detection by Matt Harrigan Comments are off
Knock Knock. Bash Bash.  The scope and scale of the bash attack is fairly well known even though its only been public for 24 hours, but in case you’ve missed out on it somehow, here’s a quick recap of the CVE summary that will provide some context:
GNU Bash through 4.3 processes trailing strings after function definitions in the values of environment variables, which allows remote attackers to execute arbitrary code via a crafted environment, as demonstrated by vectors involving the ForceCommand feature in OpenSSH sshd, the mod_cgi and mod_cgid modules in the Apache HTTP Server, scripts executed by unspecified DHCP clients, and other situations in which setting the environment occurs across a privilege boundary from Bash execution.
Most of the early exploit activity has leveraged CGI to exploit shell commands. Exploit POCs for other protocols will follow for sure, and we’ll be posting about detections for those in real-time. For the time being, we can make one broad assertion, which is that the payload of the attack will exist in a relatively obvious place, as it did with the HTTP proof that’s out. In order to properly detect and investigate sessions that leverage this vulnerability in your environment, the first step is to isolate the various attributes and their values (HTTP headers in this example). The good news –  If you already have PacketSled, you have all of this data preloaded, not only for HTTP, but for telnet, DHCP, and 2000 other protocols. As Troy Hunt mentioned in his post today, there will likely be additional iterations of the vulnerability. Automatic detection of ShellShock atttempts on a running system: PacketSled behavioral detections were updated for all customers today with a level 10 alert that will fire for ShellShock exploit attempts. The severity of this attack will likely go down over time, as systems become immune or at least improve their resistance to the problem. If you see this detection on your dashboard,
ShellShock Detection in Dashboard

ShellShock Detection in Overview

then the good news is that you already have the detection built in, and any incidents that have been identified are made available for investigation simply by clicking them. If you do not see this security incident, and want to manually identify any sessions that may be exhibiting shellshock behavior, you can also use the natural language search engine in investigator to look through all of your network traffic, with one easy query. For instance:
today http user_agent like /\(\)\s{\s.*;/
 
ShellShock-New

ShellShock Investigation in PacketSled

This same query can be run against any attribute in an http flow. Other common places to look include:
  • The User-Agent HTTP header (user_agent)
  • Cookie (cookie)
  • Host (server)
  • Referrer (referrer)
  Finding your highest risk hosts: Now that we’ve identified attempts to exploit this vulnerability, the next logical step is to determine how many hosts we have that are actively executing code out of /cgi-bin. Simply use the investigator to search through all the web transactions that happened in your production environment:
today 10.1.1.0/24 http user_agent like /\(\)\s{\s.*;/ uri like /cgi-bin 
This search will identify hosts with active applications that run out of cgi-bin, who are currently targets of the attack. Total elapsed time: 30 seconds. 

The Challenges of Threat Detection in a Changing Landscape

in Articles, Big Data Security Analytics, PacketSled, Threat Detection by Matt Harrigan Comments are off
Having spent 20 years being exposed to the real world security problems of customers as a trusted advisor, I’ve had the unique privilege of observing the common challenges that large financial, retail, defense, and critical infrastructure organizations face when trying to address complex, evolving threats. I’ve also watched the security community go from a niche, perhaps misunderstood group of ex-hackers trying to solve the problems that they discover in the soft underbelly of the early Internet to building an entire multi-billion dollar industry who’s goal it is to address the problems that plagues every employee, officer, director, and consumer. Companies have spent millions of dollars on security products to protect themselves and their customer’s data. However, there are still holes, and so-called advanced threats are getting through the cracks. What is needed, once and for all, is a solution that does a better job of detecting advanced threats, as well as providing a means for containing, analyzing and remediating unknown threats that have gotten through traditional defenses. Today’s security practitioners, engineers, and operating teams are currently using SIEM and/or full packet network forensics. Let’s talk about these options. Screen Shot 2013-12-17 at 6.15.33 PMOption 1- SIEM Tools – The idea here is that millions and millions of events being generated by hundreds or thousands of devices are cataloged in a central repository for correlation and processing. Let’s consider this for a minute; In order for technologies that rely on log data to correlate events, three things have to happen: 1. The devices that matter in the context of a security event have to be pointing log data at the SIEM. Most organizations typically prioritize this by starting with their security infrastructure (FW, IPS/IDS, Endpoint). This requirement to prioritize the output of these technologies seems to ignore the fact that the attack vector may have been designed specifically to avoid or elude them, and that the entry point for advanced threats may be somewhere else entirely. 2. Accuracy of the logs you require relies on integrity of the endpoints generating these logs – Attackers compromise IPS/FW technologies routinely, pivoting from internally compromised hosts into the admin interface of these technologies. At that point, its fairly trivial to create or disable rules that enable them to add a pathway for tunneled traffic that goes undetected by said devices, and certainly isn’t generating any useful log data that might be caught by the SIEM. 3. Alerts setup correctly – So what do you do with all the data you now have in one pane of glass? (if you’re even able to do that). Most analysts will tell you that the bulk of alerts they currently look at are not valuable, and that a very small percentage of the data coming across the console pertains to real security threats. Getting meaningful event correlation out of standalone SIEM using noise reduction techniques is difficult to do correctly, and even some of the best security organizations struggle with it.   full packetOption 2- Full Packet Solutions– Traditional full packet technologies offer the promise of much deeper insight into the activities of attackers than traditional log-only solutions, but they exhibit some fundamental problems that preclude them from operating in a manner that is consistent with anything other than “after the fact” scenarios. Areas of concern include: 1. Historical forensic record length – If a user issues a SQL query that results in a dump of thousands of records, and the network forensics solution has visibility into that session, guess what – you’re storing those thousands of records now. If you place a sensor on the ingress/egress points of your network, and someone watches a 40 minute youtube video on bonsai gardening, you get to keep that too. Neither of these things are valuable in the context of a security incident, but nevertheless, they’re quickly eating up expensive storage on your network. While your SOC team may learn some new root trimming techniques, they aren’t getting security value from the data. The fundamental challenge here is that storage requirements exceed the need to retain the data. One saturated 10Gbps link can produce up to 108TB of stored data per day, or 3.2 petabytes per month. It’s no mystery why these tools aren’t being deployed enterprise-wide. The incidents are buried in forests of tiny trees. 2. Fast, intelligent access to the data – Full IP packet data is by definition binary. Other than augmentation of the data with descriptors, full packet solutions have no way of answering user queries or machine generated analytical inquires in a reasonable timeframe that is acceptable for a typical incident response plan. In short – pulling back full packet is slow and ineffective in a real-time solution. It also makes it impossible to distribute the solution over very large environments, because sending the same data to a remote central repository would absolutely destroy even the most resilient networks. 3. Cost of retention –78% of all breaches are being discovered months, or sometimes even years after they’ve occurred. The data is already gone, and the damage is already done. Buying more disks may sound like a cheap option, but ask anyone in charge of buying these solutions, and you will quickly be corrected to understand that an enterprise-wide deployment of full packet capture simply isn’t an option financially. At PacketSled, we care deeply about the issues that plague today’s technology-enabled companies. We’re delivering solutions to these very difficult problems by building a next generation of products that can produce answers about attacker behavior, assets that are or may be impacted, and fastest path to threat eradication / remediation. We do it at scale, in real-time, for the entire enterprise, and in the cloud, at a cost that is commensurate with the security budgets of our customers. I’m very excited to be here, and I look forward to working with you. Matt Harrigan President & CEO PacketSled

© 2016 PacketSled, Inc.