Incident Response

Home » Incident Response

Hunting for DoublePulsar in your networks

in Incident Response, Network Visibility, Security Research, Threat Detection by Mike Spohn Leave a comment
The recent release of the Equation Group’s (NSA) FuzzBunch software by the Shadow Brokers has caused quite a stir in the security community. The volume of files released, (6,547 in one of the dumps), is an extraordinary collection of malicious software including many zero-day exploits.

One of the binaries that caught my interest was DoublePulsar. This is the main tool used by the Equation Group to compromise Windows hosts using a SMB and RDP zero-day exploit.

Although the attack surface is complicated, my fellow researchers at did a highly competent job of describing it. To me, the most interesting step in the attack is the patching of the function dispatch table of the device driver Srv.sys in memory. Slot 0x20 (14) in this table originally pointed to the SrvTransactionNotImplemented() dispatch function. It is hijacked by the malware.

Why is this interesting? Because even though the implementation of this attack is quite brilliant, it is trivial to identify this attack in your network.

I refer you to page 426 of (Microsoft’s Common Internet File System (CIFS) Protocol) protocol document: TRANS2_SESSION_SETUP (0x000E)
“This Transaction2 subcommand was introduced in the NT LAN Manager dialect. This subcommand is reserved but not implemented. Clients SHOULD NOT send requests using this command code. Servers receiving requests with this command code SHOULD return STATUS_NOT_IMPLEMENTED (ERRDOS/ERRbadfunc).”
The CIFS/SMB TRANS2_SESSION_SETUP subcommand was never implemented by Microsoft. The standard states that any call to the command by a Windows client should return a STATUS_NOT_IMPLEMENTED reply. Once DoublePulsar redirects the Srv.sys SrvTransactionNotImplemented[] function pointer to its own code injected in memory, any SMB call to a NOT_IMPLEMENTED SMB subcommand will end up calling the DoublePulsar code.

Knowing this, is it possible to identify DoublePulsar in your network by simply looking for SMB requests for NON_IMPLEMENTED subcommands, or even simpler, any SMB STATUS_NOT_IMPLEMENTED response? Yes.

To illustrate this, look at the WireShark screenshot below that shows a SMB call to the TRANS2_SESSION_SETUP (0x000E) subcommmand.

Figure-1: TRANS2_SESSION_SETUP (0x0E) Request

Read more

Analyzing a Spear Phishing Email

in Incident Response by Mike Spohn Leave a comment
About every week or so I receive one of those obvious Phishing Emails telling me a package was not deliverable or some such foolishness. After being very careful not to click on the attachment, I typically permanently delete these Emails. When I got another one of these Emails a few days ago, my curiosity got the best of me, so I decided to figure out how the cyber-punks build these social engineering attacks, and how they work.

I documented my analysis in a Research Paper, “Analyzing a Spear Phishing Email.” You can download the report here.
The findings of my research are summarized below:

1. The appearance of the Phishing Email is very primitive, alleging the postal service could not deliver a package. The from Email address was completely unrelated to the signature line. (k(at) karastel,ru, Eugene Lee.

2. The weaponized payload was a JavaScript file that has “.doc.” in its name, embedded in two zip files. This means the recipient has to open two zip files and click on the JavaScript file for the bad guys to win.

3. The JavaScript file uses Microsoft’s ActiveX framework to create an Object to connect to the Internet and download a malicious dropper JavaScript dropper file. This script is run using an Eval() statement. The script also connects to the Internet and downloads a malicious Window PE file.

4. The miscreants compromise legitimate web sites to host the malicious binaries. The scripts contain multiple dowload URL’s to protect against detection of compromised servers.

5. The Malicious PE file is a Cerber Ransomware binary that encrypts files in the logged on user’s Documents folder, and any attached USB devices.

6. There is a sophisticated web site on the DarkNet that instructs a victim how to obtain BitCoins and pay the ransom.

7. The cyber-punks who send out these Emails bank on economies of scale. If they send out 1 million weaponized Emails, if they have a 5% hit rate – that is 50,000 victims.

8. The ransomware problem is not going away anytime soon. In fact, evidence suggests the cyber-crimnals are getting more aggressive in their tactics. Not only are they encrypting files, they are also wiping out the master boot record (MBR) on compromised systems preventing them from booting.

We are continually adding enhancements to the PacketSled Platform to identify advanced ransomeware compromise techniques.

Packetsled UDP and TCP fallback analyzers

in Incident Response by Leo Linsky Comments are off
There are hundreds of protocols that we need to see in detail to have a clear picture of our customers’ networks, and we have developed a suite of proprietary analyzers to address these. There are hundreds — or even thousands — more relevant protocols and services which we need to identify, but which are not significant enough on their own to warrant their own custom protocol analyzers. Added together, however, these protocols become major obstacles to network visibility and security.

Interestingly enough, this problem is not unique to networking — it’s a statistical phenomenon with numerous socioeconomic implications. For example, the disease advocacy group Global Genes has estimated that more than 300 million people worldwide are living with one of the 7,000 diseases they define as rare in the United States. That’s almost 5% of the world population living with a “rare” disease. And yet, treatment for each is wildly different, so it’s less productive and less lucrative to investigate each of these rare diseases.

Unfortunately, there isn’t a universal ‘rare disease treatment,’ but at Packetsled, we *have* developed a ‘universal treatment’ for unidentified UDP/TCP protocols and services. For flows in which we haven’t associated a high level of metadata, we capture key excerpts from the exchange to allow for analysis by Bro scripts or our platform, which, if interesting, can be used to pull the rest of the flow from our network data recorder.

The entries we find are frequently plaintext, or provide protocol preambles that make their nature obvious, such as the potential WinRM vulnerabilities discussed here:

Here’s an example of a UDP log showing a JSON protocol header (note that excerpt sizes are variable, and we could capture the entire JSON object here):
#fields ts      uid     id.orig_h       id.orig_p       id.resp_h       id.resp_p       excerpt excerpt_size    payload_size
#types  time    string  addr    port    addr    port    string  count   count
1486600189.506264       589bb7fd0000000000000003    17500 17500   {"host_int": 59724991715298692108921997512624305849, "version":         64      365
This is especially useful for IoT protocols, many of which we have fleshed out into more detailed, dedicated analyzers and many of which are facilitated in healthcare. Recent announcements from leading healthcare device manufacturers indicate that IoT is now being used with fetal monitors, electrocardiograms, glucose monitors, and tracking vital health information. We’ve even heard of pressure sensors being used to determine the ratio of empty hospital beds during disasters, such as floods and fires.

However, the thing to remember is IoT is still in a sort of infancy. Some established standards have been developed, but are not widely adopted. This includes communication protocols and methods of properly handling sensitive data. As personally identifiable data could be transferred using protocols such as HL7, it is highly recommended that a close eye be kept on these valuable data streams.

An example shown below:
OBR|1|341856649^HNAM_ORDERID|000002006326002362|648088^Basic                            Metabolic Panel|||20061122151600|||||||||1620^Hooker^Robert^L||||||20061122154733|||F|||||||||||20061122140000|
OBX|1|NM|GLU^Glucose Lvl|59|mg/dL|65-99^65^99|L|||F|||20061122154733|
As IoT breaks away from traditional data networks, and ways of engaging patients, organizations will find value in how PacketSled dissects these streams of data. This will aid in understanding the risk the organization is accepting with the adoption and implementation of IoT initiatives.

IoT, like Cloud, is extremely disruptive and often noisy and misunderstood. This makes it even tougher to be a Security Professional and unfortunately, executives and board members generally care very little about that. We do.

With the PacketSled platform, it is our goal to perform the following:
  • Reduce the efforts associated with non-contextual, traditional logs by applying Machine Learning and well-researched attack models.
  • Detect multi-staged attacks within ordinary-looking data flows, regardless of protocol type. As most advanced attackers aren’t facilitating 0-days, you need modeling and adaptive baselines that can spot attacks in YOUR network. “One Size Fits All” doesn’t really fit anyone, any longer.
  • Enable you to perform Incident Response at scale. Finding the piece of hay in the haystack is a tough grind and we’re here to help, with event correlation and applied threat data.
It’s time for us to break out of the same problematically applied daily grind, where you are attacked, then respond, then mop up the disaster, then you start all over again.

It’s time to talk to PacketSled.

Co-written by Leo Linsky and Patrick Kelley

Lowering The Poverty Line Of Incident Response

in Incident Response, Security Research by Patrick Kelley Comments are off

Over the years I’ve been part of monumental projects using several forms of technology, including SIEMs, in attempts to offset the talent gaps that plague the Information Security industry and shorten the “dwell” timeframe of attacks.  Taking a moment to define the anticipated goals and define the problem space, I find it necessary to state that properly launching a SIEM and similar technologies isn’t trivial and gaining a meaningful return on the investment is a significant challenge. Personally, I determine that return on investment to be a reduction in MTTD (Mean Time To Detection) and MTTR (Mean Time To Remediate).

The ultimate goal is revealing actionable intelligence that takes into consideration the true risk posed to the business and weights the alerts in that light.  This task cannot be achieved by leveraging and applying “threat intelligence” feeds as the primary form of correlation. Instead, it requires developing context around the core assets, networks, and layers of security controls that comprise the network.

In and of itself, this isn’t an easy problem to solve as businesses and their networks develop organically, as opposed to the more traditional networks created in the past.  This matters as demand for an urgent business needs drives the need for additional computing resources, leaving security as an afterthought.  This makes building contextual solutions around security problems a near impossible feat for most organizations.

Coincidentally, this is where SIEM implementations often fail. The concept of the SIEM is sound, but most implementations make decisions based on single, atomic events that are universally weighted.  This could be the match of an IP address or similar. This fails as security incidents are not singular, but more often contextual and long-playing.  Often I’ve been asked, “If we had signatures for Pass-The-Hash attack binaries and we used UAC, why were we breached? We had controls!”. 

Simply stated. This is a significant problem to solve.

The tough truth is the evolution of our networks and the way organizations communicate accelerate faster that the limited amount of resources the organization has to apply. So, just hire more professionals, right?

Here’s the harsh truth. Developing a proper Security Operations Center takes well over a year on average, and that’s just to nail down the basics.  Once developed, you will need a minimum of 8 well-trained engineers and a Sr. Security Architect to maintain the ability to respond 24/7/365, with ever-evolving breach detections. 

So, what’s the solution?  

In fairness, there isn’t a single one-shot answer for this.  However, I can offer the following advice that can greatly improve the security posture of an organization. 

Maintain a clear understanding that a threat is not just the presence of events, but also the absence of events. A phishing email with a malicious binary is certainly interesting, but so is the failed synchronization of an HA or failure of vMotion Snapshots.

Acknowledging that breaches can remain active for over 100 days, a platform that collects artifacts at the lowest level of ground truth and hold that evidence for hundreds of days, is becoming a requirement.

Threat Intelligence feeds are not the glue that binds together a security platform.  They are simply atomic indicators that can be applied to network flows. That’s not to say there’s no value in these sources of data, but a need to correlate those indicators with other sources.  Additionally, more Threat Feeds isn’t the answer.  It is far more valuable to choose the proper feeds for your environment and prevent as much overlap as possible.

Leveraging platforms, such as PacketSled, that apply multiple sources of enrichment to wire-level data, can provide greater confidence in incident relevance and the reduction of false-positives. For example, a Threat Intelligence match for an Apache attack methodology likely doesn’t represent a true threat when leveraged against a Windows environment.

We need to rely less on signatures and leaning more on User and Entity-based behavioral analytics. This isn’t an easily applied solution, but one that establishes dynamic baselines on observed behaviors with entities, allowing it to determine anomalies.  Anomalies could be determined as user accounts conducting database queries after midnight, which could not symbolize a threat, but an indicator worth additional investigation.

A means of properly prioritizing cases in a meaningful and actionable way. This is a tough problem, but one that will maximize the limited resources an organization has available. 

What’s the overall message in this?  Security is hard. It’s less the process of efficiently identifying the probable state of all enterprise platforms and users, but more so with identifying and understanding the applied risk of all possible operational states of systems and people. 


Reach out today and learn what we are doing to help you close the gaps and lower the poverty line in Incident Response.

Modeling Multistep Attack Scenarios for Detection

in Incident Response by Troy Molsberry Comments are off
Many incidents that impact an organization’s security involve multiple steps. For example, an alert that a malicious email was transferred over the network is of concern, but there can be many thousands of these per day in a typical environment, and vetting each one out individually is prohibitive. Of more interest to the defender would be information about a malicious email being delivered, followed by a user clicking on a link contained in the email, followed by any downloads initiated by that user from blacklisted servers. In this example, we have an indicator (malicious email) followed by an action (clicked a link), followed by another action (download). In general, a “behavior” or “attack” consists of a sequence of causally related activities. Vetting these complex behaviors out by hand can be tedious at best, and intractable in most cases. You have to manually implement an algorithm known as “forward chaining”. Start with the first step in the sequence, and use attributes from the sensor data to perform a query for the second step using results from the previous query, and continue through the sequence until either a result is found or the trail goes cold. One interesting aspect of the “forward chaining” algorithm is that it explodes in both data and time. Performing this task by hand is more or less impossible, yet we commonly refer to this practice as “incident response”. These “incident responders” rely on a tremendous amount of experience, domain knowledge, and expertise to extract out behaviors that could potentially be security incidents. At Packetsled, we chose to capture that knowledge in a repeatable way.

Capturing knowledge from domain experts into models is a broad research topic, but I think we would all agree that at some point you will need a designer, e.g., a method for users to build models of attacks, so let’s start there. We chose a graphical notation for our models. Domain experts can visually model the causal relationships between queries, and those queries can contain forward- and backward- chaining references, e.g., those queries can depend on the results of previous or future queries. This is the magic.

Let’s create an example model for the example of a user clicking a malicious email link followed by an infection.

Read more

Attacks On Routers and IoT

in Incident Response, Malware, Network Visibility, Security Research by Patrick Kelley Comments are off
Here at PacketSled, we live on the forefront of technological innovation.  Our deployed platform captures multitudes of attacks each day against devices that have just barely made it into the market. To keep up with attacks against “bleeding edge” IoT devices is no small feat.

To understand how to best protect these new assets, it’s best to understand what brought them here.  The adoption of IoT and BYOD has increased with the intent of streamlining technology for the user and easing adoption in the marketplace..  Apps in IoT can become seamless, removing the barrier that a user used to have in determining if they were sharing information with a co-worker or with the Internet.  The more covert we make these transactions, the more risk we are accepting (think tap to pay, passwords transferred through nfc, and magic links as Slack likes to call them).  

Without a clear understanding of standards and services, it’s a challenge to determine how to best approach securing them. In fact, it’s such a challenge that Gartner has claimed that nearly all security vendors will fail at this task. 

Personally, I agree.  

Security budgets are rarely earmarked for efforts around IoT and business scenarios require a delivery mechanism that can also grow and keep pace with security requirements in monitoring, detection, and access control.  Despite this, users continually add more devices to the enterprise network, due to ease of doing so.

Fortunately, PacketSled has been thinking about IoT for quite some time.  Our team has spent many years of focused research on the most common IoT attacks.

One of the most common attacks we witness is authentication bypass. This could be due to poor session handling with predictable IDs or backdoors using hardcoded credentials. Regardless of the means, the outcome is the same – unauthorized access to sensitive information.

A quick search of Shodan will likely provide access to nearly any device, including devices in your infrastructure, an attacker would be interested in. In fact, we need not look further than a recent IoT attack which was seen with Mirai. It worked by scanning the Internet for devices with default credentials and enrolling them into the command and control platform. Once done, all of these devices can be remotely controlled and used to perform nearly any action conceivable. These attacks occurred across a wide spectrum of devices from smart TV’s to routers to really anything with the “smart” monicker attached.

Packetsled is here to help protect by proactively building detections in our platform to look for these behaviors, but our recommendation is to make sure your organization is covering the basics.

Where should you begin? 

Start with the CIS Critical Security Controls with emphasis on the 1st six. 
  1. Inventory of Authorized and Unauthorized devices
  2. Inventory of Authorized and Unauthorized software
  3. Security configurations for hardware and software on mobile and IoT devices
  4. Continuous Vulnerability Assessment and Remediation
  5. Controlled user of administrative or root privileges
  6. Maintenance, Monitoring, and Analysis of Audit Logs

With these basics addressed, you will have a more clear understanding of what devices and accounts are in use on your network and how you should expect them to behave.  

Among the many detections that PacketSled provides, you should look for unencrypted credentials present on the network, by issuing the following query in the investigator:

last 24 hours cluster password on [password]

This simple query will provide you with every password observed on the network in the last day.  This list is exportable and can be used to aid in the mitigation of these vulnerabilities.

We also provide extensive support for newer IoT protocols, along with our raw TCP and UDP analyzers, which will allow you to see inside network flows, even when a specific protocol analyzer isn’t available. 

Along with our security platform and the recommendations outlined above, we suggest signing up with each vendor for updates related to security patches, firmware updates and any available alerts provided from the devices, themselves.

Written by Patrick Kelley and Chris Mitzlaff 

Incident Response Strategy – Establish a Perimeter via Network Visibility

in Incident Response by Mike Spohn Comments are off

As a seasoned incident response practitioner, I am always looking for better ways to manage serious security breaches. Over the last decade, the cyber-security community has refined many strategies and best-practices to help organizations identify, investigate, contain, and remediate advanced threat attacks. This has been enormously helpful.

I have also found it useful to look beyond our own realm in cyber-space and observe how other industries manage large security incidents. A few years ago, I spent some time researching and interviewing public safety, fire, and military professionals. My goal was to determine if there are patterns of behavior in their response tactics that might apply to our IR space. 

Establish a Perimeter

It did not take long to realize that the foundation of most public safety incident handling practices is to, “Establish and secure a perimeter.” This may seem obvious to you, but it is important to realize the safety of human lives is often at stake if this is not done right. When you think about it, almost all public safety, search-and-rescue, and military operations begin with this strategy.

The most obvious example is the fighting of a wildfire. A large percentage of the effort is spent on surrounding the fire and creating a “Dozer line” free of debris to starve the fire. Granted, the firefighters are usually at the mercy of temperature, wind, and humidity. Regardless of the weather, the containment strategy is to surround the fire and work inward to contain it.

Network Visibility is like setting a fire line

Cutting a fire line – Image courtesy of FEMA


You see the same behavior when law enforcement agencies are faced with an act of terrorism. From the Boston Marathon attack to the bombing of the Brussels airport, the response was identical. Establish and secure a perimeter and work inward to determine the scope of the incident and look for suspects.

Sometimes this is really difficult. Consider the disappearance of Malaysia Air Flight MH370 on March 8, 2014. Lacking any reliable telemetry to determine where to search for the aircraft, a primary search area (perimeter) of 23,000 square miles was established. Folks, that is a big perimeter. Regardless, the same rule applied: establish a perimeter and search inward.


I immediately realized the value of this strategy in cyber-attack incident response investigations. In a cyber-attack response, the “perimeter” is almost always network boundaries. Why? If the source of the attack is not an insider, and the attacker(s) do not have physical access to your computing resources, the source of their attack will be an external network. This dynamic is obvious and compelling.

This makes it easy for incident responders to determine where to ‘establish’ a perimeter. It will always be where any external network has a route to your internal network. The first place to look is where your Internet points-of-presence (POP) are located.

Once you know the “scope” of your perimeter, you have to make some quick decisions on whether or not you “secure” it.

In the case of PCI, HIPAA, or other regulated data loss, you really have no choice but to secure the perimeter by shutting down the network segment. In other cases you need to make a hard decision. Do you lock out the intruders by securing the perimeter, or do you monitor it to learn more about the attacker TTP’s?

Tough choice.

If you secure the perimeter you tip off the attackers you know of their presence, and you lose the ability to collect additional, often critical, evidence. If you monitor the perimeter you run the risk of watching your precious data head to the Far East.

Here at PacketSled, we are all believers in the “Establish/Secure the perimeter and work inward” strategy when dealing with advanced threat actors. In fact, many of our customers rely on PacketSled network sensors to monitor their network perimeters during high-profile incidents.

Network Visibility is the Key to Establishing a Perimeter

Deploying a PacketSled sensor to establish a perimeter is painless. We have an IR deployment package that can have you up and running in minutes. Provide a SPAN/TAP feed to our sensor(s) and you have perimeter network visibility when you need it most.


PacketSled Network Visibility Automated Investigation Advanced Threat Hunting

PacketSled Dashboard


Let us show you the value of PacketSled network visibility tools. To arrange a product demonstration or talk IR strategies give us a call.

The Biggest Companies in the World Are Failing at Basic Detection

in Incident Response, Malware, PacketSled, Threat Detection by Web Admin Comments are off
We received a really interesting email from our upstream Internet provider yesterday – AT&T, indicating that we were effectively infected with Tinba malware. For those of you who don’t remember it, Tinba is 4 year old Windows specific Malware based on the old Blackhole exploit kit that does all kinds of fun stuff like steal facebook credentials, attempt to access your online banking accounts, etc. The email was as follows: att-incident Technical and operational observations are as follows:
  • First, we don’t possess any Windows machines capable of executing this Malware, with the exception of some VMs that are turned off unless otherwise needed.
  • Second, all of those machines have adequate endpoint protection on them, such that they would have averted this ancient attack, should it have been, you know, actually happening.
Regarding the email we received, if you haven’t already found the funny part, I’d like to point that out first. Obfuscating the IP address of the site that (according to them) indicates an infection is extraordinarily unhelpful, almost to the point where I feel like they might be trying to live up to some weird obtuse legacy of what it was like to order circuits in the 1990s. Thankfully, they took the time to include the domain name, which, you know…resolves to the obfuscated IP address listed directly above it. The unfortunate part about this situation is that they simply didn’t have the proper tools to make a quick determination that this was in fact, not a threat at all, and that we were running some tests on this network against legacy threat intel. Of course, we knew that, and we could easily validate that such testing was occurring during the timeframe that they mention with a simple query in the investigator as follows:
Querying specific IPs during specific timeframes using natural language

Querying specific IPs during specific timeframes using natural language

We get one session record returned:
PacketSled Investigator

PacketSled Investigator

And immediately determine that this “threat” originated from a Mac running Wget:
Hey look, not a Windows machine. Not even a real browser.

Hey look, not a Windows machine. Not even a real browser.

Nevertheless, we received this email, meaning that someone in the AT&T SOC had to investigate this non-incident, put in a ticket, send the email and manage the eventual closure of that ticket. This scenario is extremely commonplace. The number of SOC personnel, incident responders, and general infosec professionals that routinely chase non-incidents due to only having partial data is extremely perplexing. The data is in the packets. Making it accessible to analysts and automated processes such that it can be intelligently used is the goal. With something this simple getting incorrectly bubbled up to the top, we are clearly a long way off from that goal.

© 2017 PacketSled, Inc.