Big Data Security Analytics

Home » Big Data Security Analytics

The Challenges of Threat Detection in a Changing Landscape

in Articles, Big Data Security Analytics, PacketSled, Threat Detection by Matt Harrigan Comments are off
Having spent 20 years being exposed to the real world security problems of customers as a trusted advisor, I’ve had the unique privilege of observing the common challenges that large financial, retail, defense, and critical infrastructure organizations face when trying to address complex, evolving threats. I’ve also watched the security community go from a niche, perhaps misunderstood group of ex-hackers trying to solve the problems that they discover in the soft underbelly of the early Internet to building an entire multi-billion dollar industry who’s goal it is to address the problems that plagues every employee, officer, director, and consumer. Companies have spent millions of dollars on security products to protect themselves and their customer’s data. However, there are still holes, and so-called advanced threats are getting through the cracks. What is needed, once and for all, is a solution that does a better job of detecting advanced threats, as well as providing a means for containing, analyzing and remediating unknown threats that have gotten through traditional defenses. Today’s security practitioners, engineers, and operating teams are currently using SIEM and/or full packet network forensics. Let’s talk about these options. Screen Shot 2013-12-17 at 6.15.33 PMOption 1- SIEM Tools – The idea here is that millions and millions of events being generated by hundreds or thousands of devices are cataloged in a central repository for correlation and processing. Let’s consider this for a minute; In order for technologies that rely on log data to correlate events, three things have to happen: 1. The devices that matter in the context of a security event have to be pointing log data at the SIEM. Most organizations typically prioritize this by starting with their security infrastructure (FW, IPS/IDS, Endpoint). This requirement to prioritize the output of these technologies seems to ignore the fact that the attack vector may have been designed specifically to avoid or elude them, and that the entry point for advanced threats may be somewhere else entirely. 2. Accuracy of the logs you require relies on integrity of the endpoints generating these logs – Attackers compromise IPS/FW technologies routinely, pivoting from internally compromised hosts into the admin interface of these technologies. At that point, its fairly trivial to create or disable rules that enable them to add a pathway for tunneled traffic that goes undetected by said devices, and certainly isn’t generating any useful log data that might be caught by the SIEM. 3. Alerts setup correctly – So what do you do with all the data you now have in one pane of glass? (if you’re even able to do that). Most analysts will tell you that the bulk of alerts they currently look at are not valuable, and that a very small percentage of the data coming across the console pertains to real security threats. Getting meaningful event correlation out of standalone SIEM using noise reduction techniques is difficult to do correctly, and even some of the best security organizations struggle with it.   full packetOption 2- Full Packet Solutions– Traditional full packet technologies offer the promise of much deeper insight into the activities of attackers than traditional log-only solutions, but they exhibit some fundamental problems that preclude them from operating in a manner that is consistent with anything other than “after the fact” scenarios. Areas of concern include: 1. Historical forensic record length – If a user issues a SQL query that results in a dump of thousands of records, and the network forensics solution has visibility into that session, guess what – you’re storing those thousands of records now. If you place a sensor on the ingress/egress points of your network, and someone watches a 40 minute youtube video on bonsai gardening, you get to keep that too. Neither of these things are valuable in the context of a security incident, but nevertheless, they’re quickly eating up expensive storage on your network. While your SOC team may learn some new root trimming techniques, they aren’t getting security value from the data. The fundamental challenge here is that storage requirements exceed the need to retain the data. One saturated 10Gbps link can produce up to 108TB of stored data per day, or 3.2 petabytes per month. It’s no mystery why these tools aren’t being deployed enterprise-wide. The incidents are buried in forests of tiny trees. 2. Fast, intelligent access to the data – Full IP packet data is by definition binary. Other than augmentation of the data with descriptors, full packet solutions have no way of answering user queries or machine generated analytical inquires in a reasonable timeframe that is acceptable for a typical incident response plan. In short – pulling back full packet is slow and ineffective in a real-time solution. It also makes it impossible to distribute the solution over very large environments, because sending the same data to a remote central repository would absolutely destroy even the most resilient networks. 3. Cost of retention –78% of all breaches are being discovered months, or sometimes even years after they’ve occurred. The data is already gone, and the damage is already done. Buying more disks may sound like a cheap option, but ask anyone in charge of buying these solutions, and you will quickly be corrected to understand that an enterprise-wide deployment of full packet capture simply isn’t an option financially. At PacketSled, we care deeply about the issues that plague today’s technology-enabled companies. We’re delivering solutions to these very difficult problems by building a next generation of products that can produce answers about attacker behavior, assets that are or may be impacted, and fastest path to threat eradication / remediation. We do it at scale, in real-time, for the entire enterprise, and in the cloud, at a cost that is commensurate with the security budgets of our customers. I’m very excited to be here, and I look forward to working with you. Matt Harrigan President & CEO PacketSled

© 2016 PacketSled, Inc.