The number of reports of networks that are rampaged by adversaries is staggering. In the past few weeks alone we’ve seen reports from The New York Times, The Washington Post and Twitter. I would argue that the public reports are just the tip of the iceberg. What about the hacks that never were? What about the companies that absorbed the blow and just kept on trucking or … perhaps even those companies that never recovered?
When there’s an uptick in media attention over security breaches, the question most often asked – but rarely answered – is “What if this happens to me?”
Today you don’t want to ask that question too loudly – else you’ll find product vendors selling turn-key solutions and their partners on your doorstep, closely followed by ‘Managed Security Services’ providers. All ready to solve your problems once you send back their signed purchase order… if you want to believe that.
Most recently they’ve been joined by the “let’s hack the villains back” start-ups. That last one is an interesting evolution but not important for this post today.
I’m not here to provide a side-by-side comparison of service providers or product vendors. I encourage you to have an open conversation with them when you’re ready for it; but what I want to share today is my experience being involved in SIEM projects at scale, and working hands-on with the products as a security analyst. The following lessons were gained through a lot of sweat and tears:
- SIEM projects are always successful, even when delivered over budget, over time and with only 1/3rd of the requirements actually realized.
- Managed services don’t manage your services, they provide the services they can manage at the price you are willing to pay for them.
- There is no replacement for knowing your environment. Whatever the price you are willing to pay for it.
This obviously begs the question whether installing a SIEM is worth spending a limited security budget upon.
It is my personal opinion that tooling to facilitate Incident Response, including SIEM to delve through piles and piles of log data, is always an asset. However, it’s also my opinion that buying a tool RIGHT NOW is not your priority if you can not confidently answer “YES” to the following questions :
- I know where my most valuable assets reside on my network, which controls are implemented to protect them and how to obtain security-related data from those controls.
- I know the hardware and software components that support those most valuable assets and know how to obtain security-related data from them.
- I know, for those most valuable assets, which devices communicate with them, at which rate and at which times. I can gather relevant data about that.
- I know, and can verify, which machines can (and should) communicate with external networks (including the internet). I can gather relevant data about that.
In case of a resounding “NO” or a reluctantly uttered “maybe”, I would argue that there are things you should do before acquiring a SIEM product. It is your priority to understand your network and to have control over it, unless you look forward to paying big money for shiny data aggregators.
Advice
With that challenge identified how do you move ahead and regain control of your network. Here’s some advice :
The most important first step is very much like what Forensics investigators call “walking the grid”. You will need to break down your network in logical chunks and scrutinize them. Which are the components that are most important to our business and what are the controls protecting them. Which data sources can tell us about the security health of those components and how? how frequently? in which detail? Depending on the size and complexity of the network this may seem like a daunting task but at the same time you’ll have to realize that this is not a one time exercise. You’ll be doing this for the foreseeable future so it’s important that it is turned into a repeatable process. A process that can be reliably executed by other people than you, with consistent results.
Next, epiphany! Nobody but yourself can supplement the data delivered by appliances and software distributed across your network with actual knowledge about your own network. Out of the box rulesets don’t know about your network and -however bright they may be- security analysts on follow-the-sun teams don’t either. Every event and every alert only makes sense in context and when that context is removed, you’re no longer doing correlation. You’re swatting flies. While I came up with this analogy on the fly, it makes sense. You don’t swat flies with a Christian Louboutin or a Jimmy Choo, you use a flip-flop. There are plenty of information security flip-flops out there on the internet. Log aggregators (syslog, syslog-ng), Full-blown open-source NIDS systems (Snort, Bro) or HIDS systems (OSSEC), and even dedicated distributions (e.g. Security Onion) or open source SIEM (like OSSIM) can go a long way in helping you understand what is going on in your network. The most amazing part is that, until now, you’ve spend $0 on software licensing and all of your resources are dedicated to YOUR people learning about YOUR network and what is going on there. While cracking one of the toughest nuts in IT Security, you’re literally training your staff to be better security analysts and incident responders.
The first push-back I generally receive when I talk (passionately, I admit) about open source security tooling is in the form of concern. The software is not controlled, we can’t buy a support contract for it (not always true by the way!!), our staff doesn’t have the training, we are too small/big for these solutions…It’s impossible to argue closed vs. open source in this blogpost.
I believe it’s worth looking at to solve this problem, others may disagree. To the point of training staff I will say that what those tools largely do is what your staff currently does manually in an ad hoc fashion. They do understand logging and network traffic, learning how the specific tools work and how they can make their job easier will be only a fraction of the time they spend on implementing them. It is my experience that the enthusiasm of people that get to work with tools –commercial or otherwise– that makes their actual job easier, compensates for any budget you have to set aside for ‘training’. To the point of size, I have personally deployed open source security tools in SMB environments as well as in 1000+ enterprise UNIX farms. It is my strongest belief that, as security engineers, it is not our job to buy products. It is our task to build solutions for the problems at hand, using the tools that best fit the purpose. Commercial or not.
It makes sense that, as you further mature in your monitoring capability, the free tools might not continue to scale and you’ll be looking to work with the commercial products or service providers I mentioned above. The biggest gain at that moment is that you perfectly understand what you need, which parts of the capability you can delegate to a third party and what your expectations are, which parts of the problem space you can’t solve without dedicated products. From experience, most of the building blocks will be easily reused and integrated with commercial solutions. Many of those commercial solutions have support for the open source data generators (Snort, Bro, OSSEC, p0f, …).
Let’s be realistic: if you’re as serious about information security as I think you are, you don’t want to be a “buyer of boxes” or a cost center. You want to (re)build networks that allow you to defend your most valuable assets against those adversaries that matter and, maybe as important as anything else, you want to stop running behind the facts on fancy high heels.
*For the purpose of this post SIEM stands for Security Information and Event Management. It is often referred to as SIM, SEM and a bunch of other acronyms and we’re ok with those too.