Category: INSIGHTS

INSIGHTS | July 16, 2013

2013 ISS Conference, Prague

By Lucas Lundgren
I had the opportunity to attend the 2013 ISS conference in Prague a few weeks ago. The conference is a place where company representatives and law enforcement (and other government agency) officials can meet to share ideas and products information (such as appliances). Even though I had a sore throat, I still found it quite interesting; although not necessarily in terms of the products and presentations – which I felt was overall a bit flat.

It was easy to differentiate between company representatives and government officials. Government officials wore yellow ID tags, while company representatives wore purple ID tags. These tags stated the individual’s first and last name and the company or government agency they represented.

I didn’t know what to expect, because I had never been to an ISS conference. However, I quickly realized that the conference itself could be an attacker’s paradise. For example, one could easily use a camera phone to take undetected photographs of the various government officials.

Being inquisitive by nature, I decided to conduct an experiment. First, I turned my ID tag (which I was supposed to visibly wear around my neck at all times) backwards so that nobody could see who I was. Even with hotel security guards stationed at all entrances, nobody stopped me or checked my badge.

This is an attack scenario in and of itself. My only interaction with a security guard occurred when I tried to take a shortcut to exit the conference—but even then, I was not asked to show my badge.

The first day had many presentations given. I attended a few of these but found that they were not particularly interesting. A law enforcement official illustrated how to find and check EXIF data in pictures. I asked a question relating to a page with “right-click” protection on. He answered by saying that they needed a third-party program (an open source one) to download the page. I was baffled and wondered what ever happened to “View Source”, which exists in most browsers. What about blocking JavaScript? What happened to File > Save As? This disturbed me, but not as much as what happened later on in the day.

Some of the presentations required a yellow (government) ID tag to attend. With security guards stationed outside of each presentation, I wanted to see if I could successfully enter any of these. I used a yellow piece of paper, borrowed the hotel printer to create a new ID tag, and wrote my name as Mr. Sauron Chief of Security, Country of Mordor. Just like that! I smiled, nodded, and entered the presentation as a yellow-badged participant. To be clear, the presentation had not yet begun, so I quickly exited after a minute or two.

Later in the day, I attended a presentation on TOR, Darknet, and so on. During the presentation, I overheard a number of yellow-badged participants indicating they had never heard of TOR. As the presentation went on, I had a general feeling that the presenter viewed TOR as a safe way to stay anonymous. However, I see it as a network by which attackers can obtain a substantial amount of data (usernames, passwords, credentials, and so on) after setting up their own TOR networks.

There were problems with Wi-Fi access at the hotel. Guests had to pay, for example, 80 Euros per day for a 1MB line (per day). No cables and wiring had been set up beforehand for attendees, so technicians were busy setting these up during the conference. I found this to be a bit dangerous.

When I checked the wireless network, I found that the hotel had a “free” access point to which I quickly connected. I found it ironic, being at an ISS conference, that the hotel used insecure items such as clear text as part of their free connection!

If you happen to represent law enforcement in your country, do not (I repeat, DO NOT)…
  • Connect to anything anywhere in the area,
  • Accept invalid SSL certificates,
  • Use your Viber or What’s Up messenger to send messages (clear text protocols),
  • Use non-encrypted protocols to check your email,
  • Publicly display your name and the agency you represent unless asked to do so by a security representative wearing the proper badge

 

The same rules that apply to the Defcon and Blackhat conferences should apply at the ISS conference—or any security conference for that matter!

If I had been an evil attacker at the ISS conference, I could have easily sat in the lounge downstairs all day and overheard all kinds of conversations about products, firewalls, and solutions used by a variety of countries. Also, by simply using the “free” hotel Wi-Fi, I could have gained access to a number of participant email messages, text messages, and web pages sending username and password credentials in clear text. Imagine what I could have done with a hotel voucher containing a locked account!

A colleague of mine attending the conference decided to perform a quick experiment using the SSLstrip tool to test for hotel network vulnerabilities. Moxie Marlinspike introduced this tool at the Black Hat DC 2009 conference to demonstrate how attackers can perform HTTP stripping attacks. SSLstrip prompts users to use an invalid certificate, which they can accept or reject. Much to our surprise, ISS conference participants accepted our invalid certificate. My colleague and I were completely baffled and blown away by this! I would like to note that we were not performing this experiment for malicious reasons. We simply wanted to verify the network vulnerability at the conference and provide our feedback to ISS conference and hotel stakeholders in this blog.

Using a tool similar to SSLstrip, an attacker would not even have to enter the main conference area to perform attacks. He could sit back in the smoker’s lounge, order a beverage of choice, set up sniffing, lean back on the couch, and let participants do the rest of the work!
Don’t get me wrong. The conference offered a lot of interesting topics and presentations. Someone presented a board equipped with Bluetooth, wireless, and a 3g module (for listening to calls) that had Linux as a base operating system. Anyone can buy this, not just government officials. The potential an attacker getting this into his hands is huge, and it is only the size of a Rasberry Pi.

Another security concern at the conference involved the use of Bluetooth and Wi-Fi. People everywhere were using the Internet on their phones and had Bluetooth activated. You have to ask yourself, would these be activated at a Blackhat conference?

It’s obvious that law enforcement and other governmental agencies need training with regard to the popular hacking techniques used at conferences. We encourage such agencies to contact us at IOActive for help in this arena.

 

Perhaps you are reading this blog post and realizing that you too have used free Wi-Fi to check email, turned Bluetooth/Wi-Fi on in public places, or accepted a faulty SSL certificate. Promise me one thing… at the next conference you attend make sure everything at the hotel is safe and turn Bluetooth/Wi-Fi off on your devices. Do not talk loudly about things that are supposed to be confidential Do so after the conference! Also, if you are an organizer at the next ISS conference, please be sure to properly check participant badges. Also, consider using something more secure than a paper ID tag with a name on it.
INSIGHTS | July 11, 2013

Why Vendor Openness Still Matters

By Mike Davis
When the zombies began rising from their graves in Montana it had already been over 30 days since IOActive had reported issues with Monroe Electronics DASDECS.
 
And while it turned out in the end that the actual attacks which caused the false EAS messages to be transmitted relied on the default password never having been changed, this would have been the ideal point to publicize that there was a known issue and that there was a firmware update available, or would soon be to address this and other problems… maybe a mitigation or two in the mean time right?

At a minimum it would have been an ideal time to provide the simple mitigation:
“rm ~/.ssh/authorized_keys”
 
Unfortunately this never happened, leading to a phenomena I like to call “admin droop”.  This is where an administrator, after discovering the details of a published vulnerability, determines that he’s not vulnerable because he doesn’t run the default configuration and everything is working and doesn’t bother to upgrade to the next version.
 
… it happens…
 
In the absence of reliable information other outlets such as the FCC provided pretty minimal advice about changing passwords and using firewalls; I simply commented to the media that this was “inadequate” advice.
 
Eventually, somewhere around April 24 Monroe, with the assistance of CERT begin contacting customers about a firmware fix, we provided a Shodan XML file with a few hundred vulnerable hosts to help track them down. Finally it looked like things were getting fixed but I was somewhat upset that I still had not seen official acknowledgement of the issue from Monroe but on June 13 Monroe finally published this https://ioactive.com/wp-content/uploads/2013/07/130604-Monroe-Security-PR.pdf 
security advisory where it stated “Removal of default SSH keys and a simplified user option to load new SSH keys”, ok its not much of an “announcement” but its something and  I know it says April 24, but both the filename and metadata (pdfinfo) point to cough later origins…
 
Inside the advisory is this wonderful sentence: 
“The company notes that most of its users already have obtained this update.”… That sound s like something worth testing!
 
Then it happened, before I could say “admin droop”… 
 
Found/Vulnerable hosts before we reported the issue:   222
Found hosts After the patch was released, as found by a survey on July  11:  412
 
Version numbers       
         Hosts Found
       Vulnerable (SSH Key)
1.8-5a
                1
                 Yes
1.8-6
                2
                 Yes
2.0-0
              110
                 Yes
2.0-0_a01
                1
                 Yes
2.0-1
               68           
                 Yes
2.0-2 (patched)
               50
                  No
unreachable                              180

While most users may have “obtained” the update, someone certainly didn’t bother applying it…
 
Yep… it’s worse now than it was before we reported the issue in January, and everyone thinks everything is just fine. While I’m sure this would still be a problem had a proper security notice been issued.
 
I can’t say it any better than these awesome folks over at Google http://googleonlinesecurity.blogspot.com/2013/05/disclosure-timeline-for-vulnerabilities.html : “Our standing recommendation is that companies should fix critical vulnerabilities within 60 days — or, if a fix is not possible, they should notify the public about the risk and offer workarounds.”

 

INSIGHTS | July 4, 2013

Why sanitize excessed equipment

By Reid Wightman

My passion for cybersecurity centers on industrial controllers–PLCs, RTUs, and the other “field devices.” These devices are the interface between the integrator (e.g., HMI systems, historians, and databases) and the process (e.g., sensors and actuators). Researching this equipment can be costly because PLCs and RTUs cost thousands of dollars. Fortunately, I have an ally: surplus resellers that sell used equipment.

I have been buying used equipment for a few years now. Equipment often arrives to me literally ripped from a factory floor or even a substation. Each controller typically contains a wealth of information about its origin. I can often learn a lot about a company from a piece of used equipment. Even decades-old control equipment has a lot of memory and keeps a long record about the previous owner’s process. It is possible to learn the “secret recipe” with just a few hours of work at reverse engineering a controller to collect company names, control system network layout, and production history. Even engineers’ names and contact information is likely to be stored in a controller’s log file. For a bad guy, the data could be useful for all sorts of purposes: social engineering employees, insider trading of company stock, and possibly direct attacks to the corporate network.
I reach out to the origin of used equipment when I find these types of information. I help them wipe the equipment, and I point them to where the rest of the equipment is being sold in an attempt to recall it before the stored information ends up in the wrong hands. I am not the only one doing this kind of work. Recently, Billy Rios and Terry McCorkle revealed surplus equipment that they had purchased from a hospital. It had much of the same information about its origin.

These situations can be prevented by sanitizing the equipment before it’s released for disposal. Many equipment manufacturers should be able to provide instructions for this process. One option may be to send the controller back to the manufacturer to be sanitized and refurbished.
A way to provide another layer of protection against information disclosure is to have a robust and well-practiced Incident Response plan. Most places that I contact are great to work with and are receptive to the information.

Ignoring the issue, especially where a public utility is concerned, may be considered a violation. Set up an Incident Response program now and make sure that your process control engineers know to send equipment disposal issues through the IR group.

 

A great deal can be accomplished to keep a control system secure. With a little planning, proper equipment disposal is one of the cheapest things that can be done to keep proprietary process information safe.
INSIGHTS | June 20, 2013

FDA Safety Communication for Medical Devices

By Gunter Ollmann
The US Food and Drug Agency (FDA) released an important safety communication targeted at medical device manufacturers, hospitals, medical device user facilities, health care IT and procurements staff, along with biomedical engineers in which they warn of risk of failure due to cyberattack – such as through malware or unauthorized access to configuration settings in medical devices and hospital networks.
Have you ever been to view a much anticipated movie based upon an exciting book you happened to have read when you were younger, only to be sorely disappointed by what the director finally pulled together on the big screen? Well that’s how I feel when I read this newest alert from the FDA. Actually it’s not even called an alert… it’s a “Safety Communication”… it’s analogous to Peter Jackson deciding that his own interpretation of JRR Tolkien’s ‘The Hobbit’ wasn’t really worthy of the title so to forestall criticism he named the movie ‘Some Dwarves and a Hobbit do Stuff’.
This particular alert (and I’m calling it an alert because I can’t lower myself to call it a safety communication any longer) is a long time coming. Almost a decade ago me and my teams at the time raised the red flag over the woeful security of hospital networks, then back in 2005 my then research teams raised new red flags related to the encroachment of unsecured WiFi in to medical equipment, for the last couple of years IOActive’s research team have been raising new red flags over the absence of security within implantable medical devices, and then on June 13th 2013 the FDA releases a much watered down alert where the primary recommendations and actions section simply states “[m]any medical devices contain configurable embedded computer systems that can be vulnerable to cybersecurity breaches”. It’s as if the hobbit has been interpreted as a midget with hairy feet.
Yes I joke a little, but I am very disappointed with the status of this alert covering an important topic.
The vulnerabilities being uncovered on a daily basis within hospital networks, medical equipment and implantable devices by professional security teams and researchers are generally more serious than what outsiders give credit. Much of the public cybersecurity discussion as it relates to the medical field to date has been about people hacking hospital data systems for patient records and, most recently, the threat of targeted slayings of people who happen to have vulnerable implanted insulin pumps and heart defibrillators. While both are certainly possible, they’re what I would associate with fringe events.
I believe that the biggest and most likely threats lie in non-malicious actors – the tinkerers, the cyber-crooks, and the “in the wrong place at the wrong time” events. These medical systems are so brittle that even the slightest knock or tire-kicking can cause them to fail. I’ll give you some examples:
  • Wireless heart and drug monitoring stations within emergency wards that have open WiFi connections; where anyone with an iPhone searching for an Internet connection can make an unauthenticated connection and have their web browser bring up the admin portal of the station.
  • Remote surgeon support and web camera interfaces used for emergency operations brought down by everyday botnet malware because someone happened to surf the web one day and hit the wrong site.
  • Internet auditing and scanning services run internationally and encountering medical devices connected directly to the Internet through routable IP addresses – being used as drop-boxes for file sharing groups (oblivious to the fact that it’s a medical device under their control).
  • Common WiFi and Bluetooth auditing tools (available for android smartphones and tablets) identifying medical devices during simple “war driving” exercises and leaving the discovered devices in a hung state.
  • Medial staff’s iPads without authentication or GeoIP-locking of hospital applications that “go missing” or are borrowed by kids and have applications (and games) installed from vendor markets that conflict with the use of the authorized applications.
  • NFC from smartphone’s and payment systems that can record, playback and interfere with the communications of implanted medical devices.
These are really just the day-to-day noise of an Internet connected life – but one that much of the medical industry is currently ill prepared to defend against. Against an experienced attacker or someone determined to cause harm – well, it’s as one sided as a lone hobbit versus the combined armies of Middle Earth.
I will give the alert some credit though, that did clarify a rather important point that may have been a stumbling block for many device vendors in the past:
“The FDA typically does not need to review or approve medical device software changes made solely to strengthen cybersecurity.”
IOActive’s experience when dealing with a multitude of vulnerable medical device manufacturers had often been disheartening in the past. A handful of manufacturers have made great strides in securing their devices and controlling software recently – and there has been a change in the hearts and minds over the last 6 months (pun intended) as more publicity has been drawn to the topic. The medical clients we’ve been working most closely with over recent months have made huge leaps in making their latest devices more secure, and their next generation of devices will be setting the standard for the industry for years to come.
In the meantime though, there’s a tremendous amount of work to be done. The FDA’s alert is significant. It is a formal recognition of the poor state of security within the industry – providing some preliminary guidance. It’s just not quite a call to arms I’d have liked to see after so many years – but I guess they don’t want to raise too much fear, nor the ire of vendors that could face long and costly FDA re‑evaluations of their technologies. Gandalf would be disappointed.
(BTW I actually liked Peter Jackson’s rendition of The Hobbit).
INSIGHTS | June 14, 2013

Red Team Testing: Debunking Myths and Setting Expectations

By Ian Amit
The term “cyber” seems to be overused in every corner of the information security industry. Now there is a new buzz phrase in computer security, “red team engagements.” Supposedly (to get “cyber” on you), you can have a red team test, and it will help move your organization in the correct “cyber direction.”
But what is red team testing really? And what is it not? In this post I’ll try to make some sense of this potent term.
The red team concept has been around for ages. It started as a military term for a team dedicated to simulating all of an enemy’s activities, including everything from methodology to doctrine, strategy, techniques, equipment, and behaviors. The red team was tasked with mastering how the adversary thinks and operates, and then executing the enemy’s strategies and tactics in the field. This allows your own forces to experience what it would be like to combat this enemy in real life − without the risk of getting injured or killed.
Fast forward 10-12 years and red teams are being used in civilian industry as well as in the military. In private industry, red teams simulate adversaries (not necessarily foreign armies) that could impact the organization. The adversary could be criminals vying to get your money, competitors trying to get their hands on your latest designs, or random attackers who want to exploit, destroy, or simply harm your organization. Their motivations can range from social activism, political strategy, financial gain, and so on.

When IOActive is asked to conduct a red team test, our main goal is to accurately and realistically simulate these types of adversaries. So the first, and probably most important, element of a red team test is to define the threat model:

·      Who is your adversary?
·      What are their motivations?
·      Which adversaries are you most afraid of? (This is usually any party that wants to put you out of business.)
Defining the threat model is crucial to the success of a red team engagement, because it determines the value your organization will get from the project.
After we articulate the threat model, the fun part of the engagement begins.
Fun? Yes, because in the real world most tests, such as penetration tests do not really depict a persistent adversary. Instead, engagements such as pen tests simulates specific strategies that a persistent adversary will use as part of an overall attack.
The red team engagement, on the other hand, includes all possible elements that an adversary might use in such an attack (which is why it is often referred to as “no scope” or “full scope” testing).
In this context, everything including your employees, your infrastructure, the physical office locations, your supply chain − that’s every third party you use as part of your ongoing operations  − and more. When developing attack scenarios for red team engagements, every element has to fit in perfectly.

Think of it as an “Ocean’s Eleven” type of attack that can include:

·      Social engineering
·      Electronic and digital attacks
·      Bypassing physical controls
·      Equipment tampering
·      Getting into the supply chain to access your assets
·      And more

This is full scope testing. Unlike in other types of engagement, all or almost all assets are “in scope”.

Note: Red team engagements do commonly use “reverse scoping” techniques to identify assets that are critical to operations and the types of tampering, access, or removal that are off limits for these assets. These sensitive assets are still in scope. But reverse scoping defines and restricts actions that may substantially disrupt operations.)
So far this sounds like a fun game. But hold on, it isn’t just about the attack. What I like the most is seeing how an organization’s ongoing security operations handle red team attacks.
In a red team test, very few people in the organization know about the test, and even fewer actually know when the test will happen. This means that from an operational security view, all red team activities are treated as if they involve a real adversary.

We gain a lot of insights from the actions and reactions of the organization’s security team to a red team attack. These are the types of insights that matter the most to us:

  • Observing how your monitoring capabilities function during the intelligence gathering phase. The results can be eye opening and provide tremendous value when assessing your security posture.
  • Measuring how your first (and second) responders in information security, HR, and physical security work together. Teamwork and coordination among teams is crucial and the assessment allows you to build processes that actually work.
  • Understanding what happens when an attacker gains access to your assets and starts to exfiltrate information or actually steals equipment. The red team experience can do wonders for your disaster recovery processes.
These are some of the rewards and benefits of a red team test. As you can see, they go well above and beyond what you would get from other types of focused tests.
I hope this explanation eliminates some of the confusion about red team testing that I have seen lately in Internet discussions. I am not saying that there is no value in pen tests, social engineering engagements, physical assessments, or anti-phishing campaign. However, to see how all of these different types of security considerations work in the real world, they also need to be considered as part of a larger (and relevant) context so that you can see how well your organization is prepared for any type of attack.
Last but not least, if you’d like to get hands-on training in how red team engagements are conducted, considering attending our two-day Red Team Training (https://www.blackhat.com/us-13/training/red-team-training.html) at BlackHat 2013 in Las Vegas.
Chris Nickerson and I will cover everything discussed in this post, with particular focus on the elements that go beyond penetration testing. Our topics will include lock picking, social engineering, physical assessments, and, most importantly, how to combine all of these elements into a realistic and successful simulation of an adversary. We also plan to give each of our students a very interesting goodie bag.
Hint: You’ll have fun walking through airport security with that bag :-).
INSIGHTS | June 11, 2013

Tools of the Trade – Incident Response, Part 1: Log Analysis

By Wim Remes
There was a time when I imagined I was James Bond zip lining into a compromised environment, equipped with all kinds of top-secret tools. I would wave my hands over the boxes needing investigation, use my forensics glasses to extract all malware samples, and beam them over to Miss Moneypenny (or “Q” for APT concerns) for analysis. I would produce the report from my top-notch armpit laser printer in minutes. I was a hero.
As wonderful as it sounds, this doesn’t ever happen in real life. Instead of sporting a classy tuxedo, we are usually knee deep in data… often without boots! I have recently given a few presentations(1) on Incident Response (IR). The question I am most often asked concerns the tool chain that would enable an individual or a team to perform the basic actions one would expect from an Incident Responder.

Let me be clear. There is an immense difference between walking on to a crime scene in which your organization’s intent is to bring a perpetrator to court and approximately 90% of the actual IR work you will perform. The biggest difference has to do with how you collect and handle evidence, and you can rest assured that any improperly handled evidence will be relentlessly attacked by the defense.
However, I am writing this blog post from the point of view of a blue team member. I’ll focus on the tools that are available to determine what has happened, when it happened, and (most importantly) how to make sure it does not happen again.
TL;DR : Many, if not all, of the tools you will need are available in the SANS   Investigate Forensics Toolkit (SIFT) VMWare image(2) . You can skip the rest of this post and start playing with these tools right away if you want to, but I hope I can at least give you some pointers on where to look first.
As I illustrated in the introduction, IR work is incredibly unsexy. Almost without exception, you will run into a special case that requires you to work on the command line for hours trying to figure out what has occurred. The next-next finish of many a security product installation and use process is something you will nostalgically reflect on over a burning log (pun intended).
You will typically have in your possession three important pieces of evidence:
  • Log files
  • Memory images
  • Disk images
I will address these in three separate blog posts and try to give you some ideas about the tools you can use during your investigation.
In this blog post, we will start off by looking at log files.
Every device on the network creates a gigantic amount of data. Some of it is used for monitoring purposes, but most of it is (hopefully) stored and never to be looked at again. Until a system is compromised, this data is generally useless with the exception of a few hidden golden threads. These threads will give you a good starting point for figuring out what has happened. They might also be useful in tying the whole story together.  Essentially, you must figure out what the log files want to tell you. (Cue the X-Files soundtrack.)
Sifting through log files is like looking for gold. Your initial sift will be very coarse grained. As you get rid of more and more of the dirt, the mesh of your sift will get finer and finer.
In my humble opinion, there is nothing better than having a thorough understanding of the Unix command-line tools ‘grep’, ‘sed’, and ‘awk’.  The good thing about these tools is that you will find them on almost any Unix or Linux box you can get your hands on. The bad thing is that you will either need to know or need to find out what you are looking for.
In the absence of a full-fledged SIEM infrastructure, two tools exist that I recommend you become familiar with:
  • OSSEC(3)
  • log2timeline(6)
First up, OSSEC. Not only is this a good (and open-source) host-based intrusion detection system, but it also allows you to feed it collected logs through its command-line tools!  The events will trigger OSSEC out-of-the-box alerts and give you a good overview of the most interesting events that have taken place. You can also easily develop custom rule sets based on (as an example) publicly available data from other compromises(4)
As you become more experienced with OSSEC, you can build your own set of scripts, drawing inspiration from a variety of places. In cases where ye olde *nix is not available, make sure you are also familiar with a Windows shell tool. (Powershell is one heck of an alternative!) Also, keep your Command Line Kung Fu(5) handy!
You will not be able to perform IR without the powerful yet extremely elegant log2timeline(6) tool. This blog post does not offer enough space to provide a decent overview of log2timeline. In essence, it groks data from different sources and creates a ‘Super Timeline’ that helps you to narrow in on those timeframes that are relevant to your investigation. It is one thing to dig through information on a command line, but it is a completely different ball game when you can visualize events as they have occurred.
In upcoming blog posts, I will dig into the juicy secrets that can be found in memory and disk images. A major lesson to be learned as you climb on the ladder of IR is that many of the tools you will use are produced by your peers. Sure, IR software is available and works well. However, the bulk of tools available to you have been developed by your fellow incident responders to address a particular problem. Many responders generously release their tools to the public domain for others to use. We learn from others probably more than we teach.
 

(1) BYO-IR: Build Your Own Incident Response: http://www.slideshare.net/wremes/isc2-byo-ir
(2) http://computer-forensics.sans.org/community/downloads
(3) http://www.ossec.net
(4) e.g. APT1 Indicators of Compromise @ http://intelreport.mandiant.com/
(5) http://blog.commandlinekungfu.com/
(6) http://log2timeline.net/

INSIGHTS | June 4, 2013

Industrial Device Firmware Can Reveal FTP Treasures!

By Sofiane Talmat
Security professionals are becoming more aware of backdoors, security bugs, certificates, and similar bugs within ICS device firmware. I want to highlight another bug that is common in the firmware for critical industrial devices: the remote access provided by some vendors between their devices and ftp servers for troubleshooting or testing. In many cases this remote access could allow an attacker to compromise the device itself, the company the device belongs to, or even the entire vendor organization.
I discovered this vulnerability while tracking connectivity test functions within the firmware for an industrial device gateway. During my research I landed on a script with ftp server information that is transmitted in the clear:
The script tests connectivity and includes two subtests. The first is a simple ping to the host “8.8.8.8”. The second test uses an internal ftp server to download a test file from the vendor’s ftp server and to upload the results back to the vendor’s server. The key instructions use the wget and wput commands shown in the screen shot.
In this script, the wget function downloads a test file from the vendor’s ftp server. When the test is complete, the wput function inserts the serial number of the device into the file name before the file is uploaded back to the vendor.
This is probably done to ensure that file names identify unique devices and for traceability.
This practice actually involves two vulnerabilities:
  • Using the device serial number as part of a filename on a relatively accessible ftp server.
  • Hardcoding the ftp server credentials within the firmware.
This combination could enable an attacker to connect to the ftp server and obtain the devices serial numbers.
The risk increases for the devices I am investigating. These device serial numbers are also used by the vendor to generate default admin passwords. This knowledge and strategy could allow an attacker to build a database of admin passwords for all of this vendor’s devices.
Another security issue comes from a different script that points to the same ftp server, this time involving anonymous access.
This vendor uses anonymous access to upload statistics gathered from each device, probably for debugging purposes. These instructions are part of the function illustrated below.
As in the first script, the name of the zipped file that is sent back to the ftp server includes the serial number.
The script even prompts the user to add the company name to the file name.
An attacker with this information can easily build a database of admin passwords linked to the company that owns the device.

 

The third script, which also involves anonymous open access to the ftp server, gathers the device configuration and uploads it to the server. The only difference between this and the previous script is the naming function, which uses the configs- prefix instead of the stats- prefix.
The anonymous account can only write files to the ftp server. This prevents anonymous users from downloading configuration files. However, the server is running an old version of the ftp service, which is vulnerable to public exploits that could allow full access to the server.
A quick review shows that many common information security best practice rules are being broken:
  1. Naming conventions disclose device serial numbers and company names. In addition, these serial numbers are used to generate unique admin passwords for each device.
  2. Credentials for the vendor’s ftp server are hard coded within device firmware. This would allow anyone who can reverse engineer the firmware to access sensitive customer information such as device serial numbers.
  3. Anonymous write access to the vendor’s ftp server is enabled. This server contains sensitive customer information, which can expose device configuration data to an attacker from the public Internet. The ftp server also contains sensitive information about the vendor.
  4. Sensitive and critical data such as industrial device configuration files are transferred in clear text.
  5. A server containing sensitive customer data and running an older version of ftp that is vulnerable to a number of known exploits is accessible from the Internet.
  6. Using Clear text protocols to transfer sensitive information over internet
Based on this review we recommend some best practices to remediate the vulnerabilities described in this article:
  1. Use secure naming conventions that do not involve potentially sensitive information.
  2. Do not hard-code credentials into firmware (read previous blog post by Ruben Santamarta).
  3. Do not transfer sensitive data using clear text protocols. Use encrypted protocols to protect data transfers.
  4. Do not transfer sensitive data unless it is encrypted. Use high-level encryption to encrypt sensitive data before it is transferred.
  5. Do not expose sensitive customer information on public ftp servers that allow anonymous access.
  6. Enforce a strong patch policy. Servers and services must be patched and updated as needed to protect against known vulnerabilities.
INSIGHTS | May 29, 2013

Security 101: Machine Learning and Big Data

By Gunter Ollmann

The other week I was invited to keynote at the ISSA CISO Forum on Incident Response in Dallas and in the weeks prior to it I was struggling to decide upon what angle I should take. Should I be funny, irreverent, diplomatic, or analytical? Should I plaster slides with the last quarter’s worth of threat statistics, breach metrics, and headline news? Should I quip some anecdote and hope the attending CISO’s would have an epiphany that’ll fundamentally change the way they secure their organizations?

In the end I did none of that… instead I decided to pull apart the latest batch of security buzzwords – “Big Data” and “Machine Learning”.

If you attended RSA USA (or any major security vendor/booth conference) this year you can’t have missed the fact that everything from Antivirus through to USB memory sticks now come with a dab of big data, a sprinkling of machine learning, and a dollop of cloud for good measure. Thankfully I’m a cynic; or else I’d have been thrashing around on the ground in an epileptic fit from all the flashy marketing claims and trademarked nonsense phrases.

I guess I’m lucky to be in the position of having had several years of exposure to some of the greatest minds at Georgia Tech as they drummed in to me on a daily basis the “what and how” of machine learning in the context of solving many of today’s toughest security problems.

So, it was with that in mind that I thought “If I’m a CISO and everything I know about machine learning and big data came from carefully rehearsed vendor sound bites and glossy pamphlets, would I be able to tell the difference between Chanel #5 and cow manure?” The obvious answer would result in some very happy farmers.

What was the net result of this self-inflection and impending deadline? I crafted a short presentation for CISO’s… a 101 course on machine learning and big data… and it included ducks.

If you’re in the upper tiers of your organization and you’ve had sales folks pimping you their latest cloud-infused, big data-munching, machine learning, and world-hunger-solving security solution, please carry on reading as I attempt to explain the basics of the latest and greatest in buzzwords…

First of all – some context! In the world of breach detection and incident response there’s a common idiom: “If it walks like a duck, flies like a duck, and quacks like a duck… it must be a duck.”

Now I could easily spend another 5,000 words explaining why such an idiom doesn’t apply to modern security threats, targeted attacks and advanced persistent threats, but you’ll have to wait for a different blog post. Rather, for this 101 lesson, it’s important to understand the concept of “Feature Selection” – which in the case of this idiom includes: walking, flying and quacking.

If you’ve been tasked with dealing with a duck problem, ideally you’d be an aficionado on the feet, wings and sounds of ducks. You’d be able to apply this knowledge to each bird you have the time to focus your attention on and make a determination: Duck, or Not a Duck. As a security professional, you’d be versed in the various attributes of certain threats – and able to derive a conclusion as to the nature of the security problem.

The problem though is that at scale things break down.
What do you do when there’s too many to analyze, when time is too short, and when you can’t make out all the duck features from afar? This is typical of the big data problem (and your everyday network traffic). Storing the data is the easy part. Retrieving the data is mechanically complicated, but solvable.

Meanwhile, making decisions and taking actions upon the data is typically the most difficult part. With every doubling of data, your problem grows exponentially.

The traditional method of dealing with the situation has been to employ signature matching systems. In essence, we build rules based upon the features we’ve previously identified as significant and capable of bounding the problem (or duck selection). We then compare these rules against the sample animal and receive a binary answer – Duck, or Not a Duck.

Signature systems can be very efficient at classification. Just look at your average Intrusion Prevention System (IPS). A problem though lies in the scope of the features that had been defined.

If those features (or parameters) used for classification are too narrow (or too broad) then evasion is not only probable, but guaranteed. In essence, for a threat (or duck) to be classified, it must have been observed in the past or carefully predicted (although rare).

From an attacker’s perspective, knowledge of those features and triggering parameters makes it a trivial task to evade or to conduct false flag operations. Just think – hunters do this all the time with their floating duck decoys. Even fellow duck hunters have been known to mistakenly take pot-shots at them too.

Switching pace a little, let’s look at the network a little.
The big green blob is all the network traffic for an organization for a week. The red blog right-of-center is traffic associated with an active breach, and the lighter red blob with the dotted lines are just general malicious traffic observed within the network. In this two-dimensional view (if I hadn’t color-coded it previously) you’d have a near impossible task differentiating between them. As it is, the malicious traffic is mixed with both the “safe” and “breach” traffic.

The trick in differentiating between the network traffic types lies in increasing the dimensionality of the problem. What was a two-dimensional blob suddenly becomes much clearer when an appropriate view or perspective to the data is added. In the context of the above diagram, the addition of a z-axis and an extension in to the third-dimension allows the observer (i.e. analyst) to easily differentiate between the traffic types – for example, the axis could represent “country code of destination IP address”. In this context, the appropriate feature selection can greatly simplify the detection problem. Choosing appropriate features is important – nay, it’s critical!

This is where advances in machine learning over the last half-decade have really come to the fore in computer science and more recently in information security.

Without getting in to any of the math behind the various machine learning algorithms or techniques, the key concept you need to understand is “training”. It can mean many things to many a mathematician, but since we’re likely not one of those, what training means in our context is that we already have samples of what we’re going to be looking for, and samples of things we know we’re definitely not interested in. The better we define and populate these training sets, the more precise the machine learning system we’re employing will be in differentiating between them – and potentially classifying other contenders.

So, in this example we’ve taken a bunch of ducks and grouped them together. They become our “+ve class” – which basically means these are the things we’re interested in. But, equally important, is our “-ve class” – our collection of things we know not to be ducks. In many cases our -ve class may be more important than our +ve class because it contains all those false positives and “nearly” ducks – the things that may have caught us out once before.

One function of a good machine learning system is to automatically determine which attributes make the most sense in differentiating between your +ve and -ve classes.
While our poor old hunter (or analyst) was working with three features – walks, flies, and talks – the computer-based system may have reviewed all the attributes that were available and determined which ones are the most useful in differentiating between “ducks” and “not ducks”. In many cases the system will have weighted the various features (or attributes) to indicate which features are more deterministic of the classes.

For example, texture may be a good indicator of “not a duck” – since none of the +ve class were made from plastic or wood. Meanwhile features such as “wing length” may not be such a good criteria and will be weighted in a way to not have an influence on determining whether a duck is a duck or not – or may be dropped by the system entirely.

The number of features reviewed, assessed and weighted by the machine learning system will eventually be determined by the type of data being used and how “rich” it is. For example, in the network security realm we may be feeding the system with collated samples of firewall logs, DNS traffic samples, IP blacklists/whitelists, IPS alerts, etc. It’s important to note though that the “richer” the data set (i.e. the more possible features there could be), the more complex the problem is for the computer to solve and the longer it’ll take to train the system.

Now let’s switch back to that “big data” we originally talked about. In the duck realm we’re talking about all the birds within a national park (say). Meanwhile, in the network security realm, we may be talking about all the traffic observed in real-time across a corporate network and all the alerting instrumentation (e.g. firewalls, IPS, etc.)

I’m going to do some hand-waving here because it can get rather complex and there’s a lot of proprietary tweaks that can be undertaken here… but in one representation we can get our trained system to automatically group and cluster events on our network.
Using our original training data, or some other previously labeled datasets, it’s possible to automatically label the clusters output by a machine learning system.
For example, in the graphic above we see a number of unique clusters (or blobs if you insist). Through automatic labeling we know that the green blobs are types of ducks, the red blobs are various groupings of not ducks, and the gray blobs are clusters of previously unknown or unlabeled clusters – each one mathematically distinct from the other – based upon the features the system chose.
What the system can also do is assign a probability that the unknown clusters are associated with our +ve or -ve training sets. For example, in this particular graphical representation the proximity of the unlabeled clusters to labeled (and classified) clusters allows the system to assign a probability of whether the cluster is a duck or not a duck – even though the system had never seen these things before (i.e. “birds” the system hasn’t encountered before).
The next (and near final) stage is to manually label these new clusters. For example, we ask an ornithologist to look at each cluster of “ducks” and “not ducks” in turn and to label them… “rubber duckies”, “robot duckies”, and “Madagascar mallard ducks”.

Then, to improve our machine learning system further, we add these newly labeled clusters to our +ve and -ve training sets… and the system continues to learn and become more precise over time.

In addition, since we’ve now labeled these clusters, in the future we’re able to automatically flag new additions to these clusters and correctly label the duck (or threat).

And, if we’re a really smart CISO, we can use this clustering system (and labeled clusters) to automatically alert us to new threats or to initiate automatic network security actions – e.g. enable blocking of a new malicious URL, stop blocking a new cloud service delivering official updates to applications, etc.

The application of machine learning techniques to the toughest security problems facing business today has come along in leaps and bounds recently. However as with any buzz word that falls in to the hands of marketers and gets manipulated until it squeaks and glitters, or oozes onto every product in this year’s price list, senior technical staff need to take added care not to be bamboozled by well-practiced but meaningless word salad.

 A little understanding of the concepts behind big data and machine learning can not only cut through the latest batch of sales promises, but can also form the basis of constructing a new generation of meaningful breach detection and incident response solutions.
INSIGHTS | May 23, 2013

Identify Backdoors in Firmware By Using Automatic String Analysis

By Ruben Santamarta
The Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) this Friday published an advisory about some backdoors I found in two programmable gateways from TURCK, a leading German manufacturer of industrial automation products.
Using hard-coded account credentials in industrial devices is a bad idea. I can understand the temptation among manufacturers to include a backdoor “support” mechanism in the firmware for a product such as this. This backdoor allows them to troubleshoot problems remotely with minimal inconvenience to the customer.

On the other hand, it is only a matter of time before somebody discovers these ‘secret’ backdoors. In many cases, it takes very little time.

The TURCK backdoor is similar to other backdoors that I discussed at Black Hat and in previous blog posts. The common denominator is this: you do not need physical access to an actual device to find its backdoor. All you have to do is download device firmware from the vendor’s website. Once you download the firmware, you can reverse engineer it and learn some interesting secrets.
For example, consider the TURCK firmware. The binary file contains a small header, which simplifies the reverse engineering process:

 

 

The offset 0x1c identifies the base address. Directly after the header are the ARM opcodes. This is all the information you need to load the firmware into an IDA disassembler and debugger.

A brief review revealed that the device has its own FTP server, among other services. I also discovered several hard-coded credentials that are added when the system starts. Together, the FTP server and hard-coded credentials are a dangerous combination. An attacker with those credentials can completely compromise this device.
Find Hidden Credentials Fast
You can find hidden credentials such as this using manual analysis. But I have a method to discover hidden credentials more quickly. It all started during a discussion with friends at the LaCon conference regarding these kinds of flaws. One friend (hello @p0f ! ) made an obvious but interesting comment: “You could just find these backdoors by running the ‘strings’ command on the firmware file, couldn’t you?”

This simple observation is correct. All of the backdoors that are already public (such as Schneider, Siemens, and TURCK) could have been identified by looking at the strings … if you know common IT/embedded syntax. You still have to verify that potential backdoors are valid. But you can definitely identify suspicious elements in the strings.
There is a drawback to this basic approach. Firmware usually contains thousands of strings and it can take a lot of time to sift through them. It can be much more time consuming than simply loading the firmware in IDA, reconstructing symbols, and finding the ‘interesting’ functions.
So how can one collect intelligence from strings in less time? The answer is an analysis tool we created and use at IOActive called Stringfighter.
How Does Stringfighter Work?
Imagine dumping strings from a random piece of firmware. You end up with the following list of strings:
[Creating socket]
ERROR: %n
Initializing 
Decompressing kernel
GetSrvAddressById
NO_ADDRESS_FOUND
Xi$11ksMu8!Lma

 

From your security experience you notice that the last string seems different from the others—it looks like a password rather than a valid English word or a symbol name. I wanted to automate this kind of reasoning.
This is not code analysis. We only assume certain common patterns (such as sequential strings and symbol tables) usually found in this kind of binary. We also assume that compilers under certain circumstances put strings close to their references.
As in every decision process, the first step is to filter input based on selected features. In this case we want to discover ‘isolated’ strings. By ‘isolated’ I’m referring to ‘out-of-context’ elements that do not seem related to other elements.
An effective algorithm known as the ‘Levenshtein distance’ is frequently used to measure string similarity:
To apply this algorithm we create a window of n bytes that scans for ‘isolated’ strings. Inside that window, each string is checked against its ‘neighbors’ based on several metrics including, but not limited to, the Levenshtein distance.
However, this approach poses some problems. After removing blocks of ‘related strings,’ some potentially isolated strings are false positive results. For example, an ‘isolated’ string may be related to a distant ‘neighbor.’
We therefore need to identify additional blocks of ‘related strings’ (especially those belonging to symbol tables) formed between distant blocks that later become adjacent.
To address this issue we build a ‘first-letter’ map and run this until we can no longer remove additional strings.
At this point we should have a number of strings that are not related. However, how can we decide whether the string is a password? I assume developers use secure passwords, so we have to find strings that do not look like natural words.
We can consider each string as a first-order Markov source. To do this, use the following formula to calculate its entropy rate:
We need a large dictionary of English (or any other language) words to build a transition matrix. We also need a black list to eliminate common patterns.
We also want to avoid the false positives produced by symbols. To do this we detect symbols heuristically and ‘normalize’ them by summing the entropy for each of its tokens rather than the symbol as a whole.
These features help us distinguish strings that look like natural language words or something more interesting … such as undocumented passwords.
The following image shows a Stringfighter analysis of the TURCK BL20 firmware. The first string in red is highlighted because the tool detected that it is interesting.
In this case it was an undocumented password, which we eventually confirmed by analyzing the firmware.
The following image shows a Stringfighter analysis of the Schneider NOE 771 firmware. It revealed several backdoor passwords (VxWorks hashes). Some of these backdoor passwords appear in red.

Stringfighter is still a proof-of-concept prototype. We are still testing the ideas behind this tool. But it has exceeded our initial expectations and has detected backdoors in several devices. The TURCK backdoor identified in the CERT advisory will not be the last one identified by IOActive Labs. See you in the next advisory!
INSIGHTS | May 7, 2013

Bypassing Geo-locked BYOD Applications

By Gunter Ollmann
In the wake of increasingly lenient BYOD policies within large corporations, there’s been a growing emphasis upon restricting access to business applications (and data) to specific geographic locations. Over the last 18 months more than a dozen start-ups in North America alone have sprung up seeking to offer novel security solutions in this space – essentially looking to provide mechanisms for locking application usage to a specific location or distance from an office, and ensuring that key data or functionality becomes inaccessible outside these prescribed zones.
These “Geo-locking” technologies are in hot demand as organizations try desperately to regain control of their networks, applications and data.

Over the past 9 months I’ve been asked by clients and potential investors alike for advice on the various technologies and the companies behind them. There’s quite a spectrum of available options in the geo-locking space; each start-up has a different take on the situation and has proposed (or developed) a unique way in tackling the problem. Unfortunately, in the race to secure a position in this evolving security market, much of the literature being thrust at potential customers is heavy in FUD and light in technical detail.
It may be because marketing departments are riding roughshod over the technical folks in order to establish these new companies, but in several of the solutions being proposed I’ve had concerns over the scope of the security element being offered. It’s not because the approaches being marketed aren’t useful or won’t work, it’s more because they’ve defined the problem they’re aiming to solve so narrowly that they’ve developed what I could only describe as tunnel-vision to the spectrum of threat organizations are likely to face in the BYOD realm.
In the meantime I wanted to offer this quick primer on the evolving security space that has become BYOD geo-locking.
Geo-locking BYOD
The general premise behind the current generation of geo-locking technologies is that each BYOD gadget will connect wirelessly to the corporate network and interface with critical applications. When the device is moved away from the location, those applications and data should no longer be accessible.
There are a number of approaches, but the most popular strategies can be categorized as follows:
  1. Thick-client – A full-featured application is downloaded to the BYOD gadget and typically monitors physical location elements using telemetry from GPS or the wireless carrier directly. If the location isn’t “approved” the application prevents access to any data stored locally on the device.
  2. Thin-client – a small application or driver is installed on the BYOD gadget to interface with the operating system and retrieve location information (e.g. GPS position, wireless carrier information, IP address, etc.). This application then incorporates this location information in to requests to access applications or data stored on remote systems – either through another on-device application or over a Web interface.
  3. Share-my-location – Many mobile operating systems include opt-in functionality to “share my location” via their built-in web browser. Embedded within the page request is a short geo-location description.
  4. Signal proximity – The downloaded application or driver will only interface with remote systems and data if the wireless channel being connected to by the device is approved. This is typically tied to WiFi and nanocell routers with unique identifiers and has a maximum range limited to the power of the transmitter (e.g. 50-100 meters).

The critical problem with the first three geo-locking techniques can be summed up simply as “any device can be made to lie about its location”.

The majority of start-ups have simply assumed that the geo-location information coming from the device is correct – and have not included any means of securing the integrity of that device’s location information. A few have even tried to tell customers (and investors) that it’s impossible for a device to lie about its GPS location or a location calculated off cell-tower triangulation. I suppose it should not be a surprise though – we’ve spent two decades trying to educate Web application developers to not trust client-side input validation and yet they still fall for web browser manipulations.
A quick search for “fake location” on the Apple and Android stores will reveal the prevalence and accessibility of GPS fakery. Any other data being reported from the gadget – IP address, network MAC address, cell-tower connectivity, etc. – can similarly be manipulated. In addition to manipulation of the BYOD gadget directly, alternative vectors that make use of private VPNs and local network jump points may be sufficient to bypass thin-client and “share-my-location” geo-locking application approaches.
That doesn’t mean that these geo-locking technologies should be considered unicorn pelts, but it does mean that organization’s seeking to deploy these technologies need to invest some time in determining the category of threat (and opponent) they’re prepared to combat.
If the worst case scenario is of a nurse losing a hospital iPad and that an inept thief may try to access patient records from another part of the city, then many of the geo-locking approaches will work quite well. However, if the scenario is that of a tech-savvy reporter paying the nurse to access the hospital iPad and is prepared in install a few small applications that manipulate the geo-location information in order to remotely access celebrity patient records… well, then you’ll need a different class of defense.
Given the rapid evolution of BYOD geo-locking applications and the number of new businesses offering security solutions in this space, my advice is two-fold – determine the worst case scenarios you’re trying to protect against, and thoroughly assess the technology prior to investment. Don’t be surprised if the marketing claims being made by many of these start-ups are a generation or two ahead of what the product is capable of performing today.
Having already assessed or reviewed the approaches of several start-ups in this particular BYOD security realm, I believe some degree of skepticism and caution is warranted.
— Gunter Ollmann, CTO IOActive