INSIGHTS | June 4, 2013

Industrial Device Firmware Can Reveal FTP Treasures!

Security professionals are becoming more aware of backdoors, security bugs, certificates, and similar bugs within ICS device firmware. I want to highlight another bug that is common in the firmware for critical industrial devices: the remote access provided by some vendors between their devices and ftp servers for troubleshooting or testing. In many cases this remote access could allow an attacker to compromise the device itself, the company the device belongs to, or even the entire vendor organization.
I discovered this vulnerability while tracking connectivity test functions within the firmware for an industrial device gateway. During my research I landed on a script with ftp server information that is transmitted in the clear:
The script tests connectivity and includes two subtests. The first is a simple ping to the host “8.8.8.8”. The second test uses an internal ftp server to download a test file from the vendor’s ftp server and to upload the results back to the vendor’s server. The key instructions use the wget and wput commands shown in the screen shot.
In this script, the wget function downloads a test file from the vendor’s ftp server. When the test is complete, the wput function inserts the serial number of the device into the file name before the file is uploaded back to the vendor.
This is probably done to ensure that file names identify unique devices and for traceability.
This practice actually involves two vulnerabilities:
  • Using the device serial number as part of a filename on a relatively accessible ftp server.
  • Hardcoding the ftp server credentials within the firmware.
This combination could enable an attacker to connect to the ftp server and obtain the devices serial numbers.
The risk increases for the devices I am investigating. These device serial numbers are also used by the vendor to generate default admin passwords. This knowledge and strategy could allow an attacker to build a database of admin passwords for all of this vendor’s devices.
Another security issue comes from a different script that points to the same ftp server, this time involving anonymous access.
This vendor uses anonymous access to upload statistics gathered from each device, probably for debugging purposes. These instructions are part of the function illustrated below.
As in the first script, the name of the zipped file that is sent back to the ftp server includes the serial number.
The script even prompts the user to add the company name to the file name.
An attacker with this information can easily build a database of admin passwords linked to the company that owns the device.

 

The third script, which also involves anonymous open access to the ftp server, gathers the device configuration and uploads it to the server. The only difference between this and the previous script is the naming function, which uses the configs- prefix instead of the stats- prefix.
The anonymous account can only write files to the ftp server. This prevents anonymous users from downloading configuration files. However, the server is running an old version of the ftp service, which is vulnerable to public exploits that could allow full access to the server.
A quick review shows that many common information security best practice rules are being broken:
  1. Naming conventions disclose device serial numbers and company names. In addition, these serial numbers are used to generate unique admin passwords for each device.
  2. Credentials for the vendor’s ftp server are hard coded within device firmware. This would allow anyone who can reverse engineer the firmware to access sensitive customer information such as device serial numbers.
  3. Anonymous write access to the vendor’s ftp server is enabled. This server contains sensitive customer information, which can expose device configuration data to an attacker from the public Internet. The ftp server also contains sensitive information about the vendor.
  4. Sensitive and critical data such as industrial device configuration files are transferred in clear text.
  5. A server containing sensitive customer data and running an older version of ftp that is vulnerable to a number of known exploits is accessible from the Internet.
  6. Using Clear text protocols to transfer sensitive information over internet
Based on this review we recommend some best practices to remediate the vulnerabilities described in this article:
  1. Use secure naming conventions that do not involve potentially sensitive information.
  2. Do not hard-code credentials into firmware (read previous blog post by Ruben Santamarta).
  3. Do not transfer sensitive data using clear text protocols. Use encrypted protocols to protect data transfers.
  4. Do not transfer sensitive data unless it is encrypted. Use high-level encryption to encrypt sensitive data before it is transferred.
  5. Do not expose sensitive customer information on public ftp servers that allow anonymous access.
  6. Enforce a strong patch policy. Servers and services must be patched and updated as needed to protect against known vulnerabilities.
INSIGHTS | May 23, 2013

Identify Backdoors in Firmware By Using Automatic String Analysis

The Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) this Friday published an advisory about some backdoors I found in two programmable gateways from TURCK, a leading German manufacturer of industrial automation products.
Using hard-coded account credentials in industrial devices is a bad idea. I can understand the temptation among manufacturers to include a backdoor “support” mechanism in the firmware for a product such as this. This backdoor allows them to troubleshoot problems remotely with minimal inconvenience to the customer.

On the other hand, it is only a matter of time before somebody discovers these ‘secret’ backdoors. In many cases, it takes very little time.

The TURCK backdoor is similar to other backdoors that I discussed at Black Hat and in previous blog posts. The common denominator is this: you do not need physical access to an actual device to find its backdoor. All you have to do is download device firmware from the vendor’s website. Once you download the firmware, you can reverse engineer it and learn some interesting secrets.
For example, consider the TURCK firmware. The binary file contains a small header, which simplifies the reverse engineering process:

 

 

The offset 0x1c identifies the base address. Directly after the header are the ARM opcodes. This is all the information you need to load the firmware into an IDA disassembler and debugger.

A brief review revealed that the device has its own FTP server, among other services. I also discovered several hard-coded credentials that are added when the system starts. Together, the FTP server and hard-coded credentials are a dangerous combination. An attacker with those credentials can completely compromise this device.
Find Hidden Credentials Fast
You can find hidden credentials such as this using manual analysis. But I have a method to discover hidden credentials more quickly. It all started during a discussion with friends at the LaCon conference regarding these kinds of flaws. One friend (hello @p0f ! ) made an obvious but interesting comment: “You could just find these backdoors by running the ‘strings’ command on the firmware file, couldn’t you?”

This simple observation is correct. All of the backdoors that are already public (such as Schneider, Siemens, and TURCK) could have been identified by looking at the strings … if you know common IT/embedded syntax. You still have to verify that potential backdoors are valid. But you can definitely identify suspicious elements in the strings.
There is a drawback to this basic approach. Firmware usually contains thousands of strings and it can take a lot of time to sift through them. It can be much more time consuming than simply loading the firmware in IDA, reconstructing symbols, and finding the ‘interesting’ functions.
So how can one collect intelligence from strings in less time? The answer is an analysis tool we created and use at IOActive called Stringfighter.
How Does Stringfighter Work?
Imagine dumping strings from a random piece of firmware. You end up with the following list of strings:
[Creating socket]
ERROR: %n
Initializing 
Decompressing kernel
GetSrvAddressById
NO_ADDRESS_FOUND
Xi$11ksMu8!Lma

 

From your security experience you notice that the last string seems different from the others—it looks like a password rather than a valid English word or a symbol name. I wanted to automate this kind of reasoning.
This is not code analysis. We only assume certain common patterns (such as sequential strings and symbol tables) usually found in this kind of binary. We also assume that compilers under certain circumstances put strings close to their references.
As in every decision process, the first step is to filter input based on selected features. In this case we want to discover ‘isolated’ strings. By ‘isolated’ I’m referring to ‘out-of-context’ elements that do not seem related to other elements.
An effective algorithm known as the ‘Levenshtein distance’ is frequently used to measure string similarity:
To apply this algorithm we create a window of n bytes that scans for ‘isolated’ strings. Inside that window, each string is checked against its ‘neighbors’ based on several metrics including, but not limited to, the Levenshtein distance.
However, this approach poses some problems. After removing blocks of ‘related strings,’ some potentially isolated strings are false positive results. For example, an ‘isolated’ string may be related to a distant ‘neighbor.’
We therefore need to identify additional blocks of ‘related strings’ (especially those belonging to symbol tables) formed between distant blocks that later become adjacent.
To address this issue we build a ‘first-letter’ map and run this until we can no longer remove additional strings.
At this point we should have a number of strings that are not related. However, how can we decide whether the string is a password? I assume developers use secure passwords, so we have to find strings that do not look like natural words.
We can consider each string as a first-order Markov source. To do this, use the following formula to calculate its entropy rate:
We need a large dictionary of English (or any other language) words to build a transition matrix. We also need a black list to eliminate common patterns.
We also want to avoid the false positives produced by symbols. To do this we detect symbols heuristically and ‘normalize’ them by summing the entropy for each of its tokens rather than the symbol as a whole.
These features help us distinguish strings that look like natural language words or something more interesting … such as undocumented passwords.
The following image shows a Stringfighter analysis of the TURCK BL20 firmware. The first string in red is highlighted because the tool detected that it is interesting.
In this case it was an undocumented password, which we eventually confirmed by analyzing the firmware.
The following image shows a Stringfighter analysis of the Schneider NOE 771 firmware. It revealed several backdoor passwords (VxWorks hashes). Some of these backdoor passwords appear in red.

Stringfighter is still a proof-of-concept prototype. We are still testing the ideas behind this tool. But it has exceeded our initial expectations and has detected backdoors in several devices. The TURCK backdoor identified in the CERT advisory will not be the last one identified by IOActive Labs. See you in the next advisory!
INSIGHTS | April 30, 2013

Fact or Fiction: Is Huawei a Risk to Critical Infrastructure?

How much of a risk does a company like Huawei or ZTE pose to U.S. national security? It’s a question that’s been on many peoples lips for a good year now. Last year the U.S. House of Representatives Permanent Select Committee on Intelligence warned American companies to “use another vendor”, and earlier in that year the French senator and former defense secretary Jean-Marie Bockel recommended a “total prohibition in Europe of core routers and other sensitive IT equipment coming from China.” In parallel discussions, the United Kingdom, Australia and New Zealand (to name a few) have restricted how Huawei operates within their borders.
Last week Eric Xu – executive vice-president and one of the triumvirate jointly running Huawei – stunned analysts when he told them that Huawei was “not interested in the U.S. market any more.”
Much of the analysis has previously focused upon Huawei’s sizable and influential position as the world’s second largest manufacturer of network routers and switching technology – a critical ingredient for making the Internet and modern telecommunications work – and the fact that it is unclear as to how much influence (or penetration) the Chinese government has in the company and its products. The fear is that at any point now or in the future, Chinese military leaders could intercept or disrupt critical telecommunications infrastructure – either as a means of aggressive statecraft or as a component of cyber warfare.
As someone who’s spent many years working with the majority of the world’s largest telecommunication companies, ISP’s, and cable providers, I’ve been able to observe firsthand the pressure being placed upon these critical infrastructure organizations to seek alternative vendors and/or replace any existing Huawei equipment they may already have deployed. In response, many of the senior technical management and engineers at these organizations have reached out to me and to IOActive to find out how true these rumors are. I use the term “rumor” because, while reports have been published by various government agencies, they’re woefully lacking in technical details. You may have also seen François Quentin (chairman of the board of Huawei France) claiming that the company is a victim of “rumors”.
In the public technical arena, there are relatively few bugs and vulnerabilities being disclosed in Huawei equipment. For example, if you search for CVE indexed vulnerabilities you’ll uncover very few. Compared to the likes of Cisco, Juniper, Nokia, and most of the other major players in routing and switching technology, the number of public disclosures is miniscule. But this is likely due to a few of the following reasons:
  • The important Huawei equipment isn’t generally the kind of stuff that security researchers can purchase off Ebay and poke around with at home for a few hours in a quest to uncover new bugs. They’re generally big-ticket items. (This is the same reason why you’ll see very few bugs publicly disclosed in Cisco’s or Nokia’s big ISP-level routers and switches).
  • Up until recently, despite Huawei being such a big player internationally, they haven’t been perceived as such to the English-speaking security researcher community – so have traditionally garnered little interest from bug hunters.
  • Most of the time when bugs are found and exploitable vulnerabilities are discovered, they occur during a paid-for penetration test or security assessment, and therefore those findings belong to the organization that commissioned the consulting work – and are unlikely to be publicly disclosed.
  • Remotely exploitable vulnerabilities that are found in Huawei equipment by independent security researchers are extremely valuable to various (international) government agencies. Any vulnerability that could allow someone to penetrate or eavesdrop at an international telecommunications carrier-level is worth big bucks and will be quickly gobbled up. And of course any vulnerability sold to such a government agency most certainly isn’t going to be disclosed to the vulnerable vendor – whether that be Huawei, Cisco, Juniper, Nokia, or whatever.

 

What does IOActive know of bugs and exploitable vulnerabilities within Huawei’s range of equipment? Quite a bit obviously – since we’ve been working to secure many of the telecommunications companies around the world that have Huawei’s top-end equipment deployed. It’s obviously not for me to disclose vulnerabilities that were uncovered on the dime of an IOActive client, however many of the vulnerabilities we’ve uncovered during tests have given great pause to our clients as remedies are sought.
Interesting enough, the majority of those vulnerabilities were encountered using standard network discovery techniques – which to my mind is just scratching the surface of things. However, based upon what’s been disclosed in these afore mentioned government reports over the last year, that was probably their level of scrutinization too. Digging deeper in to the systems reveals more interesting security woes.
Given IOActive’s expertise history and proven capability of hardware hacking, I’m certain that we’d be able to uncover a whole host of different and more significant security weaknesses in these critical infrastructure components for clients that needed that level of work done. To date IOActive the focus has be on in-situ analysis – typically assessing the security and integrity of core infrastructure components within live telco environments.
I’ve heard several senior folks talk of their fears that even with full access to the source code that that wouldn’t be enough to verify the integrity of Chinese network infrastructure devices. For a skillful opponent, that is probably so, because they could simply hide the backdoors and secret keys in the microcode of the devices semiconductor chips.
Unfortunately for organizations that think they can hide such critical flaws or backdoors at the silicon layer, I’ve got a surprise for you. IOActive already has the capability strip away the layers of logic within even the most advanced and secure microprocessor technologies out there and recover the code and secrets that have been embedded within the silicon itself.
So, I’d offer a challenge out there to the various critical infrastructure providers, government agencies, and to manufacturers such as Huawei themselves – let IOActive sort out the facts from the multitude of rumors. Everything you’ve probably been reading is hearsay.
Who else but IOActive can assess the security and integrity of a technology down through the layers – from the application, to the drivers, to the OS, to the firmware, to the hardware and finally down to the silicon of the microprocessors themselves? Exciting times!
— Gunter Ollmann, CTO IOActive Inc.