INSIGHTS | February 14, 2014

The password is irrelevant too

In this follow up to a blog post on the Scalance-X200 series switches, we look at an authentication bypass vulnerability. It isn’t particularly complicated, but it does allow us to download configuration files, log files, and a firmware image. It can also be used to upload configuration and firmware images, which causes the device to reboot.
 
The code can be found in IOActive Labs github repository.
 
If an attacker has access to a configuration file with a known password, they can use this code to update the configuration file and take over the switch’s management functions. It can also be used to mirror ports and enable or disable other services, such as telnet, SSH, or SNMP. Lastly, the same script can be used to upload a firmware image to the device sans authentication. In other words, it is *delightfully reprogrammable* until you install the patch.
 
 
This brings us to an interesting point. I asked Siemens if the SSH keys in Firmware V5.X (the fixed version) are unique per device, and I was assured that they are. If this is true, there should be no problem with me publishing a hash of the private key for my device. Don’t worry damsels and chaps, I can always patch my device with a new key later, as a demonstration of my enthusiasm for firmware. 
 
Anyway, here are two fingerprints of the private SSH key: 
 
MD5   6f09a4d77569236fd90483a85920912d
SHA256    505166f90ee05761b11a5feda24d0ccdc53ef902cdd617b330db3634cc2788f7
 
If you have one of these devices and have patched to the version that contains fixes, you could assist the community greatly by verifying that the key gets a different finger-print. This will independently confirm what those outstanding gentry at Siemens told me and promote confidence in their security solutions.
 
This neatly segues into some changes we’ve seen in the ICS-space over the last few years. 
 
The primary change in behavior I’d like to applaud is how companies are striving to develop better relationships with independent security researchers such as myself. The increase in constructive dialogue is evidenced by Siemen’s ability to receive notification and then produce a patch within three months. Years ago we were regularly waiting six months to two years for fixes.
 
In fact, I challenged vendors at S4x14 to commit to an AVERAGE TIME of security patching for externally supplied vulnerabilities. We purposefully chose the average time for this challenge, because we know that providing quality assurance for these systems is difficult and can be time consuming. After all, some bugs are just thornier than others
 
Incidentally, this is backed up by empirical research shared with me by the inimitable Sean McBride during our conversations at S4x14. I wouldn’t want you to think I am just some un-gentlemanly shuffler or simkin, challenging hecatonchires for the sport of it (hat-tip @sergeybratus).
 

 

Follow @digitalbond to see the response I got to committing to an average security patch time, when my ”Red/Blue Live” talk goes online. You’ll also notice that my two attackers (red team) did not manage to use the script to take over the device, despite doing so in practice sessions the night before. The ingenious Rotem Bar (blue team) demonstrated that the secret of ICS security is to simply *patch*. Apparently, it is not only possible, but effective!
…and btw, happy Valentine’s!
INSIGHTS | January 13, 2014

The password is irrelevant

This story begins with a few merry and good hearted tweets from S4x13. These tweets in fact:
 
 
Notice the shared conviviality, and the jolly manner in which this discussion of vulnerabilities occurs.
 
It is with this same lightness in my heart that I thought I would explore the mysterious world of the.

 
So I waxed my moustache, rolled up my sleeves, and began to use the arcane powers of Quality Assurance. 
 
Ok, how would an attacker who doesn’t have default credentials or a device to test on go about investigating one of these remotely? Why, find one on Shodan of course!
 
 
Personally, I buy mine second hand from eBay with the fortune I inherited from my grandfather’s moustache wax empire.
 
The first thing an attacker would normally do is scan the device to get familiar with the ports and services. A quick nmap looks like this:
 
Nmap scan report for Unknown (192.168.0.5)
Host is up (0.0043s latency).
Not shown: 994 closed ports
PORT    STATE SERVICE  VERSION
22/tcp  open  ssh      OpenSSH 4.2 (protocol 2.0)
|_ssh-hostkey: 1024 cd:b4:33:49:62:3b:58:1a:67:5a:a3:f0:50:00:71:86 (RSA)
23/tcp  open  telnet?
80/tcp  open  http     WindWeb 4.00
|_http-methods: No Allow or Public header in OPTIONS response (status
code 501)
|_http-title: Logon to SCALANCE X Management (192.168.0.5)
84/tcp  open  ctf?
111/tcp open  rpcbind  2 (RPC #100000)
| rpcinfo:
|   program version   port/proto  service
|   100000  2            111/tcp  rpcbind
|_  100000  2            111/udp  rpcbind
443/tcp open  ssl/http WindWeb 4.00
|_http-methods: No Allow or Public header in OPTIONS response (status
code 501)
|_http-title: Logon to SCALANCE X Management (192.168.0.5)
| ssl-cert: Subject: organizationName=Siemens
AG/stateOrProvinceName=BW/countryName=DE
| Not valid before: 2008-02-04T14:05:57+00:00
|_Not valid after:  2038-01-18T14:05:57+00:00
|_ssl-date: 1970-01-01T00:14:20+00:00; -43y254d14h08m05s from local time.
| sslv2:
|   SSLv2 supported
|   ciphers:
|     SSL2_DES_192_EDE3_CBC_WITH_MD5
|     SSL2_RC2_CBC_128_CBC_WITH_MD5
|     SSL2_RC4_128_WITH_MD5
|     SSL2_RC4_64_WITH_MD5
|     SSL2_DES_64_CBC_WITH_MD5
|     SSL2_RC2_CBC_128_CBC_WITH_MD5
|_    SSL2_RC4_128_EXPORT40_WITH_MD5
MAC Address: 00:0E:8C:A3:4E:CF (Siemens AG A&D ET)
Device type: general purpose
Running: Wind River VxWorks
OS CPE: cpe:/o:windriver:vxworks
OS details: VxWorks
Network Distance: 1 hop
 
So we have a variety of management interfaces to choose from: Telnet (really in 2014?!?), SSH, HTTP, and HTTPS. All of these interfaces share the same users and default passwords you would expect, but we are looking for something more meaningful. 
 
Now that we’ve found them on Shodan (wait, they’re all air-gapped, right?), we quickly learn from the web interface that there are only two users: admin and user. Next we view the web page source and search for “password” which gives us this lovely snippet:
 
document.submitForm.encoded.value = document.logonForm.username.value + “:” + md5(document.logonForm.username.value + “:” + document.logonForm.password.value + “:” + document.submitForm.nonceA.value)
 
 
This is equivalent to the following command on Linux:
 
echo -n “admin:admin:C0A8006500005F31” | md5sum
 
 
Which is then posted to the device in a form such as this (although this one has a different password*):
 
encoded=admin%3Aafc0dc177659c8e9d72dec8a3c68650e&nonceA=C0A800610000CE29
 
Setting aside just how weak the use of MD5 is (and in fact I have written a script to brute-force credentials snatched off the wire), that nonceA value is very important. A nonce is a ‘number used once’, which is typically used to prevent cryptographic replay attacks. In other words, this random challenge value is provided by the server, to keep an attacker from simply capturing the hash on the wire and replaying it to the server later when they want to login. This nonce then, deserves our attention.
 
It appears that this is an ID field in the cookie, and that it is also the session ID. If I can predict session Ids, I can perform session hijacking while someone is managing the switch. So we set about estimating the entropy of this session ID, which initially appears to be 16 hex values. However, we won’t even need to create an equation since it turns out to be COMPLETELY predictable, as you will soon see. 
 
We can use WGET to fetch the page a few times and get a small sample of these nonceA values. This is what we see:
 
C0A8006500005F31,C0A8006500001A21,C0A8006500000960,C0A80065000049A6
 
This seems distinctly non-random. In fact, when I measured it more precisely, it became clear that it was sequential! A little further investigation revealed that SNMP is sometimes available to us. So we use snmpwalk on one of the devices I have at home:
 
snmpwalk -Os -c public -v 1 192.168.0.5
iso.3.6.1.2.1.1.1.0 = STRING: “Siemens, SIMATIC NET, SCALANCE X204-2,
6GK5 204-2BB10-2AA3, HW: 4, FW: V4.03″
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.4196.1.1.5.2.22
iso.3.6.1.2.1.1.3.0 = Timeticks: (471614) 1:18:36.14
 
Well look at that! 
 
47164 in base 10 = 7323E in hex! I wonder if the session ID is simply uptime in hex?
 
We do a WGET at approximately the same time and get this as a session ID:
 
C0A800610007323F
 
So if we assume the last 8 hex chars are uptime in hex (or at least close enough for us to brute-force the few values around it), then where do the first 8 hex come from? 
 
I initially imagined they were unique for each device and set out to test that theory by getting a session ID from another device found on Shodan. Keep in mind I did not attempt a login, I just fetched the page to see what the session ID was. Sure enough it was different, but the one from the switch I have wasn’t the MAC address or any other unique identifier on the switch. I spent a week missing the point here, but luckily I have excellent company at IOActive Labs. 
 
It was during discussions with my esteemed colleague Reid Wightman, he suggested it was an IP address. He pointed out the C0 and A8 are 192 168 in decimal. So I went and checked the IP address of the switch I have, and it was not 192.168.0.97. So again I was stumped until I realized it was the IP address of my own client machine!
 
In other words, the nonceA was simply the address of the web client (converted to hex) connecting to the switch, concatenated to the uptime of the switch (in hex). I can practically see the developer who thought this would be a good idea. I can hear them thinking that each session is clearly distinguished by the address it is connecting from, and made impossible to brute-force with time. Time+Space, that’s too large to brute-force or estimate right? You must admit, it has a kind of perverse logic to it. Unfortunately, it is highly predictable and insecure.
 
Go home Scalance X200 family session IDs, you’re drunk. Aside from being completely predictable, they are too small. 32 hex is a far cry from using the 128 bits recommended by OWASP.
 
I guess that’s what they refer to in this announcement with the phrases “A potential vulnerability” and “that might allow attackers to hijack web sessions over the network without authentication”. 
 
There are a few more vulnerabilities to discuss about this switch, but to learn about those, you’ll have to see me at S4x14, or wait for the next blog post where I release a more reliable POC.
 
Usually I am one to organise a nice quiet coordinated disclosure, probably over a lavender scone and a cup of tea. I do my best to be polite, cheerful, and helpful, though I am under no obligation to do so, and there are considerable financial incentives for a researcher to never report a vulnerability at all
 
Siemens product CERT should be commended for being polite and helpful, and relatively quick with this fix. They acknowledged my work, and communicated clear timelines of when to expect a fix. This is precisely how they should go about communicating with us in the research community. It was at this point that I informed the good folks over at Siemens that I would verify the patch on Sep 12th. On the morning of the 12th, I tried to login to verify they patch they had provided, and found myself blocked from doing so. 
 
 
Should a firmware release with security implications only be downloadable in a forum that requires vetting and manual processing? Is it acceptable to end users that security patches are under export restriction?
 
Luckily these bans were lifted later that day and I was able to confirm the fixes. I would like to commend Siemens Product CERT the team for fixing these issues rapidly and with great professionalism. They communicated with me securely using GPG encrypted emails, set realistic timelines, and gave me feedback when those timelines didn’t work out. This leads me to a formal challenge based on their performance.
 
I challenge any other ICS vendors to match Siemens laudable response times and produce patches within 3 months for any externally submitted security vulnerabilities.
 
Stay tuned for part 2 where we release the simple Python script for authentication bypass which allows firmware and configuration upload and download.
 
*If you can crack this, and are interested in a job, please send IOActive your CV and the cleartext password used to create that credential. It is not hard, but it might take you a while….
INSIGHTS | October 21, 2013

NCSAM – Eireann Leverett on why magic is crucial

Late last week I had the pleasure of interviewing IOActive Labs CTO – Cesar Cerrudo on how he got into IT security. Today I am fortunate enough to have the pleasure of interviewing Eireann Leverett, a senior researcher for IOActive on this field and how magic played a part.

IOActive: How did you get into security?
 
Eireann: Actually, I was very slow to get security as an official title for a job, it was only really in the last few years. However, I always knew that’s how my mind was bent.
For example, everything I know about software security I learned from card tricks at 14. I know it seems ridiculous, but it’s true. Predicting session id’s from bad PRNGs is just like shuffle-tracking and card counting. Losing a card in the deck and finding it again is like controlling a pointer. If you understand the difference between a riffle shuffle and an overhand shuffle you understand list manipulation.
Privilege escalation is not unlike using peeks and forces in mentalism to corrupt assumptions about state. Cards led me to light maths and light crypto and zero-knowledge proofs.
From there I studied formally in Artificial Intelligence and Software Engineering in Edinburgh. The latter part took me into 5+ years Quality Assurance and Automated Testing, which I still believe is a very important place to breed security professionalism.
After that I did my Master’s at Cambridge and accepted a great offer from IOActive mixing research and penetration testing. Enough practicality so I’m not irrelevant to real world application, and enough theory & time to look out to the horizon.
IOActive: What do you find most exciting about security?
 
Eireann: The diversity of subjects. I will never know it all, and I love that it continually evolves. It is exciting to pit yourself against the designs of others, and work against malicious and deceptive people.
There are not many jobs in life where you get to pretend to be a bad guy. Deceptionists are not usually well regarded by society. However, in this industry having the mindset is very much rewarded.
There’s a quote from Houdini I once shared with our President and Founder, and we smile about it often. The quote is:
“Magic is the right way to do wrong.”
That’s what being an IOActive pirate is: the right way to do wrong. We make invisible badness visible, and explain it to those who need to understand business and process instead of worrying about the detail of the technology.
IOActive: What do you like to research, and why?
 
Eireann: Anything with a physical consequence. I don’t want to work in banking protecting other people’s money. My blood gets flowing when there are valves to open, current flowing, or bay doors to shut. In this sense, I’m kind of a redneck engineer.
There’s a more academic side to me as well though and my research passions. I like incident response and global co-operation. I took something Team Cymru & Dragon Research said to heart:
Security is more about protecting your neighbours.
If you don’t understand that your company can be compromised by the poor security of your support connections you’ve missed the point. That’s why we have dynamic information sharing and great projects like openIOC, BGPranking, honeynets, and organisations like FIRST. It doesn’t do you any good to secure only yourself. We must share to win, as a global society.
IOActive: What advice would you give to someone who would like to become a pentester/researcher?
 
Eireann: Curiosity and autodidacticism is the only way.
The root to hacking is understanding. To hack is to understand something better than it understands itself, and then nudge it to alter outputs. You won’t understand anything you listen to other people tell you about. So go do things, either on your own, or in formal education. Ultimately it doesn’t matter as long as you learn and understand, and can articulate what you learned to others.
INSIGHTS | September 11, 2012

Malware Doesn’t Care About Your Disclosure Policy, But You Better Have One Anyway

All over the world, things are changing in ICS security—we are now in the spotlight and the only way forward is, well, forward. Consequently, I’m doing more reading than ever to keep up with technical issues, global incidents, and frameworks and policies that will ensure the security of our future.

From a security researcher’s perspective, one exciting development is that .gov is starting to understand the need for disclosure in some cases. They have found that by giving companies lead time to implement fixes, they often get stonewalled for months or years. Yes, it sometimes takes years to fix specific ICS security issues, but that is no excuse for failing to contact the researcher and ICS-CERT with continually-updated timelines. This is well reflected in the document we are about to review.

The Common Industrial Control System Vulnerability Disclosure Framework was published a bit before BlackHat/Defcon/BSideLV, and I’ve just had some time to read it. The ICSJWG put this together and I would say that overall it is very informative.

For example, let’s start with the final (and most blogged about) quote of the Executive Summary:

“Inconsistent disclosure policies have also contributed to a public perception of disorganization within the ICS security community.”

I can’t disagree with that—failure to have a policy already has contributed to many late nights for engineers.

On Page 7, we see a clarification of vulnerabilities found during customer audits that is commendable:

“Under standard audit contracts, the results of the audit are confidential to the organization customer and any party that they choose to share those results with. This allows for information to be passed back to the vendor without violating the terms of the audit. The standard contract will also prevent the auditing company from being able to disclose any findings publically. It is important to note however, that it is not required for a customer to pass audit results on to a vendor unless explicitly noted in their contract or software license agreement.”

Is there a vendor who explicitly asks customers to report vulnerabilities in their license agreements? Why/why not?

On Page 9, Section 5 we find a dangerous claim, one that I would like to challenge as firmly and fairly as I can:

“Not disclosing an issue is not discussed; however it remains an option and may be appropriate in some scenarios.”

Very, well. I’m a reasonable guy whose even known to support responsible disclosure despite the fact it puts hand-cuffs on only the good guys. Being such a reasonable guy, I’m going to pretend I can accept the idea that a company selling industrial systems or devices might have a genuine reason to not disclose a security flaw to its customers. In the spirit of such a debate, I invite any vendor to comment on this blog post with a hypothetical scenario in which this is justified.

Hypothetically speaking: When is it appropriate to withhold vulnerabilities and not disclose them to your ICS customers?

While we’re at it, we also see the age-old disclosure always increases risk trope again, here:

“Public Disclosure does increase risk to customers, as any information disclosed about the vulnerability is available to malicious individuals as well as to legitimate customers. If a vulnerability is disclosed publically prior to a fix being made available, or prior to an available fix being deployed to all customers, malicious parties may be able to use that information to impact customer operations.”

Since I was bold enough to challenge all vendors to answer my question about when it is appropriate to remain silent, it’s only fair to tackle a thorny issue from the document myself. Imagine you have a serious security flaw without a fix. The argument goes that you shouldn’t disclose it publicly since that would increase the risk. However, what if the exploit were tightly constrained and detectable in 100% of cases? It seems clear that in this case, public disclosure gives the best chance for your customers to DETECT exploitation as opposed to waiting for the fix. Wouldn’t that DECREASE risk? Unfortunately, until you can measure both risk and the occurrence of 0-day vulnerabilities in the wild RELIABLY, this is all just conjecture.

There exists a common misconception in vulnerability management that only the vendor can protect the customer by fixing an issue, and that public disclosure always increases risk. With public disclosure, you widen the circle of critical and innovative eyes, and a third party might be able to mitigate where the vendor cannot—for example, by using one of their own proprietary technologies.

Say, for example, that a couple of ICS vendors had partnered with an Intrusion Detection and Prevention system company that is a known defender of industrial systems. They could then focus their early vulnerability analysis efforts on detecting and mitigating exploits on the wire reliably before they’re even fixed. This would reduce the number of days after zero the exploit can’t be detected and, to my thinking, that reduces the risk. I’m disappointed that—in the post-Stuxnet era—we continue to have ICS disclosure debates because the malware authors ultimately don’t even care. I can’t help but notice that recent ICS malware authors weren’t consulted about their “disclosure policies” and also didn’t choose to offer them.

As much as I love a lively debate, I wanted to commend the ICSJWG for having the patience to explain disclosure when the rest of us get tired.

INSIGHTS | June 28, 2012

Thoughts on FIRST Conference 2012

I recently had the opportunity to attend the FIRST Conference in Malta and meet Computer Emergency Response Teams from around the world. Some of these teams and I have been working together to reduce the internet exposure of Industrial Control Systems, and I met new teams who are interested in the data I share. For those of you who do not work with CERTs, FIRST is the glue that holds together the international collaborative efforts of these teams—they serve as both an organization that makes trusted introductions, and vets new teams or researchers (such as myself).

It was quite an honor to present a talk to this audience of 500 people from strong technical teams around the world. However, the purpose of this post is not my presentation, but rather to focus on all of the other great content that can be found in such forums. While it is impossible to mention all the presentations I saw in one blog post, I’d like to highlight a few.
A session from ENISA and RAND focused on the technical and legal barriers to international collaboration between National CERTS in Europe. I’m interested in this because during the process of sharing my research with various CERTs, I have come to understand they aren’t equal, they’re interested in different types of information, and they operate within different legal frameworks. For example, in some European countries an IP address is considered private information and will not be accepted in incident reports from other teams. Dr. Silvia Portesi and Neil Robinson covered a great wealth of this material type in their presentation and report, which can be found at the following location:
In the United Kingdom, this problem has been analyzed by Andrew Cormack, Chief Regulatory Advisor at Janet. If I recall correctly, our privacy model is far more usable in this respect  and Andrew explained it to me like this:
If an organization cannot handle private data to help protect privacy (which is part of its mission), then we are inhibiting the mission of the organization with our interpretation of the law.
This is relevant to any security researcher who works within incident response frameworks in Europe and who takes a global view of security problems.
Unfortunately, by attending this talk—which was directly relevant to my work—I had to miss a talk by Eldar Lillevik and Marie Moe of the NorCERT team. I had wanted to meet with them regarding some data I shared months ago while working in Norway. Luckily, I bumped into them later and they kindly shared the details I had missed; they also spent some of their valuable time helping me improve my own reporting capabilities for CERTs and correcting some of my misunderstandings. They are incredibly knowledgeable people, and I thank them for both their time and their patience with my questions.
Of course, I also met with the usual suspects in ICS/Smart Grid/SCADA security: ICS-CERT and Siemens. ICS-CERT was there to present on what has been an extraordinary year in ICS incident response. Of note, Siemens operates the only corporate incident response team in the ICS arena that’s devoted to their own products. We collectively shared information and renewed commitments to progress the ICS agenda in Incident Response by continuing international collaboration and research. I understand that GE-CIRT was there too, and apparently they presented on models of Incident Response.
Google Incident Response gave some excellent presentations on detecting and preventing data exfiltration, and network defense. This team impressed me greatly: they presented as technically-savvy, capable defenders who are actively pursuing new forensic techniques. They demonstrated clearly their operational maturity: no longer playing with “models,” they are committed to holistic operational security and aggressive defense.
Austrian CERT delivered a very good presentation on handling Critical Infrastructure Information Protection that focused on the Incident Response approach to critical infrastructure. This is a difficult area to work in because standard forensic approaches in some countries—such as seizing a server used in a crime—aren’t appropriate in control system environments. We met later to talk over dinner and I look forward to working with them again.
Finally, I performed a simple but important function of my own work, which comprises meeting people face-to-face and verifying their identities. This includes our mutually signing crypto-keys, which allows us to find and identify other trusted researchers in case of an emergency. Now that SCADA security is a global problem, I believe it’s incredibly important (and useful) to have contacts around the world with which IOActive already shares a secure channel