What lessons can application developers take from the Heartbleed bug? How can we use this experience to make the Internet more secure? Unless you have been offline in a cave for the past few days, odds are that you’ve seen the many stories about the Heartbleed bug (and many more stories today) and, hopefully, have taken some action to update any sites you have that use the OpenSSL library. If you haven’t, then stop reading this post and go update your systems! (You can test if your sites are vulnerable using one of the Heartbleed test tools.) While you are at it, it would be a good time to change your passwords at services that were affected (after they have fixed their servers). There is an enormous list of the Alexa top 10000 out there, but sites like Mashable have summarized the major sites affected. (And periodically changing your password is just a general “best practice”, so even if the site was not affected, why not spend a few minutes to make the changes?)
Client Applications Need Updating, Too
For application developers, though, it is also important to update any client applications you may have that use the OpenSSL libraries to implement TLS/SSL connections. While most of the attention has been focused on how attackers can gain access to information stored on servers, it is also true that a malicious site could harvest random blocks of memory from clients visiting that site. There is even demonstration code that lets you test this with your clients. Now, granted, for this attack to work an attacker would need to set up a malicious site and get you to visit the site through, for instance, a phishing email or link shared through social media. The attacker could then send malformed heartbeat messages to your vulnerable client in an attempt to read random blocks of memory… which then may or may not have any useful information in them.
Again, the path for an attacker to actually exploit this would be a bit complex, but you definitely should test any client applications you have that rely on any OpenSSL libraries.
With all that said, since we have started this “TLS For Applications” topic here are on Deploy360, what are some of the important lessons we can take away from this experience? Here are a few I see coming out of this – I’d love to hear the lessons you take from all of this in the comments.
Security Testing Is Critical
It turns out that this was an incredibly trivial coding error. As Sean Cassidy points out in his excellent Diagnosis of the OpenSSL Heartbleed Bug, the issue boils down to this:
What if the requester didn’t actually supply payload bytes, like she said she did? What if pl really is only one byte? Then the read from memcpy is going to read whatever memory was near the SSLv3 record and within the same process.
There was no checking on the input and this allowed reading from other parts of the computer’s memory. As Cassidy later writes about the fix:
This does two things: the first check stops zero-length heartbeats. The second check checks to make sure that the actual record length is sufficiently long. That’s it.
Today’s XKCD comic shows this all in an even simpler explanation.
This is the kind of trivial mistake that probably every developer has made at some point of his or her life. I am sure that if I were to go back through many lines of code in my past I’d find cases where I didn’t do the appropriate input or boundary testing. It highlights the importance of doing security testing – and of setting up security unit tests that are just done as part of the ongoing testing of the application. It also highlights the need for ongoing security audits, and for reviewers of code submissions to also be testing for security weaknesses. But again, this is a common type of error that probably every developer has made. You need testing to catch things like this.
In this instance it just happens that the mistake was in a piece of software that has now become critical for much of the Internet! Which leads to a second lesson…
Having A Rapid Upgrade Path/Plan Is Important
As people learned about this bug earlier this week there has been a massive push to upgrade software all across the Internet. Which raises the question: how easy is it for your users to upgrade their software in a high priority situation such as this?
In many cases, it may be quite easy for users to install an update either from some kind of updated package or a download from an online application store. In other cases, it may be extremely difficult to get updates out there. In the midst of all this I read somewhere that many “home routers” may be vulnerable to this bug. Given that these are often something people buy at their local electronics store, plug in, and pretty much forget… the odds of them getting updated any time soon are pretty slim.
Do you have a mechanism whereby people can rapidly deploy critical security fixes?
UPDATE: A ZDNet post notes that both Cisco and Juniper have issued update statements for some of their networking products. I expect other major vendors to follow soon.
Marketing Is Important To Getting Fixes Deployed
Finally, Patrick McKenzie had a great post out titled “What Heartbleed Can Teach The OSS Community About Marketing” that nicely hits on key elements of why we’re seeing so much attention to this – and why we are seeing fixes deployed. He mentions the value of:
- Using a memorable name (“Heartbleed” vs “CVE-2014-0160″)
- Clear writing
- A dedicated web presence with an easy URL to share
- A visual identity that can be widely re-used
His article is well worth reading for more details. His conclusion includes this paragraph that hit home for me (my emphasis added):
Given the importance of this, we owe the world as responsible professionals to not just produce the engineering artifacts which will correct the problem, but to advocate for their immediate adoption successfully. If we get an A for Good Effort but do not actually achieve adoption because we stick to our usual “Put up an obtuse notice on a server in the middle of nowhere” game plan, the adversaries win. The engineering reality of their compromises cannot be thwarted by effort or the feeling of self-righteousness we get by not getting our hands dirty with marketing, it can only be thwarted by successfully patched systems.
Exactly!
We need to make it easy for people to deploy our technologies – and our updates to those technologies. (Sound like a familiar theme?)
What other lessons have you taken from this Heartbleed bug? What else should application developers be thinking about to make TLS/SSL usage more secure?
Please do leave a comment here or on social media sites where this article is posted. (And if you’re interested in helping us get more documentation out to help app developers with TLS/SSL, how about checking out our content roadmap for the TLS area? What other items should we include? Do you know of existing documents we should consider pointing to? Interested in writing some documents? Please do let us know.)
P.S. There’s now a post out about the process the Codenomicon team went through in disclosing the bug that is worth reading.
