An advisory from DHS’s ICS CERT makes clear that ICS vendors are making progress toward fixing Heartbleed, but that customers face a long slog.
The good news about the Heartbleed vulnerability in OpenSSL is that most of the major sites that were found to be vulnerable to the flaw have been patched. As has been reported, all of the top 1,000 web sites have patched the flaw.
That’s good news for consumers, who can go back to checking Facebook or reading The Drudge Report without worrying that their web browsing will be vulnerable to snooping by a third party. The bad news: the vulnerability of high-profile web sites are just the tip of the iceberg or – more accurately – the head in front of a very long tail of vulnerable web sites and applications. Many of those applications and sites are among the systems that support critical infrastructure.
For evidence of that, look no further than the alert issued Thursday by the Department of Homeland Security’s Industrial Control System (ICS) Computer Emergency Readiness Team (CERT). The alert – an update to one issued last month – includes a list of 43 ICS applications that are known to be vulnerable to Heartbleed.
As of Thursday, just over half have patches available for the Heartbleed flaw, according to ICS CERT data. But that leaves twenty applications vulnerable, including industrial control products from major vendors like Siemens, Honeywell and Schneider Electric.
The bigger problem may be getting customers to figure out whether or not they need to patch. As ICS CERT points out in their advisory: ICS environments are notoriously difficult to audit. For one thing, ICS devices often respond poorly to any form of scanning. ICS-CERT noted that both active- and passive vulnerability scans are “dangerous when used in an ICS environment due to the sensitive nature of these devices.” Specifically: “when it is possible to scan the device, it is possible that device could be put into invalid state causing unexpected results and possible failure of safety safeguards,” ICS-CERT warned.
DHS advises customers to test products in an “isolated test laboratory, not in the production environment.” Customers who can’t do that (i.e. don’t have a spare model of every vulnerable ICS device to test offline) are advised to “contact their vendor” to determine whether their particular ICS product is vulnerable and if a patch is available.
That puts customers in something of a Catch-22, as one of the main reasons to scan at all is in instances where customers “implement end-of-life devices or their vendor is not communicating with them on this issue.” In other words: if you’ve got an unresponsive vendor, you should scan your own network. And if you can’t scan your own network, contact your (unresponsive) vendor. Huh???
The fragility of industrial control and SCADA systems has long been identified as a major problem in the industrial control space. As the ICS vulnerability researcher Billy Rios at Cylance has noted: many ICS products use a different, ICS-specific TCP/IP stack that makes them intolerant of scanning by tools developed for use on non-ICS products.
But that’s just one of a host of ‘differences’ between the industrial control world and traditional IT. ICS systems often fail to show up in scans for vulnerable systems using traditional vulnerability scanning tools.
Beyond that vendors frequently fail to inform customers of software vulnerabilities in a timely manner – or even to acknowledge that such vulnerabilities exist. That’s not the case with Heartbleed, given mainstream media attention to the problem. But its common with lower-visibility security holes. Researcher Cesar Cerrudo of IOActive found that out recently when reporting his discovery of a slew of serious, remotely exploitable holes in common traffic management systems used to control traffic signals and other critical infrastructure. The vendor in that case responded to his reports by saying that the vulnerabilities were by design – that customers wanted the products to operate that way.
Finally, as Rios notes, centralized patch management features are still uncommon in the ICS space. “Given the risks associated with patching ICS, there are no centralized, automated patching mechanisms. Nearly all ICS patches must be downloaded and applied manually. In some cases, the patch can only be installed by a certified technician from the vendor.”
What does this mean for organizations that rely on ICS software? I think it means a long slog to get Heartbleed patched in industrial settings, given the absence of patches and, even when patches exist, the difficulty identifying vulnerable systems and applying the patches to them. Heartbleed will continue to cause hearburn for industrial control system customers for months or years to come.
In the long term, ICS vendors need to re-architect products to operate more like other IT products: making them more robust and resilient, and allowing software updates that do not require the products to be taken off line or otherwise disabled.