So many everyday items in the developed world are now connected to the Internet, often inexplicably. It adds another layer of potential technology failure that for personal appliances can be something of an amusing annoyance: blinds that won’t open, microwaves that don’t adjust for time changes, refrigerators that need firmware updates.
But in the enterprise, when Internet of Things devices fail, it’s no Twitter-thread joke. Factory assembly lines grind to a halt. Heart-rate monitors in hospitals switch offline. Elementary school smart boards go dark.
Smart device failures are an increasing risk in the enterprise world, and not just because of the oft-discussed security worries. It’s because some of these devices’ root certificates — necessary for them to connect to the Internet securely — are expiring.
“Devices need to know what to trust, so the root certificate is built into the device as an authentication tool,” explains Scott Helme, a security researcher who has written extensively about the root certificate expiration issue. “Once the device is in the wild it tries to call ‘home’ — an API or manufacturer’s server — and it checks against this root certificate to say, ‘Yes, I’m connecting to this correct secure thing.’ Essentially [a root certificate is] a trust anchor, a frame of reference for the device to know what it’s speaking to.”
In practice this authentication is like a web or a chain. Certificate authorities (CAs) issue all kinds of digital certificates, and the entities “talk” to each other, sometimes with multiple levels. But the first and most core link of this chain is always the root certificate. Without it, none of the levels above could make the connections possible. So if a root certificate stops working, the device can’t authenticate the connection and won’t link to the Internet.
Here’s the problem: The concept of the encrypted Web developed around 2000 — and root certificates tend to be valid for about 20 to 25 years. In 2022, then, we’re smack in the middle of that expiration period.
The CAs have issued plenty of new root certificates in the last two-plus decades, of course, well ahead of expirations. That works well in the personal device world, where most people frequently upgrade to new phones and click to update their laptops, so they would have these newer certs. But in the enterprise, it can be far more challenging or even impossible to update a device — and in sectors like manufacturing, machines may indeed still be on the factory floor 20 to 25 years later.
Without an Internet connection, “these devices aren’t worth a thing,” says Kevin Bocek, vice president of security strategy and threat intelligence at Venafi, provider of machine identity management services. “They essentially become bricks [when their root certs expire]: They can’t trust the cloud anymore, can’t take commands, can’t send data, can’t take software updates. That’s a real risk, particularly if you’re a manufacturer or an operator of some kind.”
A Warning Shot
The risk isn’t theoretical. On September 30, a root certificate issued by the massive CA Let’s Encrypt expired — and several services across the Internet broke. The expiration wasn’t a surprise, as Let’s Encrypt had long been warning its customers to update to a new cert.
Still, Helme wrote in a blog post 10 days before the expiration, “I’m betting a few things will probably break on that day.” He was right. Some services from Cisco, Google, Palo Alto, QuickBooks, Fortinet, Auth0, and many more companies failed.
“And the weird thing about that,” Helme tells Dark Reading, “is that the places using Let’s Encrypt are by definition very modern — you can’t just go to their website and pay your $10 and download your certificate by hand. It has to be done by a machine or via their API. These users were advanced, and it was still a really big problem. So what happens when we see [expirations] from the more legacy CAs that have these big enterprise customers? Surely the knock-on effect will be larger.”
The Path Forward
But with some changes, that knock-on effect doesn’t have to happen, says Venafi’s Bocek, who views the challenge as one of knowledge and chain of command — so he sees solutions in both awareness and early collaboration.
“I’m really excited when I see chief security officers and their teams getting involved on the manufacturer and developer level,” Bocek says. “The question is not just, ‘Can we develop something that is safe?’ but ‘Can we continue to operate it?’ There’s often a shared responsibility of operation on these high-value connected devices, so we need to be clear on how we’re going to handle that as a business.”
Similar conversations are happening in the infrastructure sector, says Marty Edwards, deputy CTO for operational technology and IoT at Tenable. He’s an industrial engineer by trade who has worked with utility companies and the US Department of Homeland Security.
“Quite frankly, in the industrial space with utilities and factories, any event that leads to a production outage or loss is concerning,” Edwards says. “So in these specialty circles the engineers and developers are certainly looking at the impacts [of expiring root certificates] and how we can fix them.”
Though Edwards stresses he’s “optimistic” about those conversations and the push for cybersecurity considerations during the procurement process, he believes more regulatory oversight is also needed.
“Something like a baseline standard of care that perhaps includes language on how to maintain the integrity of a certificate system,” Edwards says. “There’s been discussions between various standards groups and governments about traceability for mission-critical devices, for example.”
As for Helme, he’d love to see enterprise machines set for updates in a way that’s realistic and not arduous for the user or the manufacturer — a new certificate issued and update downloaded every five years, perhaps. But manufacturers won’t be incentivized to do that unless enterprise customers push for it, he notes.
“In general, I do think that this is something the industry needs to fix,” Edwards agrees. “The good news is most of these challenges aren’t necessarily technological. It’s more about knowing how it all works, and getting the right people and procedures in place.”
