A critical Kubernetes security vulnerability that was patched this week affects a wide swath of container orchestration users.
CVE-2018-1002105 is a flaw that enables attackers to escalate privileges via the widely used Kubernetes API server. With Kubernetes reigning as the de facto standard for container orchestration in enterprise IT, the flaw poses a potentially huge risk for the IT industry as a whole. The Kubernetes open source community issued a patch upstream for version 1.10 and above, and it was already applied to hosted Kubernetes services, such as Google Kubernetes Engine and Red Hat OpenShift Online. However, users with on-premises deployments or self-hosted Kubernetes clusters in the cloud will have to roll out the patch themselves.
“There have been other Kubernetes flaws in the past, but this is definitely the worst,” said Gary Chen, analyst at IDC. “This is always the worry with centralized control planes — if someone hacks it, they can get access to everything else.”
Darren Shepherd, co-founder and architect at Rancher, reported the vulnerability to the Kubernetes security team last week. The flaw was patched this week and the flaw and patch were simultaneously disclosed, after a week-long embargo. Kubernetes 1.10 came out in late March 2018, so the vulnerability has been in the wild for about nine months. Kubernetes contributors, such as Red Hat, said that’s a short window compared with other critical open source vulnerabilities like Heartbleed, Spectre and Meltdown, which were exploitable for years.
“It’s always terrible to find a severe problem, but the Kubernetes team took this very seriously and issued a fast response,” said Chris Robinson, product security assurance manager at Red Hat. “That’s a trend we’ve seen with CVE embargoes in open source, in general, the last five years — on average, they’ve gone from 14 days down to about 10.”
Tech vulnerability exposes market disparities
However, some IT pros said that only large IT teams with packaged Kubernetes implementations can keep up with quarterly updates to the orchestration tool, and that this Kubernetes security vulnerability presents an additional roadblock to timely upgrades past version 1.8.
“Kubernetes may be the standard overall, but within it there are multiple standards to pick from between versions 1.8 and 1.12,” said Rick Moss, infrastructure operations engineer at MailChannels, an email service provider in Vancouver, B.C. “There are many companies facing the 1.10-plus environment, with its requirements for TLS [transport layer security] and namespace controls, slowly because the [new requirements] will make development and deployment cycles more complex.”
Worse, many enterprises freeze production updates and deployments between early December and mid-January, making the requirement to patch this Kubernetes security vulnerability particularly ill-timed in Moss’s view.
“If you want to ruin Christmas, a major CVE in early December is pretty much how you do it,” Moss said. “But, if you need to deploy this patch, not doing so could really change your Christmas plans, too.”
Companies with big IT teams dedicated to infrastructure support who have the money to buy commercially supported Kubernetes packages will have an easier time than lean DevOps teams that may not be well-versed in infrastructure security. This outcome could well amount to an argument in favor of packaged distributions over upstream code, but the reality is that cost is king in the open source software world, Moss said. He predicted that companies without the resources to respond to this kind of Kubernetes security vulnerability will instead delay upgrades or simply ignore the problem.
Rick Mossinfrastructure operations engineer, MailChannels
“It’s not that we haven’t seen vulnerabilities like this before, but this is a new problem based on the religion of DevOps and small teams,” Moss said. “The truth is that large-scale infrastructure deployments need a dedicated team to monitor the back end and keep it up to date, and smaller teams can get stuck.”
Some upstream adherents said they won’t have a problem applying the patch, but that the possibility of future Kubernetes security vulnerabilities affected their operating model with the open source distribution.
“We have application-specific clusters, and those are only accessible by the users have full admin [access] to those clusters,” said Adam Brown, co-founder of Mux Inc., a video streaming startup in San Francisco that serves media giants such as CBS and PBS. “We expect this to change in the future and bugs like this are pretty terrifying, but at the moment, it doesn’t appear to be exploitable in our clusters.”
IT vendors rush to patch Kubernetes security flaw
IDC’s Chen predicted that this Kubernetes security vulnerability and others that could arise in the future will push enterprises to invest in application security tools and take another look at packaged Kubernetes distributions from industry vendors.
“This is going to be something enterprises look at, in terms of vendors: Which ones responded the fastest and provided the best information?” Chen said.
As scary as this vulnerability was, the fact that Kubernetes’ maintainers found and quickly patched it is a point in favor of Kubernetes maturity overall, Chen added.