What should you do to secure your containers? Liz Rice, author of the book Container Security: Fundamental Technology Concepts that Protect Containerized Applications & VP of open source engineering at Aqua Security, and Eoin Woods, CTO at Endava, explore what containers are, what the implications of a shared kernel are and how to assess potential security risks that could affect your deployments. Learn best practices and better understand how containers work in this Book Club interview.
Preben Thoro: Thank you for joining us today.
Eoin Woods: I’m Eoin Woods, CTO at the software engineering services firm Endava and I’m very pleased to be joined by Liz Rice, VP of Open Source at Aqua, the cloud security tools security company, who is author of the excellent book “Container Security”. Welcome, Liz!
Liz Rice: A pleasure to be here Eoin.
How did you come to write a book on container security?
Eoin Woods: So, Liz, I thought we'd just start with perhaps the most obvious question, which is, how did you come to write a book on container security? It's a really important topic, but it's quite a specialist one.
Liz Rice: Yes. So I've been working with containers for quite a few years now and obviously, they've exploded in popularity over the last four or five years. I think we've gone from like, "What even is this Docker thing," to lots of developers, there's a huge proportion, using containers. And I've been working in container security now for about four years. Yeah, I've come from the developer angle, and I've always been interested in kind of how things work, what's happening under the covers. With my sort of security perspective, it's like, "The way that a container works, how do that present issues for security? How does it create attack surfaces?"
I've been doing talks and writing blog posts and creating content about containers and container security for quite a while. Then when O'Reilly approached and asked if I had an idea for a book, I kind of knew that well, container security is what I've been working in and what I know about, and it was nice to have the opportunity to dig in more detail than you can necessarily cover in a 40-minute conference talk. In a book, you can dive into the details.
How does your position at CNCF Technical Oversight Committee relate to your work on security?
Eoin Woods: Which you do. There's a lot of detail in the book to help people understand it, which I mean, it's a great aspect of the book. And I noticed, of course, your day job's at Aqua, in the area of cloud security, but you've also got a CNCF affiliation, how does that relate to container security?
Liz Rice: Yes. So, I am currently the chair of the technical oversight committee for the CNCF, which in case anyone doesn't know, it's the organization — part of the Linux Foundation — that owns Kubernetes, Prometheus, a whole host of cloud native projects. With the TOC, we're the group that assesses whether projects are cloud native and should join the foundation and at what level of like sandbox incubation or graduation. So that gives me a really interesting vantage point over the world of cloud native in general, and I get to meet and talk to some interesting people.
Security is just one part of the cloud native landscape, but it is a very important part. So it's a privilege to be getting that view across the broader landscape, not just security.
Eoin Woods: Yes, there's a lot of hard work probably in your spare time I suspect.
Liz Rice: Spare time? What's that?
What is the essence of a container?
Eoin Woods: It's rather nice to have somebody leading a kind of technical committee who's focused on security. That's really good. I was wondering because clearly, you've got quite a broad view of containers. One of the nice things in the book, I just love the way that you sort of build a container at the shell prompt. I think there are lots of us who have come to containers, and we got Docker and we thought this is this magical packaging technology and seems to keep everything safe, and then we weren't quite sure how safe it was, and then you wrote a book on that. Can you just explain for someone who's just used containers, what the essence of a container really is?
Liz Rice: I think if your experience of containers is running them, you know, Docker run and maybe executing into a container, there is this sort of tendency to think of them as virtual machines. Like, you used to hear this phrase, lightweight virtual machine. It's so misleading because all the containers on a given machine share the same kernel, which is the fundamental difference between a container and a virtual machine. Virtual machines have their own kernel.
And really what a container is, it's a process or maybe several processes running under Linux, and that process is isolated from the other processes. And it uses mechanisms called namespaces and cgroups and it has its own view of the file system. But really, from the host perspective, it's just another process. It's really nothing special. And that has huge implications for security because the isolation is, it's only so much, it's not the same strength of isolation as you would get from virtual machines, largely due to that shared kernel.
Eoin Woods: That's interesting. So, the VMs kind of thing with an operating system kernel in it, the container isn't, you sort of hinted a bit there in that question: they are different from a security perspective. Well, from the perspective of people who use them as opposed to people who are deeply embedded in building them, how would you compare and contrast them specifically, from a security perspective? What does that shared kernel mean in terms of security things you need to be aware of?
Key takeaway no. 1
In its essence, a container is a process running under Linux which is isolated from the other processes. This has deep implications for security.
What does that shared kernel mean in terms of security things you need to be aware of?
Liz Rice: So the main consequence of that is what happens when things go wrong. If you do manage to escape a container, you're a bad actor, and you've managed to compromise a container. If there is a way of escaping onto the host, you can potentially escape through the kernel, if there's a vulnerability in the kernel, and we have seen vulnerabilities in the kernel. Or if there's a vulnerability in the container runtime, it's possible to escape from the container onto the host.
I mean, it's not like this isn't ever possible in virtual machines. There have been security issues in virtual machines as well. But the fact that you have this shared kernel makes it just so much more likely that if something goes wrong, you're gonna get from the container onto that host machine where all of the containers are running. And once you're on the host, particularly if you're root, you've got access to everything that’s there.
Eoin Woods: Most containers, container systems I've worked with, they are root, right? I mean, that's one of the questions.
Liz Rice: This is one of the kind of really fundamental things that I really want people to improve about their container security: you're running as root by default today, at least. There are exceptions. If you're using Podman, that doesn't run as root by default. If you're using OpenShift, that doesn't run as root by default. But for most people's experience running with Docker, running containers on Kubernetes, unless they've specified a user by default, they're root. And as I've shown in the book, and as I've shown in a few talks before, that root identity is exactly the same in the container as it is on the host. So if you do somehow escape from the container, you're root on the host.
And I guess another thing that's quite a big contrast between containers and virtual machines is you can escape the container through just misconfiguration. It doesn't necessarily require an actual vulnerability in the kernel or the runtime. You can, for example — if you mount the whole host file system, there's nothing to stop you from doing this. — you can mount root on the host into a container. Running as root inside a container with access to the full host file system, the world is your oyster as an attacker.
Key takeaway no. 2
A big contrast between containers and virtual machines: you can escape the container through misconfiguration.
Software supply chain security with containers
Eoin Woods: That can only sound like a bad thing. Interesting. Thanks. And, I mean, one thing you talk about a bit in the book, which I think has got a lot of... You were just talking about parallels with other previous technologies, which also is a concern of the places, but strikes me as really important with containers, is software supply chain security. So, that's the posh name for pulling in all your dependencies. This is where it is with things like Maven and Node, and you import one package and you seem to end up with half the open source universe, and you don't know much about it.
Containers have always worried me a little bit more. I don't even really know why. Maybe it's emotional or maybe I don't know enough about them. But it seems like you're pulling down really complicated bits of software, which you probably aren't very familiar with and they're all being stacked in this kind of integrated stack, and you need all of it to make it work. I mean, how do you even go about thinking about software supply chain security with containers?
Liz Rice: I think the reason why it feels more complicated in the world of containers is because every single one of your containers is running with its own set of dependencies. One of the wonderful features of containers is you can package up an application and all its dependencies into one image and chip that image around, so every image has its full set of dependencies. So every image could be different. And you've got to check every one of those images, because they all might have... You've got this forest of images and somewhere in there maybe there's a vulnerability and it can seem quite daunting, particularly if you're sort of used to a traditional security process where you SSH into a machine and apply patches. You can't really do that in a containerized environment, particularly if those containers are coming up and down pretty frequently. The average lifetime of a container in production is around about a day, so imagine having to patch 1,000s of containers, you just can't do it manually.
So automation is really your friend. Like everything else in the cloud native world, the more we can automate things, the less human interaction you have, the less chance there is that somebody's going to make a mistake. A lot of security issues are not people behaving badly deliberately, they're just slip ups. People leaving things open inadvertently. So, automation is your friend. Vulnerability scanning is particularly important for the supply chain.
Eoin Woods: That point about people who don't do daft things deliberately is important. I think that's a really important point, isn't it? That there seems to be so much technology that from a security perspective, it's just a minefield for people to kind of wade through. I always describe in my security talks that if you had to create an attack vector, you'd start with a web browser. It's such a rich environment to build attacks in. Actually, as soon as you look at this, the security person goes, "Yeah, that's a huge security problem." How do you think about containers? I mean, do you think there are things that are in configuration today that maybe should be hard to change, or do you think there are particular vulnerabilities in the way containers are configured that could be improved, I mean, especially from a CNCF perspective?
Key takeaway no. 3
Automation is your friend. The more we can automate things, the less human interaction we have, and the less chance there is that somebody's going to make a mistake.
Security concerns with containers
Liz Rice: There are all kinds of aspects of how you run containers and how you run your overall orchestration system and these days, it's probably Kubernetes. Over time, the settings for Kubernetes have gotten better by default. So, gone are the days when by default, you would leave an administrative dashboard open to the internet by default, which was the thing. And fortunately, if you take the standard install tools now, you will get things like secure API ports rather than just allowing unauthorized access, and so the default settings have improved a lot over the past couple of years or so.
There's still a lot of room for improvement. One thing that I'm quite excited about right now is rootless containers, which is, I mean, as we were talking before about how by default, you typically run as root. Rootless containers mean that although from the containers' perspective, you appear to be root, it's not actually root on the host, and that's a huge improvement. And, in fact, yesterday, as the date of recording, yesterday, Docker released their latest update, which has moved support for rootless out of experimental.
There's still some work to go in the Kubernetes world to bring rootless. There are some experimental demonstrations of running rootless, but it's not really the production sort of GA right now. But I'm hopeful that we're not too far away from a future where, by default, we don't run it as root, and that will be a big step forward.
Eoin Woods: Yes, that sounds like great progress. Good to be able to influence that kind of thing. Because one of the technologies you've mentioned a couple of times is Kubernetes. And I think in many people's minds, you've talked to many software developers, containers and Kubernetes, Kubernetes and containers, that they're really kind of almost the same thing. And in the book, you don't spend a lot of time talking about the orchestration technology. What complications, do you think, Kubernetes brings to the container environment from a security perspective, because you alluded to the fact that it certainly doesn't make things easier?
Liz Rice: So one of the reasons why I didn't talk about Kubernetes security in the book is partly because it wouldn't have been two or three times the size, and partly because it's changing more quickly. I think that the fundamentals around how containers work are pretty well understood, whereas I think there are lots of aspects of Kubernetes and Kubernetes security and the complementary products that sit around Kubernetes like registries where there are still lots of improvements and lots of great innovation going on, not that there isn't innovation going on in the container space. But I wanted the book to not be out of date in six months or a year's time, and I think that Kubernetes security is moving sufficiently quickly that there would be a danger of things being a little bit aged after a year or two.
Eoin Woods: Where would you recommend people go to find out more about Kubernetes security aspects and how to track that because it is moving quite quickly, isn't it?
Liz Rice: It is moving quickly. In fact, I co-authored an O'Reilly report with Michael Hausenblas who's now at Amazon, and we did that a couple of years ago. I definitely feel that that report is kind of due a refresh. It's got some sensible things in it, but it's there are things I would like to update in that book. So in the absence of that, I would say the Kubernetes documentation itself has got better and there are people who are writing sections in Kubernetes docs about security aspects. So things like if you want to make sure you've got secure connections between your different Kubernetes components, that's all documented now. There's quite a lot of best practices documented there. There's also a Slack channel on the Kubernetes Slack dedicated to security.
I think one of the really nice things about Kubernetes in general, the CNCF in general, the whole community is very welcoming. If you do have a question, whether it's about security or anything else, coming into the Slack channel there will be people to help. Asking on a mailing list, there will be people to help. So, I definitely encourage people to kind of embrace the community. It's one thing to read a book, but you can also find people who know this stuff, and they're happy to help.
Eoin Woods: That's terrific to hear, yes, because it makes such a difference when you connect with a human being who can actually point out the bit of the book you've misunderstood. So that's really important. Thank you.
Key takeaway no. 4
Rootless containers: you appear to be root, but it is not really root on the host.
Who is responsible for container security?
Eoin Woods: A lot of these technologies are often sort of bundled into a broad area of people called DevOps, which is terribly undefined word. It drives me mad. But if we just take it as meaning smoothing the path from development into operations, there are, even if I'm a great fan of integrated teams, we haven't got a silo.
And a former colleague of mine had a terrific phrase for it. He said, "The desert of responsibility in between Dev and Ops, and you're trying to take that away, you do still have specialists in the teams, container security." I know the simple answer. The answer I always give my teams is security is everybody's problem. But when it comes to container security, where are the key responsibilities? What kinds of people do you need involved in your project that you're gonna get the right attention on it?
Liz Rice: Yes. You've used the phrase DevOps and people also use the word DevSecOps. I think that just goes to illustrate that actually, there are lots and lots of skills involved. And people like to pretend we can just sort of mash them together and it all be magic. But actually, that's a big cultural change.
Eoin Woods: Which won't change anyway.
Liz Rice: So I think it reflects the fact that you're going to need people who understand how to operate your cluster, your containers, how they're dealing with deployments and rollouts and scaling and all that side of things.
You've got your developers who understand the application, and you've got security specialists who understand things like, well, if we do have a particular vulnerability, what's the impact of that? There's lots of kind of jargon in the security world that can be a bit impenetrable, and you need people to know how to navigate that and really understand if we find a particular security flaw, maybe we're scanning our images, we find, "Here's this terrible, critical vulnerability. What should we do?" And you need people who understand how to investigate what that really means and can ask the right questions of the developers and the operators to figure out what the overall response should be to a particular vulnerability or an incident even.
You need all those skills and how those are organized in different organizations and how they're brought together into different teams. I think that's something that our whole industry is working on and figuring out how to deal with that effectively.
Eoin Woods: I agree. I don't think anyone's got an easy answer to that. But it's important we all, at least, agree where the goal is and sort of move in that direction.
Liz Rice: There's definitely an aspect of tooling where security tooling in particular, where I would say most security tooling is really at the level of telling people that issues exist. If we think about things like runtime security, the technology exists, there are some really great tools. I mean, I'm working for a vendor, so I would probably think that. But there are some great runtime security tooling that can prevent your containers from doing untoward actions.
But in reality, most people are not using that yet. They're mostly in the world of kind of even trying to identify what the issues are, let alone how to react to them, how to manage them, how to prioritize different security concerns. So, I think that the security industry and the container security industry is trying to level up people's understanding of security tooling, and things like how do you deal with these issues still require quite a lot of human expertise right now.