Tim Mackey works within the Synopsys Software Integrity Group as a Senior Technology Evangelist. Within this role, he engages with various technical communities to understand how to best solve application security problems. He specializes in container security, virtualization, cloud technologies, distributed systems engineering, mission-critical engineering, performance monitoring, and large-scale data center operations. Tim takes the lessons learned from these activities and delivers talks globally at conferences such as RSA, OSCON, Open Source Summit, KubeCon, Interop, CA World, Container World, DevSecCon, DevOps Days, and the IoT Summit. He is also an O'Reilly Media published author.

Containerized applications are gaining in popularity, and for good reason. They offer multiple benefits for organizations including portability, ease of deployment, and strong controls over resource consumption. This is accomplished through the addition of two new layers within the application delivery stack — a container runtime and associated orchestration solution.  These changes require organizations to adapt their security strategies for a more dynamic production environment.

A primary goal for any organization adopting a new technology is to reduce security risks in their application delivery stack. Doing so requires that existing best practices be reassessed in light of the capabilities of what the new technology provides. Organizations subject to strict regulations, or with otherwise risk-averse leadership, are often wary of the challenges of securing new technologies in production and wait for more mainstream adoption to vet best practices.

When designing an effective container security strategy, organizations must first understand the risks that attackers could exploit that may potentially lead to a data leak. Here are some key challenges that need to be considered in advance of any attempts to secure containers:

  • Isolation. Container isolation differs greatly from virtual machine isolation. In virtual machine (VM) systems, hypervisor isolation limits the ability for an attacker to move laterally within an application stack in the event that an application is breached. However, containerized applications don’t require hypervisors; rather, they share elements of the host operating system’s kernel.
  • Runtime complexities. The dynamic nature of containers introduces runtime complexities that application deployment teams need to be aware of and manage. It also makes monitoring for malicious behavior and unauthorized access more complex than in a traditional IT environment as containerized applications often have different resource requests that are shared at the host server level. To overcome this difficulty, IT operations and security teams should become partners with their development teams. Information sharing among them will help them understand expected behavior for the application and differentiate that from the unexpected. Container orchestration systems like Kubernetes are designed to quickly provision replicated instances of a container image. Containerized applications consist of one or more container images coupled to form the functionality required by the application. This then leads to the topic of application scalability — a function of the number of specific container images deployed at any given point. When a new feature is ready for deployment, the application owner creates an update strategy to ensure any existing users of the application aren’t affected by the update. This update strategy defines the percentage of images to roll forward with the update, as well as how a rollback might occur if errors are discovered.
  • Patch management. Most container application images are created from base images — images which provide the core operating environment for the application. Application container images combine a base image with application-specific elements, such as frameworks, runtimes, and the applications themselves. Each element, including the base image, is a layer within the image. The contents of these layers can harbor software vulnerabilities, which makes them an attractive attack surface if unpatched. Security testing of containerized applications must, therefore, address vulnerabilities across the layers of the image to determine the resultant vulnerability impact. It’s also not enough to periodically scan container images, given that some clusters can exceed 10,000 running containers. Organizations must continue to monitor for newly disclosed vulnerabilities in components within the image — regardless of layer and without negatively affecting performance.

While containers present unique security challenges, with the right tools, practices, and strategies it is possible to overcome them.

Organizations often conduct manual security reviews when experimenting with new technologies like containerization. According to a Forrester study, 43% of container application owners perform regular security audits of their clusters. Audits may consist of tracking components with known vulnerabilities on spreadsheets or manually testing configurations. It also takes time to determine which solutions are effective for container environments, which is why a manual process only works well for small, proof-of-concept deployments. It is therefore also important to have a dedicated security solution designed to scale up and down with container clusters.

A substantial benefit of containerization is that container images created on one machine can run anywhere, including within the technology they are disrupting: VMs. Some organizations run containerized applications within VMs to isolate containers using hypervisors. They then create application affinity based on data types within the VMs to prevent attackers from moving laterally within the application stack to access data from other applications. While this strategy can limit the severity of an attack, it will not prevent the attack from happening.

Let’s not forget runtime security solutions, which can detect and block malicious activity in running containers in real time. By monitoring network calls to the host and attempts to log into containers, these solutions build behavioral models of every application in an environment. Those models establish what activities are normal, so when something is abnormal (or potentially malicious) it is detected. Of course, when defining a runtime security strategy, it’s important to ask questions such as,  why is it possible to log into a running container in the first place? Any security paradigms present within new technologies should be utilized for best effect, including identifying the minimum runtime requirements for applications.

In contrast to runtime security, container patch management is proactive; a strategy to address vulnerabilities and mitigate attacks before they happen. In order to secure their containers, organizations must know what is present within the container image. With most container images originating with base images from public third-party sources, it is critical to know the complete composition of an image. Considering that most container applications are Linux based, this starts with an effective open source governance process designed to recognizing latent issues within all images.

The 2019 Open Source Security and Risk Analysis (OSSRA) report found open source components in 96% of audited codebases, with the average codebase made of 60% open source (up from 57% in 2017). Additionally, 60% of the codebases audited in 2018 contained at least one open source vulnerability—still significant, but far better than the 78% observed in 2017. Knowing this, no organization should expect to track all its open source components and associated vulnerabilities manually.

Existing patch management strategies may also increase risk when applied to containers. Effectively, the legacy patch model increases the attack surface and reduces application availability as the applications scale. A far more effective model is to treat a patch like an application update and update the container image, which would then be deployed using an update strategy appropriate to the container orchestration solution. The net result would be a more secure deployment paradigm which takes advantage of the inherent benefits of an immutable container image.

As application deployment using container technologies grows in production environments, security processes must scale with them. To get a full picture of the risks in a container cluster, organizations must automate the process of identifying, mitigating, and alerting on any risks — regardless of source.

Since no single tool will completely secure container clusters, organizations should look for container security solutions that are integrated with their chosen orchestration solution. This model benefits from Defense in Depth — using different techniques to address some of the risks posed by containerization.

Container runtime security solutions can help teams monitor and prevent unauthorized calls to the host, limiting the scope of breaches. And vulnerability management solutions can help organizations proactively reduce risk, automatically identifying known vulnerabilities and removing them from their clusters, which will reduce potential attack vectors at scale. Combining these techniques creates a robust security strategy designed to monitor, manage, and remediate security issues when deploying containerized applications at scale.

Feature image via Pixabay.