Just yesterday, Microsoft disclosed a new (and yet to be clarified) Azure Container Instance vulnerability — https://msrc-blog.microsoft.com/2021/09/08/coordinated-disclosure-of-vulnerability-in-azure-container-instances-service/. From the information shared in the disclosure, it seems like the flaw was fixed on 31st August 2021 and customers were notified by Microsoft. The details are yet to come out so please check back for updates to this blog.
What is ACI?
ACI is a Container as a Service (CaaS) offering in Azure (similar to AWS Fargate and Google Cloud Run). It provides an option to run containers serverlessly. Customers simply schedule a single container image or a group of containers (called “container groups”; similar to the concept of a pod in Kubernetes) to run on platform managed infrastructure — No need to deal with VMs or orchestrators.
ACI can run both Linux and Windows containers. A lot of the customer deployments will be Linux as Windows containers in ACI lacks some important capabilities like container groups, volume mounting and private network integration.
Apart from running single containers or container groups, ACI can also be leveraged as Azure Kubernetes Service (AKS) virtual nodes. This uses a “virtual kubelet” implementation to provisions pods inside ACI from AKS.
REFERENCE: For sample use cases for ACI, review this link.
What is the vulnerability that was disclosed?
The “how” is yet clarified yet but from the recommendation in the disclosure, it seems like it potentially gives an attacker access to sensitive information mounted in a container instance!
Some urgent actions!
Before you start to investigate ACI and other Azure logs for suspicious events, what actions should you take urgently?
1. Review container instance resources in your Azure subscriptions that pass sensitive data or credential information. This is a recommendation that Microsoft called out on the disclosure blog so the vulnerability most likely allowed access to these!
ACI provides multiple ways to pass sensitive information to containers in a container group at runtime.
- Passing an “environment variable” (Which makes the value at the management plane and at runtime)
- Passing a “secure environment variable” (Which only hides the value from the management plane but it is accessible at runtime)
- Using a “secret volume” (Which works in a similar way as a Kubernetes secret. It passes sensitive data in files mounted as a volume. The data is not encrypted at runtime)
Cloud native security solutions with CSPM capabilities like Prisma Cloud helps you to automate these types of checks across your Azure cloud footprint. You can also use the following checks in Azure Policy:
a. Azure Container Instance uses environment variables
Assessment: containers.environmentVariables exists
Action: If sensitive data like API tokens and credential information are used here before the flaw was fixed by Microsoft on 31st August 2021, revoke them immediately!
b. Azure Container Instance uses a secret volume
Assessment 1: volumes.secret exists
Assessment 2: containers.volumeMounts exists
Action: If a secret volume was used to pass sensitive data like API tokens and credential information, revoke them are used here before the flaw was fixed by Microsoft on 31st August 2021, revoke them immediately!
c. Azure Container Instance uses a file share volume
Assessment: volumes.azureFile exists
Action: Review the mounted file share for potentially sensitive information. The volume is mounted in read/write mode so check not only for data access but potentially malicious changes. Follow your sensitive data access investigation process for this.
The recommended way to pass sensitive configuration information is to use a managed identity with RBAC but as I touch on in point two, even that can be implemented in a way that increases impact in the case of a breach.
2. Review container instance resources in your Azure subscriptions that have an associated managed identity.
Managed identities allows an Azure service to assume permissions for accessing other Azure services (or other Azure AD protected resource) without having credentials in code (similar to IAM role in AWS). Currently, there are about 34 services in Azure that supports this capability including ACI (in preview). If an attacker can gain shell access to a container in ACI, they could potentially request and obtain an OAuth tokens which can then be used to attack other Azure services that the identity has been granted access (the default token lifetime is 24 hours). I showed an example of this in a demonstration here: https://youtu.be/P-mtmB4xYxI?t=1960
a. Azure Container Instance uses a managed identity
Assessment: identity exists
Action: Review the access granted to the identity. There is very little good reason for a managed identity to be granted permissions on a wide scope such as a subscription or a management group. This is just bad practice!! You may also want to review Azure activity logs for suspicious events performed with the identity (especially if the identity uses an “access policy” to get credentials from a key vault resource that also stores other credentials! This is because the “access policy” model for key vault is scoped at the resource level.The action here is to regenerate the identity!
Other security practices/implementation to look into
Apart from investigating your ACI and other Azure logs for suspicious events, what else can you do to protect yourself going forward. The two vulnerabilities that we’ve seen in the last few weeks for Azure won’t be the last. It’s important to design security in a way that assumes these vulnerabilites. Here are some practical ways to implement this principle in relation with this particular vulnerability:
1. Your containers should be minimalist where possible
Where possible, the container images for your applications should be super-minimalist or “distroless”. This means they should only contain an application and its runtime dependencies without a package manager, shell, or OS. This limits the ability of an attacker to connect to the container over the API! The screenshot below shows a sample error when I attempted to connect to a distroless container in ACI.
2. Implement runtime protection for production container apps regardless of where or how you run them — Azure Kubernetes Service (AKS), Azure RedHat OpenShift (ARO), Virtual Machines, App Service, Functions and even serverless services like Azure Container Instances (ACI).
Runtime protection capabilities allows you to monitor aspects of a running container like processes executed, file system calls, network connections. Policies can then be applied to either alert or take an action if a violating event occurs. There are different approaches that vendors take depending on the container runtime environment but for a serverless environment like ACI, I like the Prisma Cloud app-embedded defender approach as no code change is required it can be automated in a pipeline.
With a simple command like the one below, runtime protection can be embedded into a Dockerfile without having to modify any code. This can be done automatically in a pipeline using the twistcli tool or supported extensions like those for GitHub Actions and Azure DevOps.
$ ~/twistcli app-embedded embed -u $TWISTLOCK_USER -p $TWISTLOCK_PASSWORD --address $TWISTLOCK_CONSOLE --app-id $APP_ID --data-folder "/tmp" Dockerfile
Centralized policies can then be configured to monitor/prevent unauthorized processes or network connections (see screenshot below)
See this link for a walkthrough of Prisma cloud runtime protection for ACI — https://github.com/davidokeyode/prismacloud-workshops-labs/blob/main/workshops/azure-cloud-protection-pcce/modules/11-protect-serverless-container-workloads.md
To get into more details on Azure Security, please check out my books on Implementing Azure Security Technologies — Defense and Penetration Testing Azure for Ethical Hackers (co-authored with @kfosaeen) — Offense and myself authored — Penetration Testing Azure for Ethical Hackers