Kubernetes has become a potent tool for managing and delivering applications at scale in the constantly changing world of container orchestration. However, because of its enormous popularity, it is a favourite target for hackers.
In this blog post we bring you a detailed guide to creating a secure Kubernetes environment to safeguard clusters and sensitive data. Watch this video for more information.
The Kubernetes Attack Surface
Four critical aspects of the attack surface come into focus:
- Access via Kubernetes API Proxy and etcd API
- Exploiting Vulnerabilities in Apps or Third-Party Libraries
- Access via Kubelet API
- Access to the Servers or Virtual Machines
Let’s get into what you can do about this.
Protect your application runtime
Use static security testing like image vulnerability scanning
Exploiting vulnerabilities in your runtime can have serious consequences. To protect your containers from these threats, it’s essential to thoroughly examine container images for any known vulnerabilities before they are deployed.
If neglected, here are some of the threats you may face:
Privilege escalation: Attackers can gain higher levels of access within a container, allowing them to obtain sensitive resources that are typically restricted.
Remote shell access: Attackers may gain remote shell access to the container. This would give them control over the application and potentially even the entire cluster.
Potential security risks: This can lead to the unauthorised disclosure of sensitive information, potentially exposing it to individuals who should not have access to it.
DDoS attacks: These malicious actions can disrupt your services and significantly impact the overall user experience.
To automate this detection process, you can utilise tools such as Clair, Trivy, or other similar solutions.
Code vulnerability scanning guards against unseen threats.
This crucial practice helps you uncover security vulnerabilities in your application’s source code and dependencies. It acts like a magnifying glass, exposing hidden weaknesses before they can be exploited.
Make sure your libraries and dependencies are upto date and not running any version of vulnerable dependencies.
Even indirect use of vulnerable dependencies can introduce vulnerabilities. The recent Log4j vulnerability serves as a stark reminder of this.
Make sure you have not hardcoded any sensitive information into the code. Avoiding hard coded secrets is crucial to prevent data breaches.
Configuration scanning enforces best practices in YAML Files
Configurations in Kubernetes are defined using YAML files. These configurations determine how your pods, services, and other resources are deployed and interact. Ensuring these configurations adhere to security best practices is paramount.
Container hardening will help you maintain overall security of the cluster
1. Make sure your containers are immutable
The heart of containerization’s security philosophy. This approach treats containers as disposable entities that are replaced rather than modified. The practice is pivotal for enhanced security, consistent behaviour and easier rollbacks.
A few key strategies for hardening your runtime environment:
Disabling shell access within containers is a proactive measure against unauthorised manipulation of containers.
Make File System Read-Only
Setting your file system to read-only mode reduces the chances of malicious code or unauthorised data being written to the file system.
Run as a non root user
Running containers as non-root users mitigates the impact of potential breaches, limiting attackers' ability to access sensitive resources.
2. Enforce these security measures at the Kubernetes level without altering your container images:
Use startup probes to remove bash: Delay container readiness until certain conditions are met. This can be utilised to ensure that shells or unnecessary components are not available once the container is running.
Look at Kubernetes security settings: Enforce desired security measures at the pod or container level. By using settings such as RunAsGroup, RunAsUser, and RunAsNonRoot, non-root user access can be enforced for enhanced security.
Implement Open Policy Agent (OPA) and its policy enforcement tool, Gatekeeper: You can establish and enforce customised security policies throughout your Kubernetes environment. This approach guarantees compliance with best practices while preventing any configurations that may breach security standards.
Container Runtime Security
1. Disable privileged containers and privilege escalation
The risk of privilege escalation is a significant concern in container security. If an attacker gains access to a container, they can potentially escalate their privileges within that container and compromise the entire cluster.
2. Drop all capabilities and add only the ones that are needed
Employ a “less is more” approach when it comes to capabilities and get granular about it.
Containerized applications need specific capabilities to operate effectively. However, providing excessive capabilities can potentially introduce avoidable risks. This is essential because:
Containers should only possess the capabilities required for their intended functionality, reducing the potential attack surface.
It is best to start with no capabilities and only add the necessary ones for your application to function effectively.
By implementing advanced sandboxing techniques, you can enhance the level of control over your containers. This ensures that the processes within the containers are securely confined.
3. Use AppArmor or seccomp profiles to restrict processes running in containers
These profiles establish limits on the actions that processes running in containers can perform. By doing so, they effectively prevent unauthorised activities and restrict potential avenues for exploitation.
4. Container sandboxing to isolate and contain threats
This provides a layer of separation that isolates individual containers, helping to contain and mitigate potential threats.
By keeping containers isolated from each other, the risk of lateral movement during a breach is minimised. Thereby reducing the attack surface. It also reduces impact. If a container is compromised, sandboxing ensures that the threat does not spread to other areas of your Kubernetes environment.
5. Use a tool like Sysdig Falco to monitor abnormalities in container runtimes
Leveraging tools like Sysdig Falco allows you to monitor and detect abnormal behaviours within container runtimes. It can alert you to potentially malicious activities and deviations from expected behaviour.
Conceal information through secrets management
1. Don’t use environment variables to inject sensitive information to your containers.
Storing sensitive information like API keys or credentials in environment variables within containers can put your secrets at risk of being compromised.
Utilise a dedicated secret manager. Secret managers such as HashiCorp Vault, Azure Key Vault, AWS Secrets Manager, and Google Secret Manager provide a centralised solution for storing and managing sensitive data.
Mount secrets as files within your containers. This ensures that secrets are not accidentally exposed and reduces the vulnerability to potential attacks.
2. Make your container root file system read only.
This is a valuable defensive measure in protecting against unauthorised modifications:
By using an immutable root file system, we can ensure that critical system files remain untouched and secure. This prevents any unauthorised changes or manipulations by potential attackers.
Minimising write access limits the capacity of attackers to manipulate system configurations and install harmful software.
3. Developer discipline: Do not log sensitive information.
Developers must exercise caution and refrain from logging critical details, such as passwords or keys. By implementing proper logging practices, the accidental exposure of vital data can be prevented.
Isolate your tenants on the Network
To create secure multi-tenant Kubernetes environments:
1. Use namespaces.
This allows each tenant to have their own dedicated space for applications and resources.
2. Use network policies.
Equivalent to firewalls, Kubernetes network policies add a crucial layer of control over the flow of traffic between pods and namespaces.
Selective Communication: Configure network policies to deny all traffic by default between namespaces, then meticulously allow specific ingress and egress rules for enhanced control.
Make sure your CNI supports Network Policies in K8s
3. Make even inter service communication go through an API Gateway.
Centrally manage access and apply security measures such as authentication, authorization, and rate limiting.
4. Use mTLS easily with support of Service Mesh or an ebpf enabled CNI.
Service Mesh for mTLS: Adopting a service mesh, such as Istio or Linkerd, simplifies mutual Transport Layer Security (mTLS) implementation for enhanced communication security between services.
5. Use admission controller to enforce policies on the cluster.
Ensure that your Kubernetes environment adheres to established security standards and best practices.
You can leverage admission controllers like Open Policy Agent (OPA) and Kyverno to enforce policies during resource admission. These controllers validate incoming requests and ensure compliance before resources are created.
Protect your control plane
Hardening your Kubernetes cluster
1. Have proper RBAC policies set in your cluster
Create service accounts, manage certificate signing, and establish role bindings to regulate access. Be cautious with cluster role bindings, ensuring that they are well-considered and necessary.
2. Enable audit logging for insights and accountability
Monitoring and recording API requests and responses aids in understanding cluster behaviour and detecting suspicious activities.
3. Run CIS benchmark and do recommended fixes
Evaluate your cluster’s adherence to best practices. This bolsters your cluster’s defenses. Utilise tools like Kube Bench to automate benchmark assessments.
4. Use CIS hardened node images
Secure Node Images create a foundation of trust. Start your clusters on a secure footing with hardened node images.
5. Encrypt etcd
Make sure you are safeguarding the etcd datastore. In other words, protecting the cluster’s brain. Encrypt etcd data to prevent unauthorised access to critical configuration information.
6. Mount your secrets as files and not as environment variables
Carefully manage your secrets. Mount secrets as files rather than environment variables, minimising the risk of exposure and unauthorised access.
Kubernetes security is still new and vulnerabilities get patched frequently.
Lot of the cloud managed Kubernetes have the control plane security properly managed.
Many organisations need some level of multi-tenancy within their applications in K8s Clusters. These could be their internal apps or apps running by an outsourced vendor.
If proper standards and best practices are followed and you keep upto date with latest releases and patches, things should be alright.
As you continue on your security journey, remember that vigilance and adaptation are essential. Stay informed, remain proactive, and evolve your strategies to confront emerging threats. By integrating security into the fabric of your Kubernetes ecosystem, you not only ensure the resiliency of your apps but also set the foundation for a safe and thriving digital environment.