Threat Hunting Ideas

Key Activities to Monitor in Kubernetes

When it comes to monitoring Kubernetes, there are specific activities and events that are critical for maintaining the security, stability, and compliance of your Kubernetes clusters. Kubernetes is a complex, dynamic environment, and monitoring key activities can help you detect potential threats, misconfigurations, and unauthorized access.

1. Access Control and Authentication

• API Server Access

  • Activity: Monitoring access to the Kubernetes API server, especially from unknown or unauthorized sources.

  • Why Monitor?: The API server is the gateway to managing the Kubernetes cluster. Unauthorized access can lead to cluster compromise.

• Role-Based Access Control (RBAC) Changes

  • Activity: Creation, modification, or deletion of Roles, ClusterRoles, RoleBindings, and ClusterRoleBindings.

  • Why Monitor?: RBAC controls who can perform what actions in the cluster. Misconfigured or overly permissive RBAC settings can allow unauthorized actions.

• Service Account Token Usage

  • Activity: Monitoring the usage of service account tokens, especially those with high privileges.

  • Why Monitor?: Compromised service account tokens can be used to escalate privileges or perform unauthorized actions in the cluster.

2. Pod and Container Management

• Pod Creation, Deletion, and Modification

  • Activity: Monitoring the creation, deletion, and modification of pods, especially in critical namespaces like kube-system.

  • Why Monitor?: Unauthorized changes to pods can indicate an attempt to inject malicious code, escalate privileges, or disrupt services.

• Container Image Pulls

  • Activity: Monitoring which container images are being pulled from registries.

  • Why Monitor?: Pulling images from untrusted or unknown sources can introduce vulnerabilities or malicious software into your cluster.

• Resource Limits and Requests Changes

  • Activity: Changes to resource limits and requests for containers and pods.

  • Why Monitor?: Unauthorized changes can lead to resource exhaustion, affecting cluster performance and availability.

• Privileged Containers

  • Activity: Monitoring the deployment of containers with elevated privileges (privileged: true).

  • Why Monitor?: Privileged containers can access the host system, posing a significant security risk if misused.

3. Network and Security Configurations

• Network Policy Changes

  • Activity: Creation, modification, or deletion of NetworkPolicies.

  • Why Monitor?: NetworkPolicies control traffic flow between pods. Misconfigurations can expose services to unauthorized access.

• Ingress and Egress Traffic

  • Activity: Monitoring ingress and egress traffic rules, especially changes to Ingress resources or external IP configurations.

  • Why Monitor?: Uncontrolled traffic flow can lead to data exfiltration or unauthorized access to internal services.

• Secrets Management

  • Activity: Access to Kubernetes Secrets, especially creation, modification, or deletion.

  • Why Monitor?: Secrets often contain sensitive information like API keys or passwords. Unauthorized access can lead to security breaches.

• Service Configuration Changes

  • Activity: Changes to Kubernetes Services, especially those exposing services externally.

  • Why Monitor?: Exposing services externally can increase the attack surface and lead to unauthorized access.

4. Cluster and Node Management

• Node Status Changes

  • Activity: Monitoring when nodes go offline, come online, or experience status changes.

  • Why Monitor?: Node status changes can indicate underlying issues, such as hardware failures, resource exhaustion, or potential attacks.

• Kubelet Configuration and Logs

  • Activity: Monitoring changes to Kubelet configuration files and logs.

  • Why Monitor?: The Kubelet is responsible for managing containers on each node. Unauthorized changes can lead to node-level compromises.

• Cluster Autoscaler Activities

  • Activity: Monitoring the activities of the Cluster Autoscaler, especially unexpected scaling events.

  • Why Monitor?: Unexpected scaling can be a sign of resource misuse or DDoS attacks.

• Etcd Access and Changes

  • Activity: Monitoring access to and changes in etcd, the key-value store for Kubernetes.

  • Why Monitor?: Unauthorized access to etcd can compromise the entire cluster as it stores all cluster data.

5. Security and Compliance Monitoring

• Pod Security Policies (PSP) Changes

  • Activity: Creation, modification, or deletion of PodSecurityPolicies.

  • Why Monitor?: PSPs enforce security standards for pod deployments. Changes to these policies can weaken cluster security.

• Compliance with CIS Benchmarks

  • Activity: Monitoring for compliance with Kubernetes CIS Benchmarks, such as ensuring that certain security settings are enforced.

  • Why Monitor?: CIS Benchmarks provide guidelines for securing Kubernetes. Deviations from these benchmarks can indicate security risks.

• Falco Rules Violations

  • Activity: Monitoring for violations of Falco rules, which detect anomalous behavior at runtime.

  • Why Monitor?: Falco rules are designed to detect unexpected behavior, such as shell access within a container or changes to sensitive files.

• Admission Controller Logs

  • Activity: Monitoring logs from Admission Controllers, which validate or mutate requests to the Kubernetes API.

  • Why Monitor?: Admission Controllers can enforce security policies, so monitoring their activity helps ensure compliance and detect unauthorized changes.

6. Audit and Log Monitoring

• Audit Logs

  • Activity: Regularly review Kubernetes audit logs, which capture all requests to the API server.

  • Why Monitor?: Audit logs provide a comprehensive record of activities within the cluster, helping to detect and investigate suspicious behavior.

• Logging and Monitoring Configuration Changes

  • Activity: Changes to logging or monitoring configurations, such as Fluentd, Prometheus, or ELK stack configurations.

  • Why Monitor?: Misconfigurations can result in incomplete logs, making it difficult to detect or respond to incidents.

• Log Retention Policies

  • Activity: Changes to log retention policies for audit logs, application logs, and other critical data.

  • Why Monitor?: Ensuring that logs are retained for an appropriate period is essential for forensic analysis and compliance.

Tools for Monitoring Kubernetes Security Activities

1. Kubernetes Audit Logs

   • Description: Kubernetes provides audit logging, which tracks all requests to the API server, including who made the request, what actions were taken, and the outcome.

   • How to Use: Enable audit logging in your cluster, and configure it to capture events that are relevant to security, such as changes to RBAC or network policies.

2. Falco

   • Description: Falco is an open-source runtime security tool that monitors the behavior of your Kubernetes cluster. It uses rules to detect suspicious activity.

   • How to Use: Deploy Falco in your cluster to monitor for events like privilege escalation, namespace changes, or access to sensitive files.

3. Prometheus and Grafana

   • Description: Prometheus is a monitoring system that collects metrics, and Grafana is a visualization tool. Together, they can be used to monitor the health and performance of your Kubernetes cluster.

   • How to Use: Set up Prometheus to collect metrics from your cluster and use Grafana to create dashboards that visualize key metrics related to security and performance.

4. Kube-bench

   • Description: Kube-bench is a tool that checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.

   • How to Use: Run Kube-bench regularly to ensure your cluster complies with the CIS Benchmarks and address any identified security gaps.

5. Kubectl and Kubernetes Dashboards

   • Description: Kubectl is the command-line tool for interacting with Kubernetes clusters. Kubernetes dashboards provide a web-based interface for managing the cluster.

   • How to Use: Use kubectl to query the status of pods, services, and configurations. Regularly review the Kubernetes dashboard for an overview of the cluster's health and security status.

6. CSP Specific Kubernetes Monitoring

   • Description: If using CSP Kubernetes Service, you can utilize their specific services to provide integrated monitoring and security features.

Practical Monitoring Tips:

• Enable Detailed Auditing: Ensure that Kubernetes audit logging is enabled and configured to capture detailed information about critical events.

• Set Up Alerts: Use monitoring tools like Prometheus, Grafana, and Azure Monitor to set up alerts for unusual or unauthorized activities.

• Regular Compliance Checks: Use tools like Kube-bench to regularly check your cluster against security best practices and compliance benchmarks.

• Runtime Security Monitoring: Deploy Falco or similar tools to monitor the behavior of containers at runtime and detect anomalies.

• Centralize Logs: Use a centralized logging solution (e.g., ELK stack, Fluentd) to aggregate and analyze logs from all components of your Kubernetes cluster.

Summary

By focusing on these key activities and using the appropriate tools, you can effectively monitor your Kubernetes environment for security threats, unauthorized access, and misconfigurations, helping to ensure the integrity and security of your containerized applications.