Threat Hunting Labs

Hands-On Threat Hunting Labs

Practical, hands-on experience is essential for mastering threat hunting in Kubernetes environments. In this section, we will outline several hands-on labs that simulate real-world threat scenarios. These labs are designed to help you practice the skills necessary for detecting, investigating, and responding to security incidents in a Kubernetes environment.

Lab 1: Detecting Unauthorized API Access

Objective: Identify and respond to unauthorized access attempts to the Kubernetes API server.

Setup

1. Kubernetes Cluster: Ensure you have a running Kubernetes cluster with access to the API server logs.

2. Audit Logs: Enable Kubernetes audit logging if it’s not already configured.

3. Attack Simulation: Use a tool like kubectl from a misconfigured service account or from an external IP address to simulate unauthorized API access.

Exercise Steps

1. Access the Audit Logs: Use kubectl or a logging tool like Elasticsearch to access and search through the audit logs.

   kubectl logs -n kube-system <api-server-pod-name> | grep "Unauthorized"

2. Analyze the Logs: Look for entries that show unauthorized access attempts, such as 401 or 403 responses from the API server.

3. Investigate the Source: Identify the user or service account attempting unauthorized access, as well as the IP address from which the requests originated.

4. Response Actions: Take appropriate response actions, such as revoking the service account’s access, updating RBAC policies, or blocking the offending IP address using network policies.

Discussion

• What patterns did you identify that indicated unauthorized access?

• How can you automate the detection of such events in a real-world environment?

Lab 2: Container Breakout Detection and Response

Objective: Detect and respond to a container breakout attempt.

Setup

1. Kubernetes Cluster: Use a cluster with access to node logs and the ability to monitor container activity.

2. Falco: Install Falco, an open-source runtime security tool, to monitor system calls and detect suspicious activity.

3. Attack Simulation: Use a known container escape vulnerability or a simulated breakout tool (e.g., a custom script that attempts to access the host’s file system from within a container).

Exercise Steps

1. Monitor with Falco: Ensure Falco is running and configured to alert on suspicious system calls, such as attempts to write to /etc/passwd or execute commands from unusual directories.

   sudo falco --config /etc/falco/falco.yaml

2. Simulate the Breakout: Execute the breakout script within a container. Observe how Falco detects the breakout attempt.

3. Respond to the Alert: Upon receiving the alert, investigate the affected container and node. Isolate the node and container to prevent further damage.

4. Forensic Analysis: Perform a forensic analysis on the node. Check logs for unauthorized file access, new processes, or changes to critical files.

Discussion

• How did Falco help in detecting the breakout attempt?

• What additional measures could you implement to prevent container breakouts?

Lab 3: Detecting Data Exfiltration

Objective: Identify and respond to a data exfiltration attempt.

Setup

1. Kubernetes Cluster: Ensure your cluster has network monitoring enabled, either through native tools like Kubernetes NetworkPolicy logs or third-party solutions like Calico or Istio.

2. Sensitive Data Simulation: Deploy an application that handles sensitive data. Ensure that logging and monitoring are configured to track data access and network activity.

3. Attack Simulation: Simulate an attack where data is exfiltrated from the application to an external server. This could be done using a script that reads data from the application and sends it to a remote IP address.

Exercise Steps

1. Monitor Network Traffic: Use tools like tcpdump or network policies to monitor outgoing traffic from the pod handling sensitive data.

   kubectl exec -it <pod-name> -- tcpdump -i eth0

2. Identify Anomalous Transfers: Look for large or unexpected data transfers, particularly to external IP addresses. Use network logs to trace the source and destination of the data.

3. Investigate Data Access: Review application logs to determine how the data was accessed. Identify the user or process responsible for initiating the data transfer.

4. Response Actions: Stop the data transfer by isolating the pod or blocking the destination IP address. Rotate credentials if they were compromised, and review access controls.

Discussion

• What indicators suggested that data exfiltration was taking place?

• How can you improve detection and prevention of data exfiltration in your Kubernetes environment?

Lab 4: Persistent Malware Detection

Objective: Detect and mitigate a persistent malware infection within a Kubernetes pod.

Setup

1. Kubernetes Cluster: Ensure your cluster can deploy and manage containers. Install tools like ClamAV for malware scanning or Sysdig Secure for runtime security.

2. Malware Simulation: Deploy a pod with an image that simulates malware behavior, such as creating persistent connections to a command-and-control server or downloading additional payloads.

Exercise Steps

1. Deploy the Infected Pod: Deploy the pod with the simulated malware. Monitor its behavior using security tools like Sysdig Secure or ClamAV.

2. Detect Malware Activity: Use your monitoring tools to detect signs of malware, such as unexpected network connections, high CPU usage, or the creation of suspicious files.

   clamscan --infected --recursive --include-dir=/var/lib/docker

3. Quarantine and Mitigation: Once detected, isolate the pod to prevent the malware from spreading. Remove the pod and delete the infected container image from your registry.

4. Analyze the Infection Vector: Investigate how the malware was introduced. Was it through a compromised image, a vulnerability in the application, or another vector?

Discussion

• What challenges did you face in detecting the malware?

• How can you ensure that similar malware does not infect your Kubernetes environment in the future?

Lab 5: Denial of Service (DoS) Attack Response

Objective: Detect and mitigate a DoS attack targeting a Kubernetes application.

Setup

1. Kubernetes Cluster: Ensure your cluster can handle real-time monitoring and alerting.

2. Load Testing Tool: Use a tool like k6 or Apache JMeter to simulate a DoS attack by overwhelming the application with requests.

3. Monitoring Setup: Configure Prometheus and Grafana to monitor key metrics such as CPU usage, memory consumption, and request latency.

Exercise Steps

1. Simulate the DoS Attack: Use the load testing tool to generate a high volume of requests to the target application, simulating a DoS attack.

   k6 run --vus 1000 --duration 30s script.js

2. Monitor Resource Usage: Use Prometheus to monitor the application’s resource usage during the attack. Identify signs of resource exhaustion, such as increased pod restarts or high CPU and memory usage.

3. Implement Mitigation Strategies: Respond to the attack by scaling up the application, implementing rate limiting, or deploying a WAF (Web Application Firewall) to block malicious traffic.

4. Analyze the Attack: Review logs and metrics to understand the attack’s impact. Determine whether it was targeted or random and assess the effectiveness of your mitigation strategies.

Discussion

• How did the attack affect the application’s performance?

• What steps can you take to prevent or mitigate DoS attacks in the future?

Conclusion

These hands-on labs provide a practical, in-depth experience in detecting and responding to various security incidents in a Kubernetes environment. By simulating real-world scenarios, you can refine your threat hunting skills, improve your response strategies, and better secure your Kubernetes deployments. The next sections will explore advanced threat detection techniques, incident response automation, and continuous security improvement in Kubernetes environments.