When using kubectl for security or threat hunting in a Kubernetes environment, scripting can help automate the detection, monitoring, and response to potential security incidents. Below are examples and best practices for using kubectl scripts specifically tailored to security and threat hunting tasks.
Security and Threat Hunting with kubectl Scripts
1. Automating Security Checks
You can use kubectl scripts to regularly check for common security misconfigurations or vulnerabilities within your Kubernetes cluster.
Example: Script to Check for Privileged Pods
#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
# Find all pods running with privileged containers
echo "Checking for privileged pods in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.privileged}{"\n"}{end}' | grep true > privileged_pods.log
if [ -s privileged_pods.log ]; then
echo "Privileged pods found:"
cat privileged_pods.log
else
echo "No privileged pods found in namespace $NAMESPACE"
fi
2. Monitoring for Suspicious Activity
Monitoring Kubernetes logs and events for unusual activity can help detect potential security incidents. A script can automate the collection and analysis of logs.
Example: Script to Monitor for Exec Commands in Pods
#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
# Monitor for exec commands in pods (requires audit logging enabled)
echo "Checking for exec commands in namespace $NAMESPACE"
kubectl logs -n kube-system -l k8s-app=audit-logs | grep 'exec' > exec_activity.log
if [ -s exec_activity.log ]; then
echo "Exec activity detected:"
cat exec_activity.log
else
echo "No exec activity detected in namespace $NAMESPACE"
fi
3. Automating Incident Response Actions
When a threat is detected, scripts can automate incident response actions such as isolating compromised pods, collecting forensic data, or rolling back to a previous state.
Example: Script to Isolate a Compromised Pod
#!/bin/bash
set -e
POD_NAME=$1
NAMESPACE=$2
if [ -z "$POD_NAME" ] || [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <pod-name> <namespace>"
exit 1
fi
# Isolate the pod by removing its network access
echo "Isolating pod $POD_NAME in namespace $NAMESPACE"
kubectl patch pod $POD_NAME -n $NAMESPACE -p '{"spec":{"containers":[{"name":"'$POD_NAME'","securityContext":{"allowPrivilegeEscalation":false,"privileged":false}}],"hostNetwork":false,"dnsPolicy":"None","dnsConfig":{"nameservers":["127.0.0.1"],"searches":[""]}}}'
4. Gathering Forensic Data
After detecting a suspicious activity, you may need to gather forensic data for further investigation. Scripts can automate the collection of logs, resource states, and network information.
Example: Script to Collect Logs and Resource States
#!/bin/bash
set -e
NAMESPACE=$1
POD_NAME=$2
if [ -z "$NAMESPACE" ] || [ -z "$POD_NAME" ]; then
echo "Usage: $0 <namespace> <pod-name>"
exit 1
fi
# Collect logs
echo "Collecting logs for pod $POD_NAME in namespace $NAMESPACE"
kubectl logs $POD_NAME -n $NAMESPACE > ${POD_NAME}_logs.txt
# Collect pod description
echo "Collecting pod description"
kubectl describe pod $POD_NAME -n $NAMESPACE > ${POD_NAME}_description.txt
# Collect network information
echo "Collecting network information"
kubectl get svc,ep -n $NAMESPACE > ${NAMESPACE}_network_info.txt
5. Regular Security Audits
Scripts can automate regular security audits, checking for misconfigurations, vulnerabilities, and compliance issues.
Example: Script to Audit Role-Based Access Control (RBAC)
#!/bin/bash
set -e
# List all roles and role bindings
echo "Auditing RBAC configurations"
kubectl get roles --all-namespaces > rbac_roles.txt
kubectl get rolebindings --all-namespaces > rbac_rolebindings.txt
# Check for roles with excessive privileges
echo "Checking for roles with excessive privileges"
grep -E 'cluster-admin|admin' rbac_roles.txt > excessive_privileges.txt
if [ -s excessive_privileges.txt ]; then
echo "Excessive privileges found:"
cat excessive_privileges.txt
else
echo "No roles with excessive privileges found"
fi
6. Script to Detect Privileged Containers
This script checks for any pods running with privileged containers in a specified namespace, which can be a security risk.
#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
echo "Checking for privileged pods in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.privileged}{"\n"}{end}' | grep true > privileged_pods.log
if [ -s privileged_pods.log ]; then
echo "Privileged pods found:"
cat privileged_pods.log
else
echo "No privileged pods found in namespace $NAMESPACE"
fi
7. Script to Monitor for Suspicious kubectl exec Commands
This script monitors the Kubernetes audit logs for any kubectl exec commands, which might indicate unauthorized access or suspicious activity.
#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
echo "Monitoring for exec commands in namespace $NAMESPACE"
kubectl logs -n kube-system -l k8s-app=audit-logs | grep 'exec' > exec_activity.log
if [ -s exec_activity.log ]; then
echo "Exec activity detected:"
cat exec_activity.log
else
echo "No exec activity detected in namespace $NAMESPACE"
fi
8. Script to Isolate a Compromised Pod
This script isolates a potentially compromised pod by removing its network access, preventing it from communicating with other services or the internet.
#!/bin/bash
set -e
POD_NAME=$1
NAMESPACE=$2
if [ -z "$POD_NAME" ] || [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <pod-name> <namespace>"
exit 1
fi
echo "Isolating pod $POD_NAME in namespace $NAMESPACE"
kubectl patch pod $POD_NAME -n $NAMESPACE -p '{"spec":{"containers":[{"name":"'$POD_NAME'","securityContext":{"allowPrivilegeEscalation":false,"privileged":false}}],"hostNetwork":false,"dnsPolicy":"None","dnsConfig":{"nameservers":["127.0.0.1"],"searches":[""]}}}'
9. Script to Collect Forensic Data from a Pod
This script gathers forensic data from a specified pod, including logs, pod description, and network information.
#!/bin/bash
set -e
NAMESPACE=$1
POD_NAME=$2
if [ -z "$NAMESPACE" ] || [ -z "$POD_NAME" ]; then
echo "Usage: $0 <namespace> <pod-name>"
exit 1
fi
# Collect logs
echo "Collecting logs for pod $POD_NAME in namespace $NAMESPACE"
kubectl logs $POD_NAME -n $NAMESPACE > ${POD_NAME}_logs.txt
# Collect pod description
echo "Collecting pod description"
kubectl describe pod $POD_NAME -n $NAMESPACE > ${POD_NAME}_description.txt
# Collect network information
echo "Collecting network information"
kubectl get svc,ep -n $NAMESPACE > ${NAMESPACE}_network_info.txt
echo "Forensic data collection complete."
10. Script to Audit Role-Based Access Control (RBAC)
This script audits the RBAC configurations in your cluster to detect roles with excessive privileges, such as cluster-admin.
#!/bin/bash
set -e
# List all roles and role bindings
echo "Auditing RBAC configurations"
kubectl get roles --all-namespaces > rbac_roles.txt
kubectl get rolebindings --all-namespaces > rbac_rolebindings.txt
# Check for roles with excessive privileges
echo "Checking for roles with excessive privileges"
grep -E 'cluster-admin|admin' rbac_roles.txt > excessive_privileges.txt
if [ -s excessive_privileges.txt ]; then
echo "Excessive privileges found:"
cat excessive_privileges.txt
else
echo "No roles with excessive privileges found"
fi
12. Script to Detect Pods Running as Root
This script checks for pods that are running with root privileges, which can be a security risk if not properly controlled.
#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
echo "Checking for pods running as root in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.runAsUser}{"\n"}{end}' | grep -E '\s0$' > root_pods.log
if [ -s root_pods.log ]; then
echo "Pods running as root found:"
cat root_pods.log
else
echo "No pods running as root found in namespace $NAMESPACE"
fi
7. Script to Monitor Resource Usage for Anomalies
This script monitors resource usage (CPU, memory) for potential anomalies that could indicate a security issue, such as a DDoS attack or resource exhaustion.
bashCopy code#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
echo "Monitoring resource usage in namespace $NAMESPACE"
kubectl top pod -n $NAMESPACE > resource_usage.log
# Example check: flag any pods using more than 80% of requested CPU or memory
awk '$2 > 80 || $4 > 80 {print $1, "is using high resources: CPU:", $2 "%, Memory:", $4 "%"}' resource_usage.log > high_resource_usage.log
if [ -s high_resource_usage.log ]; then
echo "High resource usage detected:"
cat high_resource_usage.log
else
echo "No high resource usage detected in namespace $NAMESPACE"
fi
8. Script to Monitor for Unusual Network Connections
This script checks for unusual network connections from pods, which could indicate suspicious activity or a compromised container.
bashCopy code#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
echo "Checking for unusual network connections in namespace $NAMESPACE"
kubectl exec -n $NAMESPACE $(kubectl get pods -n $NAMESPACE -o jsonpath='{.items[0].metadata.name}') -- netstat -anp | grep -i established > network_connections.log
if [ -s network_connections.log ]; then
echo "Unusual network connections found:"
cat network_connections.log
else
echo "No unusual network connections found in namespace $NAMESPACE"
fi
9. Script to Automate Security Checks Before Deployment
This script performs a series of security checks before allowing a deployment to proceed.
bashCopy code#!/bin/bash
set -e
NAMESPACE="production"
DEPLOYMENT="my-app"
# Pre-deployment RBAC audit
echo "Auditing RBAC configurations before deployment..."
kubectl get roles --all-namespaces | grep -E 'cluster-admin|admin' > pre_deploy_rbac_audit.txt
if [ -s pre_deploy_rbac_audit.txt ]; then
echo "Excessive privileges found in RBAC. Aborting deployment."
exit 1
fi
# Check for privileged containers
echo "Checking for privileged containers..."
kubectl get pods -n $NAMESPACE -o jsonpath='{.items[*].spec.containers[*].securityContext.privileged}' | grep true
if [ $? -eq 0 ]; then
echo "Privileged containers found. Aborting deployment."
exit 1
fi
# Deploy the application if all checks pass
echo "All checks passed. Proceeding with deployment."
kubectl apply -f deployment.yaml -n $NAMESPACE
10. Script to Detect Configurations with Sensitive Data in Environment Variables
This script scans the environment variables in pods for sensitive information that should not be exposed, such as passwords or API keys.
bashCopy code#!/bin/bash
set -e
NAMESPACE=$1
if [ -z "$NAMESPACE" ]; then
echo "Usage: $0 <namespace>"
exit 1
fi
# Define sensitive keywords to search for
SENSITIVE_KEYWORDS="PASSWORD|SECRET|API_KEY|TOKEN"
echo "Scanning for sensitive data in environment variables in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].env[*].name}{"\t"}{.spec.containers[*].env[*].value}{"\n"}{end}' | grep -E "$SENSITIVE_KEYWORDS" >
Integrating Security Scripts into CI/CD Pipelines
Pre-deployment Security Checks: Integrate security scripts into your CI/CD pipeline to automatically check for vulnerabilities or misconfigurations before deploying applications.
Post-deployment Monitoring: Use scripts to monitor for security events after deployment, ensuring that your applications remain secure in production.
Automated Rollback: Implement scripts that automatically roll back deployments if certain security conditions are met, such as a failed security check or detection of a suspicious activity.
Example of a Full Security Script for CI/CD Integration
Full Script: Automated Security Audit and Deployment
#!/bin/bash
set -e
NAMESPACE="production"
DEPLOYMENT="my-app"
IMAGE_TAG="v1.2.0"
# Pre-deployment RBAC audit
echo "Auditing RBAC configurations before deployment..."
kubectl get roles --all-namespaces | grep -E 'cluster-admin|admin' > pre_deploy_rbac_audit.txt
if [ -s pre_deploy_rbac_audit.txt ]; then
echo "Excessive privileges found in RBAC. Aborting deployment."
exit 1
fi
# Apply Deployment
echo "Deploying application..."
kubectl set image deployment/$DEPLOYMENT my-container=my-image:$IMAGE_TAG -n $NAMESPACE
kubectl rollout status deployment/$DEPLOYMENT -n $NAMESPACE
# Post-deployment security checks
echo "Performing post-deployment security checks..."
kubectl logs -n kube-system -l k8s-app=audit-logs | grep 'exec' > post_deploy_exec_activity.log
if [ -s post_deploy_exec_activity.log ]; then
echo "Suspicious exec activity detected post-deployment. Rolling back..."
kubectl rollout undo deployment/$DEPLOYMENT -n $NAMESPACE
else
echo "Deployment successful and secure."
fi
Best Practices for Security Scripting with kubectl
Run Scripts with Least Privilege: Ensure that the service account or user running the script has the minimum necessary permissions to perform the tasks. Avoid running scripts with cluster-admin privileges unless absolutely necessary.
Use Secure Secrets Management: Avoid hardcoding sensitive information such as credentials in your scripts. Instead, use Kubernetes Secrets or environment variables to manage sensitive data securely.
Log Actions: Ensure that all actions taken by your scripts are logged, including any changes made to the cluster. This is crucial for auditing and forensic analysis.
Test Scripts in a Staging Environment: Before deploying security scripts in production, test them in a staging environment to ensure they work as expected without causing unintended disruptions.
Integrate with Alerting Systems: Combine your scripts with alerting mechanisms (e.g., Slack notifications, email alerts) to ensure that security teams are immediately notified of potential threats.
Automate Remediation: Where possible, automate the remediation of common security issues, such as isolating compromised pods or revoking access to suspicious users.
Summary
By using kubectl scripts for security and threat hunting, you can automate and enhance your cluster’s security posture, ensuring that threats are detected early, and incidents are handled quickly and efficiently.