Kubectl Scripting: Security

Overview

When using kubectl for security or threat hunting in a Kubernetes environment, scripting can help automate the detection, monitoring, and response to potential security incidents. Below are examples and best practices for using kubectl scripts specifically tailored to security and threat hunting tasks.

Security and Threat Hunting with kubectl Scripts

1. Automating Security Checks

You can use kubectl scripts to regularly check for common security misconfigurations or vulnerabilities within your Kubernetes cluster.

Example: Script to Check for Privileged Pods

#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

# Find all pods running with privileged containers
echo "Checking for privileged pods in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.privileged}{"\n"}{end}' | grep true > privileged_pods.log

if [ -s privileged_pods.log ]; then
  echo "Privileged pods found:"
  cat privileged_pods.log
else
  echo "No privileged pods found in namespace $NAMESPACE"
fi

2. Monitoring for Suspicious Activity

Monitoring Kubernetes logs and events for unusual activity can help detect potential security incidents. A script can automate the collection and analysis of logs.

Example: Script to Monitor for Exec Commands in Pods

#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

# Monitor for exec commands in pods (requires audit logging enabled)
echo "Checking for exec commands in namespace $NAMESPACE"
kubectl logs -n kube-system -l k8s-app=audit-logs | grep 'exec' > exec_activity.log

if [ -s exec_activity.log ]; then
  echo "Exec activity detected:"
  cat exec_activity.log
else
  echo "No exec activity detected in namespace $NAMESPACE"
fi

3. Automating Incident Response Actions

When a threat is detected, scripts can automate incident response actions such as isolating compromised pods, collecting forensic data, or rolling back to a previous state.

Example: Script to Isolate a Compromised Pod

#!/bin/bash
set -e

POD_NAME=$1
NAMESPACE=$2

if [ -z "$POD_NAME" ] || [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <pod-name> <namespace>"
  exit 1
fi

# Isolate the pod by removing its network access
echo "Isolating pod $POD_NAME in namespace $NAMESPACE"
kubectl patch pod $POD_NAME -n $NAMESPACE -p '{"spec":{"containers":[{"name":"'$POD_NAME'","securityContext":{"allowPrivilegeEscalation":false,"privileged":false}}],"hostNetwork":false,"dnsPolicy":"None","dnsConfig":{"nameservers":["127.0.0.1"],"searches":[""]}}}'

4. Gathering Forensic Data

After detecting a suspicious activity, you may need to gather forensic data for further investigation. Scripts can automate the collection of logs, resource states, and network information.

Example: Script to Collect Logs and Resource States

#!/bin/bash
set -e

NAMESPACE=$1
POD_NAME=$2

if [ -z "$NAMESPACE" ] || [ -z "$POD_NAME" ]; then
  echo "Usage: $0 <namespace> <pod-name>"
  exit 1
fi

# Collect logs
echo "Collecting logs for pod $POD_NAME in namespace $NAMESPACE"
kubectl logs $POD_NAME -n $NAMESPACE > ${POD_NAME}_logs.txt

# Collect pod description
echo "Collecting pod description"
kubectl describe pod $POD_NAME -n $NAMESPACE > ${POD_NAME}_description.txt

# Collect network information
echo "Collecting network information"
kubectl get svc,ep -n $NAMESPACE > ${NAMESPACE}_network_info.txt

5. Regular Security Audits

Scripts can automate regular security audits, checking for misconfigurations, vulnerabilities, and compliance issues.

Example: Script to Audit Role-Based Access Control (RBAC)

#!/bin/bash
set -e

# List all roles and role bindings
echo "Auditing RBAC configurations"
kubectl get roles --all-namespaces > rbac_roles.txt
kubectl get rolebindings --all-namespaces > rbac_rolebindings.txt

# Check for roles with excessive privileges
echo "Checking for roles with excessive privileges"
grep -E 'cluster-admin|admin' rbac_roles.txt > excessive_privileges.txt

if [ -s excessive_privileges.txt ]; then
  echo "Excessive privileges found:"
  cat excessive_privileges.txt
else
  echo "No roles with excessive privileges found"
fi

6. Script to Detect Privileged Containers

This script checks for any pods running with privileged containers in a specified namespace, which can be a security risk.

#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

echo "Checking for privileged pods in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.privileged}{"\n"}{end}' | grep true > privileged_pods.log

if [ -s privileged_pods.log ]; then
  echo "Privileged pods found:"
  cat privileged_pods.log
else
  echo "No privileged pods found in namespace $NAMESPACE"
fi

7. Script to Monitor for Suspicious kubectl exec Commands

This script monitors the Kubernetes audit logs for any kubectl exec commands, which might indicate unauthorized access or suspicious activity.

#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

echo "Monitoring for exec commands in namespace $NAMESPACE"
kubectl logs -n kube-system -l k8s-app=audit-logs | grep 'exec' > exec_activity.log

if [ -s exec_activity.log ]; then
  echo "Exec activity detected:"
  cat exec_activity.log
else
  echo "No exec activity detected in namespace $NAMESPACE"
fi

8. Script to Isolate a Compromised Pod

This script isolates a potentially compromised pod by removing its network access, preventing it from communicating with other services or the internet.

#!/bin/bash
set -e

POD_NAME=$1
NAMESPACE=$2

if [ -z "$POD_NAME" ] || [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <pod-name> <namespace>"
  exit 1
fi

echo "Isolating pod $POD_NAME in namespace $NAMESPACE"
kubectl patch pod $POD_NAME -n $NAMESPACE -p '{"spec":{"containers":[{"name":"'$POD_NAME'","securityContext":{"allowPrivilegeEscalation":false,"privileged":false}}],"hostNetwork":false,"dnsPolicy":"None","dnsConfig":{"nameservers":["127.0.0.1"],"searches":[""]}}}'

9. Script to Collect Forensic Data from a Pod

This script gathers forensic data from a specified pod, including logs, pod description, and network information.

#!/bin/bash
set -e

NAMESPACE=$1
POD_NAME=$2

if [ -z "$NAMESPACE" ] || [ -z "$POD_NAME" ]; then
  echo "Usage: $0 <namespace> <pod-name>"
  exit 1
fi

# Collect logs
echo "Collecting logs for pod $POD_NAME in namespace $NAMESPACE"
kubectl logs $POD_NAME -n $NAMESPACE > ${POD_NAME}_logs.txt

# Collect pod description
echo "Collecting pod description"
kubectl describe pod $POD_NAME -n $NAMESPACE > ${POD_NAME}_description.txt

# Collect network information
echo "Collecting network information"
kubectl get svc,ep -n $NAMESPACE > ${NAMESPACE}_network_info.txt

echo "Forensic data collection complete."

10. Script to Audit Role-Based Access Control (RBAC)

This script audits the RBAC configurations in your cluster to detect roles with excessive privileges, such as cluster-admin.

#!/bin/bash
set -e

# List all roles and role bindings
echo "Auditing RBAC configurations"
kubectl get roles --all-namespaces > rbac_roles.txt
kubectl get rolebindings --all-namespaces > rbac_rolebindings.txt

# Check for roles with excessive privileges
echo "Checking for roles with excessive privileges"
grep -E 'cluster-admin|admin' rbac_roles.txt > excessive_privileges.txt

if [ -s excessive_privileges.txt ]; then
  echo "Excessive privileges found:"
  cat excessive_privileges.txt
else
  echo "No roles with excessive privileges found"
fi

12. Script to Detect Pods Running as Root

This script checks for pods that are running with root privileges, which can be a security risk if not properly controlled.

#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

echo "Checking for pods running as root in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].securityContext.runAsUser}{"\n"}{end}' | grep -E '\s0$' > root_pods.log

if [ -s root_pods.log ]; then
  echo "Pods running as root found:"
  cat root_pods.log
else
  echo "No pods running as root found in namespace $NAMESPACE"
fi

7. Script to Monitor Resource Usage for Anomalies

This script monitors resource usage (CPU, memory) for potential anomalies that could indicate a security issue, such as a DDoS attack or resource exhaustion.

bashCopy code#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

echo "Monitoring resource usage in namespace $NAMESPACE"
kubectl top pod -n $NAMESPACE > resource_usage.log

# Example check: flag any pods using more than 80% of requested CPU or memory
awk '$2 > 80 || $4 > 80 {print $1, "is using high resources: CPU:", $2 "%, Memory:", $4 "%"}' resource_usage.log > high_resource_usage.log

if [ -s high_resource_usage.log ]; then
  echo "High resource usage detected:"
  cat high_resource_usage.log
else
  echo "No high resource usage detected in namespace $NAMESPACE"
fi

8. Script to Monitor for Unusual Network Connections

This script checks for unusual network connections from pods, which could indicate suspicious activity or a compromised container.

bashCopy code#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

echo "Checking for unusual network connections in namespace $NAMESPACE"
kubectl exec -n $NAMESPACE $(kubectl get pods -n $NAMESPACE -o jsonpath='{.items[0].metadata.name}') -- netstat -anp | grep -i established > network_connections.log

if [ -s network_connections.log ]; then
  echo "Unusual network connections found:"
  cat network_connections.log
else
  echo "No unusual network connections found in namespace $NAMESPACE"
fi

9. Script to Automate Security Checks Before Deployment

This script performs a series of security checks before allowing a deployment to proceed.

bashCopy code#!/bin/bash
set -e

NAMESPACE="production"
DEPLOYMENT="my-app"

# Pre-deployment RBAC audit
echo "Auditing RBAC configurations before deployment..."
kubectl get roles --all-namespaces | grep -E 'cluster-admin|admin' > pre_deploy_rbac_audit.txt
if [ -s pre_deploy_rbac_audit.txt ]; then
  echo "Excessive privileges found in RBAC. Aborting deployment."
  exit 1
fi

# Check for privileged containers
echo "Checking for privileged containers..."
kubectl get pods -n $NAMESPACE -o jsonpath='{.items[*].spec.containers[*].securityContext.privileged}' | grep true
if [ $? -eq 0 ]; then
  echo "Privileged containers found. Aborting deployment."
  exit 1
fi

# Deploy the application if all checks pass
echo "All checks passed. Proceeding with deployment."
kubectl apply -f deployment.yaml -n $NAMESPACE

10. Script to Detect Configurations with Sensitive Data in Environment Variables

This script scans the environment variables in pods for sensitive information that should not be exposed, such as passwords or API keys.

bashCopy code#!/bin/bash
set -e

NAMESPACE=$1

if [ -z "$NAMESPACE" ]; then
  echo "Usage: $0 <namespace>"
  exit 1
fi

# Define sensitive keywords to search for
SENSITIVE_KEYWORDS="PASSWORD|SECRET|API_KEY|TOKEN"

echo "Scanning for sensitive data in environment variables in namespace $NAMESPACE"
kubectl get pods -n $NAMESPACE -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].env[*].name}{"\t"}{.spec.containers[*].env[*].value}{"\n"}{end}' | grep -E "$SENSITIVE_KEYWORDS" >

Integrating Security Scripts into CI/CD Pipelines

1. Pre-deployment Security Checks: Integrate security scripts into your CI/CD pipeline to automatically check for vulnerabilities or misconfigurations before deploying applications.

2. Post-deployment Monitoring: Use scripts to monitor for security events after deployment, ensuring that your applications remain secure in production.

3. Automated Rollback: Implement scripts that automatically roll back deployments if certain security conditions are met, such as a failed security check or detection of a suspicious activity.

Example of a Full Security Script for CI/CD Integration

Full Script: Automated Security Audit and Deployment

#!/bin/bash
set -e

NAMESPACE="production"
DEPLOYMENT="my-app"
IMAGE_TAG="v1.2.0"

# Pre-deployment RBAC audit
echo "Auditing RBAC configurations before deployment..."
kubectl get roles --all-namespaces | grep -E 'cluster-admin|admin' > pre_deploy_rbac_audit.txt
if [ -s pre_deploy_rbac_audit.txt ]; then
  echo "Excessive privileges found in RBAC. Aborting deployment."
  exit 1
fi

# Apply Deployment
echo "Deploying application..."
kubectl set image deployment/$DEPLOYMENT my-container=my-image:$IMAGE_TAG -n $NAMESPACE
kubectl rollout status deployment/$DEPLOYMENT -n $NAMESPACE

# Post-deployment security checks
echo "Performing post-deployment security checks..."
kubectl logs -n kube-system -l k8s-app=audit-logs | grep 'exec' > post_deploy_exec_activity.log
if [ -s post_deploy_exec_activity.log ]; then
  echo "Suspicious exec activity detected post-deployment. Rolling back..."
  kubectl rollout undo deployment/$DEPLOYMENT -n $NAMESPACE
else
  echo "Deployment successful and secure."
fi

---

Best Practices for Security Scripting with kubectl

1. Run Scripts with Least Privilege: Ensure that the service account or user running the script has the minimum necessary permissions to perform the tasks. Avoid running scripts with cluster-admin privileges unless absolutely necessary.

2. Use Secure Secrets Management: Avoid hardcoding sensitive information such as credentials in your scripts. Instead, use Kubernetes Secrets or environment variables to manage sensitive data securely.

3. Log Actions: Ensure that all actions taken by your scripts are logged, including any changes made to the cluster. This is crucial for auditing and forensic analysis.

4. Test Scripts in a Staging Environment: Before deploying security scripts in production, test them in a staging environment to ensure they work as expected without causing unintended disruptions.

5. Integrate with Alerting Systems: Combine your scripts with alerting mechanisms (e.g., Slack notifications, email alerts) to ensure that security teams are immediately notified of potential threats.

6. Automate Remediation: Where possible, automate the remediation of common security issues, such as isolating compromised pods or revoking access to suspicious users.

---

Summary

By using kubectl scripts for security and threat hunting, you can automate and enhance your cluster’s security posture, ensuring that threats are detected early, and incidents are handled quickly and efficiently.