Managing Secrets with Helm and Helm Secrets

In Kubernetes, managing sensitive data such as API keys, passwords, and certificates securely is crucial. While Kubernetes provides mechanisms like Secrets to handle sensitive information, ensuring that these secrets are managed securely throughout your Helm workflows can be challenging. This lesson focuses on advanced strategies for managing secrets with Helm, including the use of Helm Secrets, a plugin that enhances the security of your sensitive data by encrypting secrets before they are stored in version control. By the end of this lesson, you’ll understand how to manage secrets securely with Helm, integrate Helm Secrets into your workflows, and avoid common pitfalls.

Overview of Kubernetes Secrets

Kubernetes Secrets are used to store and manage sensitive information, such as passwords, tokens, and certificates, in a secure way. However, managing these secrets with Helm requires careful attention to avoid exposing sensitive data.

What are Kubernetes Secrets?

Kubernetes Secrets are objects used to store sensitive information in your cluster. These secrets can be referenced by Pods to access confidential data without embedding it directly in the container images or configuration files.

Types of Kubernetes Secrets:

Opaque Secrets: Used for storing arbitrary user-defined data, such as passwords or API tokens.
TLS Secrets: Used for storing TLS certificates and keys.
Docker Registry Secrets: Used for authenticating with Docker registries.

Example of a Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm

In this example, the username and password fields are base64 encoded.

Challenges of Managing Secrets with Helm

While Kubernetes Secrets provide a secure way to manage sensitive data, incorporating them into Helm charts presents challenges:

Version Control Risks: Storing secrets directly in values.yaml files can expose them if the files are stored in version control.
Environment-Specific Secrets: Managing different secrets for different environments (e.g., development, staging, production) can be complex.
Templating Secrets: Using secrets in Helm templates requires careful handling to ensure they are not accidentally exposed.

Introducing Helm Secrets

Helm Secrets is a plugin for Helm that addresses the challenges of managing secrets by encrypting them before they are stored in version control. It integrates with tools like sops (Secrets OPerationS), which allows for encryption and decryption of secrets using various key management systems (e.g., AWS KMS, GCP KMS, PGP).

What is Helm Secrets?

Helm Secrets is a Helm plugin that allows you to store encrypted secrets in your Helm charts. It works by encrypting the values files that contain sensitive information, ensuring that they can be safely committed to version control.

Key Features of Helm Secrets:

Encryption: Encrypts sensitive data using sops before storing it in version control.
Decryption on the Fly: Decrypts secrets at runtime, just before deploying them with Helm.
Environment Integration: Supports various key management systems (KMS) for encryption keys, such as AWS KMS, GCP KMS, and PGP.

Installing Helm Secrets

Before using Helm Secrets, you need to install the plugin and set up sops for encryption and decryption.

Command to Install Helm Secrets:

helm plugin install https://github.com/jkroepke/helm-secrets

Command to Install sops:

On macOS using Homebrew:
```
brew install sops
```
On Linux:
```
sudo apt-get install sops
```

Once installed, you can start using Helm Secrets to manage encrypted values files.

Encrypting and Decrypting Secrets with Helm Secrets

Helm Secrets makes it easy to encrypt and decrypt sensitive data stored in your Helm charts.

Encrypting Secrets

To encrypt a values file containing sensitive data, use the helm secrets encrypt command. This command uses sops to encrypt the file and generate a new file with a .dec extension, indicating it is decrypted.

Example of Encrypting a Values File:

helm secrets encrypt secrets.yaml

secrets.yaml: The original values file containing sensitive data.
secrets.yaml.dec: The encrypted version of the file, which can be safely stored in version control.

Contents of an Encrypted File (Using sops with AWS KMS):

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  username: ENC[AES256_GCM,data:...]
  password: ENC[AES256_GCM,data:...]

The encrypted fields are now secure and can be safely committed to version control.

Decrypting Secrets

When deploying your Helm chart, Helm Secrets automatically decrypts the encrypted values file at runtime, just before passing it to Helm.

Command to Decrypt and Deploy a Chart:

helm secrets upgrade --install my-release ./mychart -f secrets.yaml

This command decrypts the secrets.yaml file and uses it to deploy the Helm chart.

Best Practices for Managing Secrets with Helm

Managing secrets securely in Helm requires following best practices to ensure that sensitive information is not exposed and that your workflows remain secure.

Avoid Hardcoding Secrets

Never hardcode sensitive information directly in your Helm charts or values.yaml files. Instead, use Kubernetes Secrets or Helm Secrets to manage this data securely.

Best Practice:

Store secrets in separate, encrypted files.
Use environment variables or external secret management systems to inject sensitive data into your Helm charts at runtime.

Use Role-Based Access Control (RBAC)

Ensure that only authorized users and services have access to secrets. Use Kubernetes RBAC to control who can create, update, or read secrets in your cluster.

Example of an RBAC Policy:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "watch"]

This role grants read-only access to secrets within the default namespace.

Regularly Rotate Secrets

Regularly rotating secrets reduces the risk of exposure and limits the potential damage from a compromised secret. Use Helm Secrets and sops to easily rotate and update encrypted secrets.

Best Practice:

Establish a schedule for rotating secrets (e.g., every 90 days).
Automate the rotation process using CI/CD pipelines.

Integrate Secrets Management with CI/CD

Integrate Helm Secrets into your CI/CD pipeline to automate the secure management of secrets during deployments. Ensure that secrets are decrypted only at the time of deployment and that they are not exposed in logs or other artifacts.

Example of a CI/CD Pipeline Integrating Helm Secrets:

stages:
  - deploy

deploy:
  stage: deploy
  script:
    - helm secrets upgrade --install my-release ./mychart -f secrets.yaml
  only:
    - main

This pipeline ensures that encrypted secrets are used securely during deployments.

Hands-on Example: Managing Secrets with Helm Secrets

To put your knowledge into practice, let’s create a Helm chart that uses encrypted secrets managed by Helm Secrets.

Steps:

Create a Values File with Sensitive Data:

yamlCopy codeapiVersion: v1
kind: Secret
metadata:
  name: myapp-secret
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQ=

Encrypt the Values File:

bashCopy codehelm secrets encrypt secrets.yaml

Store the Encrypted File in Version Control:
- Commit the secrets.yaml.dec file to your Git repository.

Deploy the Chart with Encrypted Secrets:

bashCopy codehelm secrets upgrade --install myapp ./mychart -f secrets.yaml

Verify the Deployment:
- Check that the secrets were deployed correctly and securely in your Kubernetes cluster.

Summary

Managing secrets securely in Helm is critical for maintaining the integrity and security of your Kubernetes applications. By using Helm Secrets, you can encrypt sensitive data, integrate secret management into your CI/CD workflows, and ensure that secrets are handled securely throughout the deployment process. Following best practices, such as avoiding hardcoding secrets and regularly rotating them, further enhances the security of your Kubernetes environments.

PreviousHelmfile and Continuous Integration NextTroubleshooting and Debugging Helm

Last updated 10 months ago