Secrets management

Vault

A secret store should be used to securely store and control sensitive data like tokens, passwords, certificates, encryption keys in one location. Vault also has other features like Dynamic Secrets, Leasing and Renewal and Revocation.

We have chosen Vault for secrets management because it's cloud agnostic, meaning we can use it in our private cloud as well as in other cloud providers.

Vault secret injection

By default Kubernetes secrets are stored in base64 which make them unsuitable for storage in git. Instead of storing the secrets in git or adding them manually on the cluster, it's possible to inject secrets via deployment/pod annotations

A vault-webhook is mutating deployment requests and adds the vault-env binary that connects with Vault. The startup command of the deployment are changed to run /vault/vault-env <container startup arguments> This way secrets are only available to the entrypoint/command that is being run in the container.

Understanding the flow

Namespace isolation and shared path

For every Kubernetes namespace, a Vault role,policy and path is created. The policy restricts this path from only being accessed by the namespace it belongs to. Example: namespace default on cluster 09999-demo-cluster is only allowed to fetch secrets from the following: path: 09999-demo-cluster/data/default

There is a shared path availabe in Vault which can be accessed by all namespaces from a certain cluster. path: 09999-demo-cluster/data/_shared/

Basic secret test

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault-09999.saas.true.nl" # vault URL
        vault.security.banzaicloud.io/vault-role: "default" # current namespace of deployment
        vault.security.banzaicloud.io/vault-path: "09999-demo-cluster" # cluster name
    spec:
      serviceAccountName: default
      containers:
      - name: alpine
        image: alpine
        command: ["sh", "-c", "echo $EXAMPLE_SECRET && echo going to sleep... && sleep 10000"]
        env:
        - name: EXAMPLE_SECRET
          value: vault:09999-demo-cluster/data/default#test

Inject all secrets from a Vault path

To add all secrets from a specific path as environment variables, use the vault-env-from-path annotation

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vault-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: vault
  template:
    metadata:
      labels:
        app.kubernetes.io/name: vault
      annotations:
        vault.security.banzaicloud.io/vault-addr: "https://vault-09999.saas.true.nl"
        vault.security.banzaicloud.io/vault-role: "default"
        vault.security.banzaicloud.io/vault-path: "09999-demo-cluster"
        vault.security.banzaicloud.io/vault-env-from-path: 09999-demo-cluster/data/default
    spec:
      serviceAccountName: default
      containers:
      - name: alpine
        image: alpine
        command: ["sh", "-c", "echo $SECRET_KEY_FROM_VAULT && echo going to sleep... && sleep 10000"]

Create Kubernetes secret from Vault

To create Kubernetes secrets that inject secret data from vault. Apply to following spec: Make sure to include annotation: vault.security.banzaicloud.io/vault-serviceaccount: default

apiVersion: v1
kind: Secret 
metadata:
  name: vault-test
  annotations:
    vault.security.banzaicloud.io/vault-addr: "https://vault-09999.saas.true.nl"
    vault.security.banzaicloud.io/vault-role: "default"
    vault.security.banzaicloud.io/vault-path: "09999-demo-cluster"
    vault.security.banzaicloud.io/vault-serviceaccount: default
stringData:
  example-secret: vault:09999-demo-cluster/data/default#test

Warnings

• Using the /vault/vault-env binary as part of a Liveness/Readiness probe command, will result in pod restarts whenever the Vault service is unreachable. Every time the /vault/vault-env binary is executed, a call to your Vault endpoint is being made. This will return a timeout, whenever Vault is unreachable.

• Keep in mind that when opening a shell in a container (for example for testing) that this overwrites the entrypoint and therefor also the injected Vault entrypoint command. If you need access to the Vault secrets in your testing you can simply prepend your command with /vault/vault-env.

Debugging errors

1) Error making API request - PUT https://vault-xxxxx.saas.true.nl/v1/auth/kubernetes/login Code: 400. Errors: missing client token Check that the deployment contains the correct vault-addr, vault-role, vault-path annotations.

2) Failed to request new Vault token x509: certificate signed by unknown authority The vault request is served with a Let's Encrypt certificate, This error indicates that the ca certificate is missing. Make sure the ca-certifcates package is installed and up to date in your container

3) Failed to request new Vault token: 403 permission denied Make sure the vault-role is set to the correct namespace, and that this namespace has access to the vault path, see #namespace-isolation

4) failed to read login data" app=vault-env err="open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory" type=jwt Make sure the pod is deployed with the default ServiceAccount. If there is no token mounted, make sure to check the automountServiceAccountToken setting on the default ServiceAccount.

5) could not mutate object: cannot fetch image descriptor: GET https://registry.services.k8s.true.nl/v2/01408-fakecompany/myrepo/manifests/v1.0.0: UNAUTHORIZED: unauthorized to access repository: 01408-fakecompany/myrepo, action: pull: unauthorized to access repository Make sure you have a pull secret defined for your pod, this error indicates that the webhook could not read the dockerfile manifest to get the entrypoint (see secret flow) to mutate it.

  imagePullSecrets:
  - name: my-pull-secret

6) level=ERROR msg="failed to request new Vault token" app=vault-env err="unable to log in with Kubernetes auth: Put \"https://127.0.0.1:8200/v1/auth/kubernetes/login\": dial tcp 127.0.0.1:8200: connect: connection refused" The /vault/vault-env binary is not able to use the VAULT_ADDR variable. This is commonly caused by missing annotations. Some other cause may be, that the /vault/vault-env is used in a command of for example a init-container.

# incorrect, it's not required to run the vault-env command again.
- args:
    - /bin/sh
    - -c
    - |
      /vault/vault-env env > .env
    command:
    - /vault/vault-env

# correct
  - args:
    - /bin/sh
    - -c
    - env > .env
    command:
    - /vault/vault-env