Secret management in Kubernetes is an interresting aspect. There are two main options how applications that require secrets can get access on them:

1. The application communicates with a central service and receives the secrets during runtime
2. The secrets are available in Kubernetes and the application assumes that the secrets are available as environment variables or as files

But how in case of option 2 you store the secrets in Kubernetes and how do you don’t expose them for example to your developers when you store them unencrypted inside your git repository. Especially GitOps as a best practice for managing applications on Kubernetes requires a way to sync secrets with Kubernetes without storing them unencrypted in Git.

I know that there are some solutions available that tries to solve the problem in a bit another way then I solved the problem for my work. One solution is storing the secrets encrypted in Git and they get decrypted during deployment. Another way is using an mutating webhook that replaces the environment variables during deployment with secrets. With Kubernetes 1.16 a new way gets available based on ephemeral volumes. But this one is still under development.

Especially if you have multiple different use-cases secret management can become tricky. Therefore I’ve developed Vault-CRD a small Operator that runs inside the Kubernetes Cluster and synchronizes secrets stored in HashiCorp Vault with Kubernetes Secrets. Therefore it authenticates itself to HashiCorp Vault and based on a Custom Resource Definition that describes the secrets that should be synchronized with Kubernetes it generates a secret in Kubernetes.

All the following use-case examples require an installation of Vault-CRD inside your Kubernetes Cluster. For more details please see the installation instructions: Install Vault-CRD

Synchronizing secrets from HashiCorp Vault to Kubernetes

The easiest one is synchronizing secrets that are stored in HashiCorp Vault with Kubernetes. Therefore we first save the secrets as KV-1 or KV-2 secrets in HashiCorp Vault:

## store secret in kv-1 mountpoint
$ vault write secret/usercredentials username=testuser password=verysecure

## store secret in kv-2 mountpoint
vault kv put versioned/github apitoken=123123123

After this we can apply our resource based on the Custom Resource Definition of Vault-CRD to Kubernetes:

## Synchronize a secret stored as KV-1 secret in HashiCorp Vault's path secret/test-secret with Kubernetes
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: usercredentials
spec:
  type: "KEYVALUE"
  path: "secret/usercredentials"

## Synchronize a secret stored as KV-2 secret in HashiCorp Vault's path versioned/example with Kubernetes
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: github
spec:
  path: "versioned/github"
  type: "KEYVALUEV2"
  versionConfiguration:
    version: 1

Now you should see that the secrets are synchronized with Kubernetes and available in it. In case a secret gets changed in HashiCorp Vault it also gets updated in Kubernetes.

$ kubectl get secret test-secret
NAME                                   TYPE                                  DATA      AGE
usercredentials                        Opaque                                2         12s
github                                 Opaque                                1         10s

Now you can use this secrets inside your application for example as environment variables:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  ...
spec:
  ...
  spec:
    ...
    containers:
      - ...
        env:
          - name: USERNAME
            valueFrom:
              secretKeyRef:
                name: usercredentials
                key: username
          - name: PASSWORD
            valueFrom:
              secretKeyRef:
                name: usercredentials
                key: password
          - name: APITOKEN
            valueFrom: 
              secretKeyRef:
                name: github
                key: apitoken
...     

Using Property files

If we are talking about few secrets as environment variables it is very easy to sync them in such a way and to mount them as environment variables as valueFrom to our running pod/container. But what about multiple secrets like 10 or 15, how to sync them automatically?

Therefore a secret type called properties is available. This one makes it easy to generate secrets from templates and sync them as one time job with Kubernetes. Based on the two secrets created in the example above we can also create a secret property that contains all the variables:

apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: properties
spec:
  type: "PROPERTIES"
  propertiesConfiguration:
    files:
      application.properties: | 
        github.apitoken={{ vault.lookupV2('versioned/github').get('apitoken') }}
        database.username={{ vault.lookup('secret/usercredentials', 'username') }}
        database.password={{ vault.lookup('secret/usercredentials', 'password') }}

Now we can use this rendered properties file for example with a Spring Boot 2 application:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  ...
spec:
  ...
  spec:
    ...
    containers:
      - ...
        env:
          - name: SPRING_CONFIG_ADDITIONAL_LOCATION
            value: /opt/properties/application.properties

        volumeMounts:
          - mountPath: /opt/properties
            name: properties
    volumes:
      - name: properties
        secret:
          defaultMode: 420
          secretName: properties

Write your own Proxy for in-cluster TLS encryption

Before Istio became so popular it was really hard to do TLS encryption in the cluster. And also today if you won’t use Istio for different reasons it is still not very easy to do this. One possible solution is mounting a TLS secret to your application and do encryption in the application. The biggest problem of this solution is, that certificates typically have only a short live time and must be refreshed. Sure you can handle this inside your application, but how do you do this without a downtime to reconfigure your server certificates?

Therefore I’ve developed a proxy based on Envoy that automatically detects when the secret in the file system has changed and performs a hot reload Vault-CRD-Proxy. It spawns a new process with the new certificates for encryption and moves all existing connections to the new process. The old process is then automatically shut down.

The setup is very simple again. First we create the server-tls Vault resource. To allow other applications to access our application inside the cluster via the service the certificate should contain not only the name of the service as Common-Name, it should also contain the namespace where the application is running inside. So other applications can access the server via the url https://application.default:8443

apiVersion: koudingspawn.de/v1
kind: Vault
metadata:
  name: server-tls
spec:
  path: "testpki/issue/testrole"
  pkiConfiguration:
    altNames: application.default
    commonName: application
  type: PKI

Now we create the deployment with a Vault-CRD-Proxy in front of it:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  ...
spec:
  ...
  spec:
    ...
    containers:
      - ...
      - name: proxy
        env:
          - name: TARGET_PORT
          value: "8080"
        image: daspawnw/vault-crd-proxy:latest
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /envoy-health-check
            port: 8443
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        ports:
          - containerPort: 8443
            name: https-proxy
            protocol: TCP
        volumeMounts:
          - mountPath: /etc/ssl/private
            name: server-tls


    volumes:
      - name: server-tls
        secret:
          defaultMode: 420
          secretName: server-tls

Now the application starts and in front of it the Vault-CRD-Proxy runs for TLS termination. When Vault-CRD refreshes the certificate Envoy performs a hot reload to update the used Certificate.

When we create as last step our Service everything is done to make our application accessible to the outside via an encrypted channel:

apiVersion: v1
kind: Service
metadata:
  name: application
spec:
  selector:
    app: application
  ports:
    - protocol: TCP
      port: 8443
      targetPort: 8443

Another application that tries to access now the Service must trust our self signed root certificate. If we are using a Java application then the certificate must be imported into the cacerts store inside the java directory:

keytool -importcert -alias self-signed-root -keystore /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/cacerts -storepass changeit -file rootca.pem

If we are using a go application it must be placed inside the ca-certificates.crt file in the /etc/ssl/certs directory:

cat rootca.pem >> /etc/ssl/certs/ca-certificates.crt

Björn Wenzel

My name is Björn Wenzel. I’m a Platform Engineer working for Schenker with interests in Kubernetes, CI/CD, Spring and NodeJS.

•