Secrets

Secrets are an out of the box way Kubernetes provides to store sensitive data. Most similar to config maps, these are treated with a bit of extra care under the hood in Kubernetes.

Secrets are meant to give developers a way of specifying common types of sensitive data (basic-auth credentials, image registry credentials, TLS certs, etc) without including it (insecurely) in the code (application or infrastructure) of their containerized application. A typical generic secret that one will come across are the credentials for accessing a database.

The heart of any secret is not displayed in plain-text by default. Instead, secret data is base64 encoded and needs to be decoded to be read.

Like most data in the Kubernetes API, secrets are stored within the etc distributed data store. Whilst access to this data is mediated by the cluster’s RBAC, it should be noted that Secrets are NOT encrypted at rest within etcd in Kubernetes by default. This can be enabled on generic Kubernetes by following these instructions. OpenShift makes this even easier as documented here

Prerequisites

Make sure you are in the correct namespace:

You will need to create the myspace if you haven’t already. Check for the existence of the namespace with

kubectl get ns myspace

If the response is:

Error from server (NotFound): namespaces "myspace" not found

Then you can create the namespace with:

kubectl create ns myspace
 
kubectl config set-context --current --namespace=myspace

Make sure nothing is running in your namespace:

kubectl get all
No resources found in myspace namespace.

Deploy myboot service:

kubectl apply -f apps/kubefiles/myboot-deployment.yml

Deploy myboot Service:

kubectl apply -f apps/kubefiles/myboot-service.yml

In a separate terminal (hereafter referred to as Terminal 2) set up a watch on the pods:

  • Terminal 2

watch -n 1 "kubectl get pods -o wide \(1)
  | awk '{print \$1 \" \" \$2 \" \" \$3 \" \" \$5 \" \" \$7}' | column -t" (2)
1 the -o wide option allows us to see the node that the pod is schedule to
2 to keep the line from getting too long we’ll use awk and column to get and format only the columns we want

Meanwhile, in the main terminal, send a request:

  • Minikube

  • Hosted

IP=$(minikube ip -p devnation)
PORT=$(kubectl get service/myboot -o jsonpath="{.spec.ports[*].nodePort}")

If using a hosted Kubernetes cluster like OpenShift then use curl and the EXTERNAL-IP address with port 8080 or get it using kubectl:

IP=$(kubectl get service myboot -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
PORT=$(kubectl get service myboot -o jsonpath="{.spec.ports[*].port}")
If you are in AWS, you need to get the hostname instead of ip. 
IP=$(kubectl get service myboot -o jsonpath="{.status.loadBalancer.ingress[0].hostname}")

Curl the Service:

curl $IP:$PORT

which should give us the by now familiar response

Aloha from Spring Boot! 1 on myboot-7cbfbd9b89-dl2hv

Creating Secrets

Previously, we used a ConfigMap to hold a database connection string (user=MyUserName;password=*****). Instead, let’s create a secret to hold this sensitive data.

The kubectl CLI has some support for creating generic (or opaque) secrets like the one we would use for a database login.

kubectl create secret generic mysecret --from-literal=user='MyUserName' --from-literal=password='mypassword'
kubectl get secrets

Which will now yield output similar to the following

  • Minikube

  • OpenShift

NAME                  TYPE                                  DATA   AGE
default-token-nxkpw   kubernetes.io/service-account-token   3      5d12h
mysecret              Opaque                                2      25s
NAME                       TYPE                                  DATA   AGE
builder-dockercfg-96ml5    kubernetes.io/dockercfg               1      3d6h
builder-token-h5g82        kubernetes.io/service-account-token   4      3d6h
builder-token-vqjqz        kubernetes.io/service-account-token   4      3d6h
default-dockercfg-bsnjr    kubernetes.io/dockercfg               1      3d6h
default-token-bl77s        kubernetes.io/service-account-token   4      3d6h
default-token-vlzsl        kubernetes.io/service-account-token   4      3d6h
deployer-dockercfg-k6npn   kubernetes.io/dockercfg               1      3d6h
deployer-token-4hb78       kubernetes.io/service-account-token   4      3d6h
deployer-token-vvh6r       kubernetes.io/service-account-token   4      3d6h
mysecret                   Opaque                                2      5s

Because this is a Secret and not a ConfigMap, the user & password are not immediately visible:

kubectl describe secret mysecret
Name:         mysecret
Namespace:    myspace
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
password:  10 bytes
user:      10 bytes
kubectl get secret mysecret -o yaml
apiVersion: v1
data:
  password: bXlwYXNzd29yZA==
  user: TXlVc2VyTmFtZQ==
kind: Secret
metadata:
  creationTimestamp: "2020-03-31T20:19:26Z"
  name: mysecret
  namespace: myspace
  resourceVersion: "4944690"
  selfLink: /api/v1/namespaces/myspace/secrets/mysecret
  uid: e8c5f12e-bd71-4d6b-8d8c-7af9ed6439f8
type: Opaque

Copy the value of the password field above into the echo command below to prove that it is base64 encoded

echo 'bXlwYXNzd29yZA==' | base64 --decode
mypassword

If pressed for time, you can run the following command instead

B64_PASSWORD=$(kubectl get secret mysecret -o jsonpath='{.data.password}')
echo "password:$B64_PASSWORD is decoded as $(echo $B64_PASSWORD | base64 --decode)"

And then do the same for the username

echo 'TXlVc2VyTmFtZQ==' | base64 --decode
MyUserName

If pressed for time, you can run the following command instead

B64_DATA=$(kubectl get secret mysecret -o jsonpath='{.data.user}')
echo "username:$B64_DATA is decoded as $(echo $B64_DATA | base64 --decode)"

Or get them using kubectl:

kubectl get secret mysecret -o jsonpath='{.data.password}' | base64 --decode

Using Secrets

Let’s take a look at a deployment, myboot-deployment-configuration-secret.yml, that will make use of our newly created secret.

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open myboot-deployment-configuration-secret.yml
myboot-deployment-configuration-secret.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: myboot
  name: myboot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myboot
  template:
    metadata:
      labels:
        app: myboot
    spec:
      containers:
      - name: myboot
        image: quay.io/rhdevelopers/myboot:v1
        ports:
          - containerPort: 8080
        volumeMounts:
          - name: mysecretvolume (1)
            mountPath: /mystuff/secretstuff
            readOnly: true
        resources:
          requests:
            memory: "300Mi"
            cpu: "250m" # 1/4 core
          limits:
            memory: "400Mi"
            cpu: "1000m" # 1 core
      volumes:
        - name: mysecretvolume (2)
          secret:
            secretName: mysecret
1 This determines where the pod will find the secret. It will be in a file in the /mystuff/secretstuff directory in the pod
2 This defines what mysecretvolume should actually mount. In this case mysecret, the secret we just created above.

One way to allow deployments (pods) to use secrets is to provide them via Volume Mounts:

        volumeMounts:
          - name: mysecretvolume
            mountPath: /mystuff/mysecretvolume

Let’s update our deployment to use this volume:

kubectl replace -f apps/kubefiles/myboot-deployment-configuration-secret.yml

Once the deployment has been updated, exec into the newly created Pod:

Results in:

total 0
lrwxrwxrwx. 1 root root 15 Jul 19 03:37 password -> ..data/password (1)
lrwxrwxrwx. 1 root root 11 Jul 19 03:37 user -> ..data/user
mypassword (2)
1 Refer back to the secret definition. Each field under the .data section of the secret has become a file in this directory that represents the mounted secret
2 cat ing the value of the password file gives the value of the .data.password field in the secret we defined above

Alternatively, you can just run the following command to rsh into the pod and poke around

PODNAME=$(kubectl get pod -l app=myboot --field-selector 'status.phase!=Terminating' -o name)
kubectl exec -it $PODNAME -- /bin/bash

But how would your application know to look in this directory for credentials? Whilst it could be hardcoded in the application (or via properties) you could also provide the path via /mystuff/mysecretvolume to the pod via an environment variable so the application knows where to look.

It’s also possible to expose secrets directly as environment variables, but that’s beyond the scope of this tutorial.

For more information on secrets, see here

Clean Up

kubectl delete deployment myboot
kubectl delete service myboot