Liveness & Readiness

Make sure you are in the correct namespace:

kubectl config set-context --current --namespace=myspace

Make sure nothing else is deployed:

kubectl get all

No resources found in myspace namespace.

Now we’re going to deploy our application with a Liveness and Readiness probe set. Take a look at myboot-deployment-live-ready.yml

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open myboot-deployment-live-ready.yml

myboot-deployment-live-ready.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myboot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myboot
  template:
    metadata:
      labels:
        app: myboot
        env: dev
    spec:
      containers:
      - name: myboot
        image: quay.io/rhdevelopers/myboot:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "300Mi"
            cpu: "250m" # 1/4 core
          limits:
            memory: "400Mi"
            cpu: "1000m" # 1 core
        livenessProbe:
          httpGet:
              port: 8080
              path: /alive
          initialDelaySeconds: 10
          periodSeconds: 5
          timeoutSeconds: 2
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 3

Now apply this deployment with the following command

kubectl apply -f apps/kubefiles/myboot-deployment-live-ready.yml

Describe the deployment:

kubectl describe deployment myboot

...
    Image:      quay.io/rhdevelopers/myboot:v1
    Port:       8080/TCP
    Host Port:  0/TCP
    Limits:
      cpu:     1
      memory:  400Mi
    Requests:
      cpu:        250m
      memory:     300Mi
    Liveness:     http-get http://:8080/ delay=10s timeout=2s period=5s #success=1 #failure=3
    Readiness:    http-get http://:8080/health delay=10s timeout=1s period=3s #success=1 #failure=3
...

Deploy a Service:

kubectl apply -f apps/kubefiles/myboot-service.yml

Minikube
Hosted

IP=$(minikube ip -p devnation)
PORT=$(kubectl get service/myboot -o jsonpath="{.spec.ports[*].nodePort}")

If using a hosted Kubernetes cluster like OpenShift then use curl and the EXTERNAL-IP address with port 8080 or get it using kubectl:

IP=$(kubectl get service myboot -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
PORT=$(kubectl get service myboot -o jsonpath="{.spec.ports[*].port}")

If you are in AWS, you need to get the hostname instead of ip.

IP=$(kubectl get service myboot -o jsonpath="{.status.loadBalancer.ingress[0].hostname}")

Curl the Service:

curl $IP:$PORT

And run loop script:

while true
do curl $IP:$PORT
sleep 0.8
done

Change the image:

kubectl set image deployment/myboot myboot=quay.io/rhdevelopers/myboot:v2

And notice the error free rolling update:

Aloha from Spring Boot! 131 on myboot-845968c6ff-k4rvb
Aloha from Spring Boot! 134 on myboot-845968c6ff-9wvt9
Aloha from Spring Boot! 122 on myboot-845968c6ff-9824z
Bonjour from Spring Boot! 0 on myboot-8449d5468d-m88z4
Bonjour from Spring Boot! 1 on myboot-8449d5468d-m88z4
Aloha from Spring Boot! 135 on myboot-845968c6ff-9wvt9
Aloha from Spring Boot! 133 on myboot-845968c6ff-k4rvb
Aloha from Spring Boot! 137 on myboot-845968c6ff-9wvt9
Bonjour from Spring Boot! 3 on myboot-8449d5468d-m88z4

Look at the Endpoints to see which pods are part of the Service:

kubectl get endpoints myboot -o json | jq '.subsets[].addresses[].ip'

These are the Pod IPs that have passed their readiness probes:

"10.129.2.40"
"10.130.2.37"
"10.130.2.38"

Readiness Probe

Exec into a single Pod and change its readiness flag:

kubectl exec -it myboot-845968c6ff-k5lcb /bin/bash

curl localhost:8080/misbehave
exit

See that the pod is no longer Ready:

NAME                      READY   STATUS    RESTARTS   AGE
myboot-845968c6ff-9wshg   1/1     Running   0          11m
myboot-845968c6ff-k5lcb   0/1     Running   0          12m
myboot-845968c6ff-zsgx2   1/1     Running   0          11m

Now check the Endpoints:

kubectl get endpoints myboot -o json | jq '.subsets[].addresses[].ip'

And that pod is now missing from the Service’s loadbalancer:

"10.130.2.37"
"10.130.2.38"

Which is also self-evident in the curl loop:

Aloha from Spring Boot! 845 on myboot-845968c6ff-9wshg
Aloha from Spring Boot! 604 on myboot-845968c6ff-zsgx2
Aloha from Spring Boot! 846 on myboot-845968c6ff-9wshg

Liveness Probe

kubectl set image deployment/myboot myboot=quay.io/rhdevelopers/myboot:v3

Let the rollout finish to completion across all 3 replicas:

watch kubectl get pods
NAME                      READY   STATUS    RESTARTS   AGE
myboot-56659c9d69-6sglj   1/1     Running   0          2m2s
myboot-56659c9d69-mdllq   1/1     Running   0          97s
myboot-56659c9d69-zjt6q   1/1     Running   0          72s

And as seen in the curl loop/poller:

Jambo from Spring Boot! 40 on myboot-56659c9d69-mdllq
Jambo from Spring Boot! 26 on myboot-56659c9d69-zjt6q
Jambo from Spring Boot! 71 on myboot-56659c9d69-6sglj

Edit the Deployment to point to the /alive URL:

If you’re running this tutorial from within VSCode or would like to use VSCode to edit the resource targeted, make sure you set the following environment variable before issuing the kubectl edit:

export KUBE_EDITOR="code -w"

kubectl edit deployment myboot

And change the liveness probe:

...
    spec:
      containers:
      - image: quay.io/rhdevelopers/myboot:v3
        imagePullPolicy: Always
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /alive
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 5
          successThreshold: 1
          timeoutSeconds: 2
        name: myboot
...

Save and close the editor, allowing that change to rollout:

watch kubectl get pods

NAME                      READY   STATUS        RESTARTS   AGE
myboot-558b4f8678-nw762   1/1     Running       0          59s
myboot-558b4f8678-qbrgc   1/1     Running       0          81s
myboot-558b4f8678-z7f9n   1/1     Running       0          36s

Now pick one of the pods, exec into it and shoot it:

kubectl exec -it myboot-558b4f8678-qbrgc /bin/bash

curl localhost:8080/shot

And you will see it get restarted:

NAME                      READY   STATUS    RESTARTS   AGE
myboot-558b4f8678-nw762   1/1     Running   0          4m7s
myboot-558b4f8678-qbrgc   1/1     Running   1          4m29s
myboot-558b4f8678-z7f9n   1/1     Running   0          3m44s

Plus, your exec will be terminated:

kubectl exec -it myboot-558b4f8678-qbrgc /bin/bash

curl localhost:8080/shot

I have been shot in the head1000610000@myboot-558b4f8678-qbrgc:/app$ command terminated with exit code 137

And your end-users will not see any errors:

Jambo from Spring Boot! 174 on myboot-558b4f8678-z7f9n
Jambo from Spring Boot! 11 on myboot-558b4f8678-qbrgc
Jambo from Spring Boot! 12 on myboot-558b4f8678-qbrgc
Jambo from Spring Boot! 206 on myboot-558b4f8678-nw762
Jambo from Spring Boot! 207 on myboot-558b4f8678-nw762
Jambo from Spring Boot! 175 on myboot-558b4f8678-z7f9n
Jambo from Spring Boot! 176 on myboot-558b4f8678-z7f9n

Clean up

kubectl delete deployment myboot

Startup Probe

Some applications require an additional startup time on their first initialization.

It might be tricky to fit this scenario into the liveness/readiness probes as you need to configure them for their normal behaviour to detect abnormalities during the running time and moreover covering the long start up time.

For instance, what if we had an application that might deadlock and we want to catch such issues immediately, we might have liveness and readiness probes that look like in apps/kubefiles/myboot-deployment-live-ready-aggressive.yml

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open myboot-deployment-live-ready-aggressive.yml

myboot-deployment-live-ready-aggressive.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myboot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myboot
  template:
    metadata:
      labels:
        app: myboot
        env: dev
    spec:
      containers:
      - name: myboot
        image: quay.io/rhdevelopers/myboot:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "300Mi"
            cpu: "250m" # 1/4 core
          limits:
            memory: "400Mi"
            cpu: "1000m" # 1 core
        livenessProbe:
          httpGet:
              port: 8080
              path: /alive
          periodSeconds: 2
          timeoutSeconds: 2
          failureThreshold: 2
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 3

Then apply that deployment

kubectl apply -f apps/kubefiles/myboot-deployment-live-ready-aggressive.yml

As we’ll see from the pod watch, the pods are continually getting restarted, sometimes after it successfully boots up (because kubelet schedules for restart) and this is due to the startup time of SpringBoot.

kubectl describe pods

Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  96s                 default-scheduler  Successfully assigned myspace/myboot-849ccd6948-8vrfq to devnation
  Normal   Pulled     92s                 kubelet            Successfully pulled image "quay.io/rhdevelopers/myboot:v1" in 3.295180194s
  Normal   Created    55s (x2 over 92s)   kubelet            Created container myboot
  Normal   Started    55s (x2 over 92s)   kubelet            Started container myboot
  Normal   Pulled     55s                 kubelet            Successfully pulled image "quay.io/rhdevelopers/myboot:v1" in 3.289395484s
  Warning  Unhealthy  52s (x4 over 90s)   kubelet            Liveness probe failed: Get "http://172.17.0.4:8080/alive": dial tcp 172.17.0.4:8080: connect: connection refused
  Normal   Killing    52s (x2 over 88s)   kubelet            Container myboot failed liveness probe, will be restarted
  Normal   Pulling    22s (x3 over 95s)   kubelet            Pulling image "quay.io/rhdevelopers/myboot:v1"
  Warning  Unhealthy  19s (x10 over 88s)  kubelet            Readiness probe failed: Get "http://172.17.0.4:8080/health": dial tcp 172.17.0.4:8080: connect: connection refused

Startup probes fix this problem, as once the startup probe has succeeded, the rest of the probes take over, but until the startup probe passes, neither the liveness nor the readiness probes can run.

myboot-deployment-startup-live-ready.yml is an example of a deployment with just such a probe

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open myboot-deployment-startup-live-ready.yml

myboot-deployment-startup-live-ready.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myboot
spec:
  replicas: 1
  selector:
    matchLabels:
      app: myboot
  template:
    metadata:
      labels:
        app: myboot
        env: dev
    spec:
      containers:
      - name: myboot
        image: quay.io/rhdevelopers/myboot:v1
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "300Mi"
            cpu: "250m" # 1/4 core
          limits:
            memory: "400Mi"
            cpu: "1000m" # 1 core
        livenessProbe:
          httpGet:
              port: 8080
              path: /alive
          periodSeconds: 2
          timeoutSeconds: 2
          failureThreshold: 2
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 3
        startupProbe:
          httpGet:
            path: /alive
            port: 8080
          failureThreshold: 6
          periodSeconds: 5
          timeoutSeconds: 1

You’ll see the difference is this section

        startupProbe:
          httpGet:
            path: /alive
            port: 8080
          failureThreshold: 6
          periodSeconds: 5
          timeoutSeconds: 1

Then apply that deployment

kubectl apply -f apps/kubefiles/myboot-deployment-startup-live-ready.yml

The startup probe waits for 30 seconds (5 * 6) to startup the application. Notice, too, that the delay on the liveness and readiness checks has gone down to 0.

watch kubectl get pods

NAME                      READY   STATUS    RESTARTS   AGE
myboot-579cc5cc47-2bk5p   0/1     Running   0          67s

Eventually your curl loop should show the pod running

Aloha from Spring Boot! 18 on myboot-849ccd6948-8vrfq
Aloha from Spring Boot! 19 on myboot-849ccd6948-8vrfq
Aloha from Spring Boot! 20 on myboot-849ccd6948-8vrfq
Aloha from Spring Boot! 21 on myboot-849ccd6948-8vrfq

Let’s show that the liveness probe has taken over. Now pick one of the pods, exec into it and shoot it:

kubectl exec -it myboot-558b4f8678-qbrgc /bin/bash

curl localhost:8080/shot

And you will see it get restarted.

Describe the pod to get the statistics of probes:

kubectl describe pod myboot-579cc5cc47-2bk5p

Limits:
  cpu:     1
  memory:  400Mi
Requests:
  cpu:        250m
  memory:     300Mi
Liveness:     http-get http://:8080/ delay=10s timeout=2s period=5s #success=1 #failure=3
Readiness:    http-get http://:8080/health delay=10s timeout=1s period=3s #success=1 #failure=3
Startup:      http-get http://:8080/alive delay=0s timeout=1s period=5s #success=1 #failure=12
Environment:  <none>
Mounts:

Clean Up

kubectl delete deployment myboot
kubectl delete svc myboot