Operators

Operators are a way of extending the functionality of our Kubernetes cluster by installing automated controllers to manage extensions we provide to the underlying Kubernetes API.

In this section we’ll take a deeper look at how operators interact with the Kubernetes API to do this

Operators in the Real World

When demonstrating this tutorial in a master class, it can be good to show the Kafka Operator in Openshift (as roughly outlined here). Key Points when showing on an OpenShift cluster:

  1. Use OperatorHub to show how many different Operators there are.

  2. Install the AMQStreams or Strimzi Operator to add Kafka support (i.e. CRDs as we’ll see) to the cluster

  3. Once the operator is installed, pick a namespace to install a Kafka CR in

  4. Show in the Developer Perspective the Kafka being created by the operator

In this section of the tutorial we’ll be demonstrating these aspects of operators with a home grown toy "Pizza Operator"

Preparation

Namespace

We’ll need a namespace where we’re house our operator deployment and our CustomResources upon which the operator will operate

kubectl create namespace pizzahat
kubectl config set-context --current --namespace=pizzahat

Watch

If it’s not open already, you’ll want to have a terminal open (call it Terminal 2) to watch what’s going on with the pods in our current namespace

  • Terminal 2

watch -n 1 "kubectl get pods -o wide \(1)
  | awk '{print \$1 \" \" \$2 \" \" \$3 \" \" \$5 \" \" \$7}' | column -t" (2)
1 the -o wide option allows us to see the node that the pod is schedule to
2 to keep the line from getting too long we’ll use awk and column to get and format only the columns we want

Logs

We’ll want to open a third terminal (call it Terminal 3) where we’ll use a tool called stern to watch the output of certain pods

VSCode: Open Terminal in Editor

If you’re doing this tutorial from within VSCode you may be running out of space to put your terminals at this point! If you have a recent release of VSCode, you might consider opening a new terminal in the editor pane by using CTRL+SHIFT+p (or CMD+SHIFT+p on Mac OSX) to run the Terminal: Create Terminal in Editor Area command

We are going to have stern watch the pizzahat namespace for p-pod

  • Terminal 3

stern -n pizzahat p-pod

CRDs

Custom Resources extend the API

Custom Controllers provide the functionality - continually maintains the desired state - to monitor its state and reconcile the resource to match with the configuration

Custom Resource Definitions (CRDs) in version 1.7

CRDs extend the Kubernetes API. We can see these api resources readily:

kubectl api-resources
NAME                              SHORTNAMES   APIVERSION                             NAMESPACED   KIND
bindings                                       v1                                     true         Binding
componentstatuses                 cs           v1                                     false        ComponentStatus
configmaps                        cm           v1                                     true         ConfigMap
endpoints                         ep           v1                                     true         Endpoint
... (1)
1 This list is truncated

In the list you will find some of the resources we’ve already learned about, like Deployments

kubectl api-resources | grep Deployment
deployments                       deploy       apps/v1                                true         Deployment

CustomResourceDefinition s are a sub-set of the Kubernetes api-resources. Let’s see if there are any CRDs already installed in our cluster

kubectl get crds --all-namespaces
  • Minikube

  • OpenShift

If you are using something like minikube, you will find that there are no CRDs installed yet

No resources found
NAME                                                              CREATED AT
alertmanagerconfigs.monitoring.coreos.com                         2021-07-12T01:37:49Z
alertmanagers.monitoring.coreos.com                               2021-07-12T01:37:53Z
apiservers.config.openshift.io                                    2021-07-12T01:37:06Z
authentications.config.openshift.io                               2021-07-12T01:37:06Z
authentications.operator.openshift.io                             2021-07-12T01:37:53Z
baremetalhosts.metal3.io                                          2021-07-12T01:38:25Z
builds.config.openshift.io                                        2021-07-12T01:37:06Z
catalogsources.operators.coreos.com                               2021-07-12T01:37:49Z
cloudcredentials.operator.openshift.io                            2021-07-12T01:37:10Z
... (1)
1 This list has been truncated

OpenShift is at its heart Kubernetes. One of the main ways OpenShift extends Kubernetes is via CRDs, which explains why you find so many of them installed even on the back of a fresh installation.

Example CRD

Let’s go ahead and create our own Custom Resource Definition. Later on, this Custom Resources created from this definition will be something that our operator will operate upon. Take a look at pizza-crd.yaml to see what the CRD we’ll be creating looks like

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open pizza-crd.yaml

pizza-crd.yaml
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: pizzas.mykubernetes.acme.org
  labels:
    app: pizzamaker
    mylabel: stuff
spec:
  group: mykubernetes.acme.org
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        description: "A custom resource for making yummy pizzas" (1)
        type: object
        properties:
          spec:
            type: object
            description: "Information about our pizza"
            properties:
              toppings: (2)
                type: array
                items:
                  type: string
                description: "List of toppings for our pizza"
              sauce: (3)
                type: string
                description: "The name of the sauce to use on our pizza"
  names:
    kind: Pizza (4)
    listKind: PizzaList
    plural: pizzas
    singular: pizza
    shortNames:
    - pz
1 This is a description that will be shown when somebody attempts to describe the CRD
2 This describes one of the values our CustomResource will have, namely, the (array) list of (string) toppings
3 This describes the second field our CustomResource can define in its spec, the (string) name of the sauce to use
4 This is the name that our CustomResources will have. Sort of like Deployment or Pod

Many CRDs include metadata about the fields that are exposed so that the CR can be validated by the Kubernetes API. Prior to API version v1 this was not enforced, after Kubernetes v1.22 all CustomResources will need to be v1 and thus will need to define their object schema

Now let’s go ahead ad add this CRD to our cluster so that we can create Pizza Custom Resources.

kubectl apply -f apps/pizzas/pizza-crd.yaml

We should now be able to see that our CRD is part of our API

kubectl get crds | grep pizza

Results:

NAME                           CREATED AT
pizzas.mykubernetes.acme.org   2020-07-01T08:12:00Z

And since CRDs are a subset of all api-resources, we should now see pizzas as extending our cluster’s api-resources:

kubectl api-resources | grep pizzas

Yields:

pizzas                            pz           mykubernetes.acme.org          true         Pizza

Finally, since we defined the schema for our CustomResourceDefinition we’ve made it easier for people to consume our api. CRDs hook into the kubectl describe functionality

kubectl explain pizza

Gives us this helpful output

KIND:     Pizza
VERSION:  mykubernetes.acme.org/v1

DESCRIPTION:
     A custom resource for making yummy pizzas (1)

FIELDS:
   apiVersion   <string>
     APIVersion defines the versioned schema of this representation of an
     object. Servers should convert recognized schemas to the latest internal
     value, and may reject unrecognized values. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources

   kind <string>
     Kind is a string value representing the REST resource this object
     represents. Servers may infer this from the endpoint the client submits
     requests to. Cannot be updated. In CamelCase. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds

   metadata     <Object>
     Standard object's metadata. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata

   sauce        <string> (2)
     The name of the sauce to use on our pizza

   toppings     <[]string> (3)
     List of toppings for our pizza'
1 Notice that this matches our overall description of the pizza
2 This is from the Schema section of the CRD for sauce. It says that it’s a string. The description comes from the description field
3 This is from the Schema section of the CRD for toppings. It says that it’s an array of strings. The description comes from the description field

Deploying the Operator

Our CRD is not limited to a particular namespace, but we do need a namespace to put our operator that is going to operate on our pizza CRs.

At its heart, an operator is just an application, like the myboot application that we deployed previously. The difference is that the operator knows to to interact with the Kubernetes API and watch for resources that it cares about.

The Pizza operator that we’re about to deploy was written in Quarkus using the java operator sdk. The code for this operator is present in this repo. See the PizzaResourceWatcher.java which is one of the key classes in the operator controller:

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open PizzaResourceWatcher.java

PizzaResourceWatcher.java
package org.acme;

import io.fabric8.kubernetes.api.model.ContainerBuilder;
import io.fabric8.kubernetes.api.model.ObjectMetaBuilder;
import io.fabric8.kubernetes.api.model.Pod;
import io.fabric8.kubernetes.api.model.PodBuilder;
import io.fabric8.kubernetes.api.model.PodSpecBuilder;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientException;
import io.fabric8.kubernetes.client.Watcher;
import io.fabric8.kubernetes.client.dsl.NonNamespaceOperation;
import io.fabric8.kubernetes.client.dsl.Resource;
import io.quarkus.runtime.StartupEvent;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.enterprise.event.Observes;
import javax.inject.Inject;

public class PizzaResourceWatcher {

    @Inject
    KubernetesClient defaultClient;

    @Inject
    NonNamespaceOperation<PizzaResource, PizzaResourceList, PizzaResourceDoneable, Resource<PizzaResource, PizzaResourceDoneable>> crClient;

    void onStartup(@Observes StartupEvent event) {
        System.out.println("Startup");
        crClient.watch(new Watcher<PizzaResource>() { (1)
            @Override
            public void eventReceived(Action action, PizzaResource resource) {
                System.out.println("Event " + action.name());
                if (action == Action.ADDED) {
                    final String app = resource.getMetadata().getName();
                    final String sauce = resource.getSpec().getSauce();
                    final List<String> toppings = resource.getSpec().getToppings();
                    final Map<String, String> labels = new HashMap<>();
                    labels.put("app", app);
                    final ObjectMetaBuilder objectMetaBuilder = new ObjectMetaBuilder().withName(app + "-pod")
                            .withNamespace(resource.getMetadata().getNamespace()).withLabels(labels);
                    final ContainerBuilder containerBuilder = new ContainerBuilder().withName("pizza-maker")
                            .withImage("quay.io/lordofthejars/pizza-maker:1.0.0").withCommand("/work/application")
                            .withArgs("--sauce=" + sauce, "--toppings=" + String.join(",", toppings));
                    final PodSpecBuilder podSpecBuilder = new PodSpecBuilder().withContainers(containerBuilder.build())
                            .withRestartPolicy("Never");
                    final PodBuilder podBuilder = new PodBuilder().withMetadata(objectMetaBuilder.build())
                            .withSpec(podSpecBuilder.build());
                    final Pod pod = podBuilder.build();
                    defaultClient.resource(pod).createOrReplace();

                }
            }

            @Override
            public void onClose(KubernetesClientException e) {
            }
        });
    }

}
1 Notice that it’s watching for our custom resource of Pizza

The creation of an operator controller is outside the scope of this tutorial. If you’d like to learn more about creating operators with Quarkus, watch this 20 minute tutorial

kubectl apply -f apps/pizzas/pizza-deployment.yaml

Soon in your watch window (Terminal 2) you should see something like this

  • Terminal 2

NAME                                        READY   STATUS    RESTARTS   AGE
quarkus-operator-example-5f5bf777bc-glfg9   1/1     Running   0          58s

Wait until the deployment STATUS of the operator is Running before moving on to the next section

Make some Pizzas

Once our operator is running, it will be on the lookout for information in our Pizza Custom Resources and use it to (pretend to) make some pizzas by spinning up a pod configurated with information from the Custom Resource instance.

For example, consider this instance of the Pizza CustomResourceDefinition:

Pay special attention to:

  • Sauce: regular

  • Toppings: mozzarella

Now let’s create this CustomResource:

kubectl apply -f apps/pizzas/cheese-pizza.yaml
kubectl get pizzas
NAME      AGE
cheesep   4s
kubectl describe pizza cheesep
Name:         cheesep
Namespace:    pizzahat
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"mykubernetes.acme.org/v1beta2","kind":"Pizza","metadata":{"annotations":{},"name":"cheesep","namespace":"pizzahat"},"spec":...
API Version:  mykubernetes.acme.org/v1beta2
Kind:         Pizza
...

And in our Terminal 2 we should see how the Operator responds…​

  • Terminal 2

NAME                                        READY   STATUS      RESTARTS   AGE
cheesep-pod                                 0/1     Completed   0          3s
quarkus-operator-example-5f5bf777bc-glfg9   1/1     Running     0          44m

And once the cheesep-pod completes we should see the following in Terminal 3

  • Terminal 3

+ cheesep-pod › pizza-maker
cheesep-pod pizza-maker __  ____  __  _____   ___  __ ____  ______ 
cheesep-pod pizza-maker  --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
cheesep-pod pizza-maker  -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
cheesep-pod pizza-maker --\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
cheesep-pod pizza-maker 2021-07-19 08:16:26,113 INFO  [io.quarkus] (main) pizza-maker 1.0-SNAPSHOT (powered by Quarkus 1.4.0.CR1) started in 1.063s.
cheesep-pod pizza-maker 2021-07-19 08:16:26,114 INFO  [io.quarkus] (main) Profile prod activated.
cheesep-pod pizza-maker 2021-07-19 08:16:26,114 INFO  [io.quarkus] (main) Installed features: [cdi]
cheesep-pod pizza-maker Doing The Base
cheesep-pod pizza-maker Adding Sauce regular
cheesep-pod pizza-maker Adding Toppings [mozzarella]
cheesep-pod pizza-maker Baking
cheesep-pod pizza-maker Baked
cheesep-pod pizza-maker Ready For Delivery
cheesep-pod pizza-maker 2021-07-19 08:16:26,615 INFO  [io.quarkus] (main) pizza-maker stopped in 0.000s

Notice that Sauce and Toppings matches what was specified in the pizza CustomResource

Make more Pizzas

Take a look at meat-pizza.yaml and veggie-lovers.yaml to show the sauce and toppings options there

If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open meat-pizza.yaml

meat-pizza.yaml
apiVersion: mykubernetes.acme.org/v1
kind: Pizza
metadata:
  name: meatsp
spec:
  toppings:
  - mozzarella
  - pepperoni
  - sausage
  - bacon
  sauce: extra

Now make the pizzas

kubectl apply -f apps/pizzas/meat-pizza.yaml
kubectl apply -f apps/pizzas/veggie-lovers.yaml
kubectl get pizzas --all-namespaces

Pod watch in the Terminal 2 should show

  • Terminal 2

NAME                                      READY  STATUS             AGE    NODE
cheesep-pod                               0/1    Completed          8m46s  devnation
meatsp-pod                                0/1    ContainerCreating  8s     devnation
quarkus-operator-example-fdb76c946-cwmnq  1/1    Running            14m    devnation
veggiep-pod                               0/1    ContainerCreating  6s     devnation

And this notice in our log terminal Terminal 3

  • Terminal 3

+ meatsp-pod › pizza-maker
meatsp-pod pizza-maker __  ____  __  _____   ___  __ ____  ______ 
meatsp-pod pizza-maker  --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
meatsp-pod pizza-maker  -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
meatsp-pod pizza-maker --\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
meatsp-pod pizza-maker 2021-07-19 08:24:48,015 INFO  [io.quarkus] (main) pizza-maker 1.0-SNAPSHOT (powered by Quarkus 1.4.0.CR1) started in 0.817s.
meatsp-pod pizza-maker 2021-07-19 08:24:48,016 INFO  [io.quarkus] (main) Profile prod activated.
meatsp-pod pizza-maker 2021-07-19 08:24:48,016 INFO  [io.quarkus] (main) Installed features: [cdi]
meatsp-pod pizza-maker Doing The Base
meatsp-pod pizza-maker Adding Sauce extra (1)
meatsp-pod pizza-maker Adding Toppings [mozzarella,pepperoni,sausage,bacon]
meatsp-pod pizza-maker Baking
meatsp-pod pizza-maker Baked
meatsp-pod pizza-maker Ready For Delivery
meatsp-pod pizza-maker 2021-07-19 08:24:48,517 INFO  [io.quarkus] (main) pizza-maker stopped in 0.000s
+ veggiep-pod › pizza-maker
veggiep-pod pizza-maker __  ____  __  _____   ___  __ ____  ______ 
veggiep-pod pizza-maker  --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
veggiep-pod pizza-maker  -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
veggiep-pod pizza-maker --\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
veggiep-pod pizza-maker 2021-07-19 08:24:55,289 INFO  [io.quarkus] (main) pizza-maker 1.0-SNAPSHOT (powered by Quarkus 1.4.0.CR1) started in 0.869s.
veggiep-pod pizza-maker 2021-07-19 08:24:55,289 INFO  [io.quarkus] (main) Profile prod activated.
veggiep-pod pizza-maker 2021-07-19 08:24:55,289 INFO  [io.quarkus] (main) Installed features: [cdi]
veggiep-pod pizza-maker Doing The Base
veggiep-pod pizza-maker Adding Sauce extra (2)
veggiep-pod pizza-maker Adding Toppings [mozzarella,black olives]
veggiep-pod pizza-maker Baking
veggiep-pod pizza-maker Baked
veggiep-pod pizza-maker Ready For Delivery
veggiep-pod pizza-maker 2021-07-19 08:24:55,790 INFO  [io.quarkus] (main) pizza-maker stopped in 0.000s
1 Matches sauce and toppings on the meat-pizza CR
2 Matches sauce and toppings on the veggie-lovers CR

Cleanup

Let’s cleanup everything in our namespace

kubectl delete all --all (1)
kubectl delete ns pizzahat
1 Whilst namespaces do tend to automatically cleanup the resources within them, it’s usually good practice to empty them out first to ensure you don’t have any finalizer issues
pizza.mykubernetes.acme.org "cheesep" deleted
pizza.mykubernetes.acme.org "meatsp" deleted
pizza.mykubernetes.acme.org "veggiep" deleted
pod "cheesep-pod" deleted
pod "meatsp-pod" deleted
pod "quarkus-operator-example-fdb76c946-cwmnq" deleted
pod "veggiep-pod" deleted
deployment.apps "quarkus-operator-example" deleted
namespace "pizzahat" deleted

And finally, let’s remove our CRD (which was not bound to a specific namespace like section-namespace)

kubectl delete crd pizzas.mykubernetes.acme.org (1)
1 When deleting a crd we need to refer to it by its fully qualified name
customresourcedefinition.apiextensions.k8s.io "pizzas.mykubernetes.acme.org" deleted

Create some Kafka

Kafka for Minikube

Create a new namespace for this experiment:

kubectl create namespace franz
kubectl config set-context --current --namespace=franz

For minikube, the instructions for installation can be found here:

What follows were the instructions from a moment in time:

Kafka for OpenShift

OperatorHub in OpenShift

Verify Install

kubectl get csv -n operators
kubectl get crds | grep kafka

Start a watch in another terminal:

watch kubectl get pods

Then deploy the resource requesting a Kafka cluster:

kubectl apply -f apps/kubefiles/mykafka.yml
NAME                                          READY   STATUS    RESTARTS   AGE
my-cluster-entity-operator-66676cb9fb-fzckz   2/2     Running   0          29s
my-cluster-kafka-0                            2/2     Running   0          60s
my-cluster-kafka-1                            2/2     Running   0          60s
my-cluster-kafka-2                            2/2     Running   0          60s
my-cluster-zookeeper-0                        2/2     Running   0          92s
my-cluster-zookeeper-1                        2/2     Running   0          92s
my-cluster-zookeeper-2                        2/2     Running   0          92s

And you can get all information from Kafka:

kubectl get kafkas
NAME         DESIRED KAFKA REPLICAS   DESIRED ZK REPLICAS
my-cluster   3                        3

Clean up

kubectl delete namespace pizzahat
kubectl delete -f apps/pizzas/pizza-crd.yaml
kubectl delete kafka my-cluster
kubectl delete namespace franz