Operators
Operators are a way of extending the functionality of our Kubernetes cluster by installing automated controllers to manage extensions we provide to the underlying Kubernetes API.
In this section we’ll take a deeper look at how operators interact with the Kubernetes API to do this
Preparation
Namespace
We’ll need a namespace where we’re house our operator deployment and our CustomResources
upon which the operator will operate
kubectl create namespace pizzahat
kubectl config set-context --current --namespace=pizzahat
Watch
If it’s not open already, you’ll want to have a terminal open (call it Terminal 2) to watch what’s going on with the pods in our current namespace
watch -n 1 "kubectl get pods -o wide \(1)
| awk '{print \$1 \" \" \$2 \" \" \$3 \" \" \$5 \" \" \$7}' | column -t" (2)
1 | the -o wide option allows us to see the node that the pod is schedule to |
2 | to keep the line from getting too long we’ll use awk and column to get and format only the columns we want |
CRDs
Custom Resources extend the API
Custom Controllers provide the functionality - continually maintains the desired state - to monitor its state and reconcile the resource to match with the configuration
https://kubernetes.io/docs/tasks/access-kubernetes-api/custom-resources/custom-resource-definitions/
Custom Resource Definitions (CRDs) in version 1.7
CRDs extend the Kubernetes API. We can see these api resources readily:
kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
bindings v1 true Binding
componentstatuses cs v1 false ComponentStatus
configmaps cm v1 true ConfigMap
endpoints ep v1 true Endpoint
... (1)
1 | This list is truncated |
In the list you will find some of the resources we’ve already learned about, like Deployments
kubectl api-resources | grep Deployment
deployments deploy apps/v1 true Deployment
CustomResourceDefinition
s are a sub-set of the Kubernetes api-resources
. Let’s see if there are any CRDs already installed in our cluster
kubectl get crds --all-namespaces
If you are using something like minikube, you will find that there are no CRDs installed yet
No resources found
NAME CREATED AT
alertmanagerconfigs.monitoring.coreos.com 2021-07-12T01:37:49Z
alertmanagers.monitoring.coreos.com 2021-07-12T01:37:53Z
apiservers.config.openshift.io 2021-07-12T01:37:06Z
authentications.config.openshift.io 2021-07-12T01:37:06Z
authentications.operator.openshift.io 2021-07-12T01:37:53Z
baremetalhosts.metal3.io 2021-07-12T01:38:25Z
builds.config.openshift.io 2021-07-12T01:37:06Z
catalogsources.operators.coreos.com 2021-07-12T01:37:49Z
cloudcredentials.operator.openshift.io 2021-07-12T01:37:10Z
... (1)
1 | This list has been truncated |
OpenShift is at its heart Kubernetes. One of the main ways OpenShift extends Kubernetes is via CRDs, which explains why you find so many of them installed even on the back of a fresh installation.
Example CRD
Let’s go ahead and create our own Custom Resource Definition. Later on, this Custom Resources created from this definition will be something that our operator will operate upon. Take a look at pizza-crd.yaml
to see what the CRD we’ll be creating looks like
If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open |
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
name: pizzas.mykubernetes.acme.org
labels:
app: pizzamaker
mylabel: stuff
spec:
group: mykubernetes.acme.org
scope: Namespaced
versions:
- name: v1
served: true
storage: true
schema:
openAPIV3Schema:
description: "A custom resource for making yummy pizzas" (1)
type: object
properties:
spec:
type: object
description: "Information about our pizza"
properties:
toppings: (2)
type: array
items:
type: string
description: "List of toppings for our pizza"
sauce: (3)
type: string
description: "The name of the sauce to use on our pizza"
names:
kind: Pizza (4)
listKind: PizzaList
plural: pizzas
singular: pizza
shortNames:
- pz
1 | This is a description that will be shown when somebody attempts to describe the CRD |
2 | This describes one of the values our CustomResource will have, namely, the (array ) list of (string ) toppings |
3 | This describes the second field our CustomResource can define in its spec, the (string ) name of the sauce to use |
4 | This is the name that our CustomResources will have. Sort of like Deployment or Pod |
Many CRDs include metadata about the fields that are exposed so that the CR can be validated by the Kubernetes API. Prior to API version |
Now let’s go ahead ad add this CRD to our cluster so that we can create Pizza
Custom Resources.
kubectl apply -f apps/pizzas/pizza-crd.yaml
We should now be able to see that our CRD is part of our API
kubectl get crds | grep pizza
Results:
NAME CREATED AT
pizzas.mykubernetes.acme.org 2020-07-01T08:12:00Z
And since CRDs are a subset of all api-resources
, we should now see pizzas
as extending our cluster’s api-resources:
kubectl api-resources | grep pizzas
Yields:
pizzas pz mykubernetes.acme.org true Pizza
Finally, since we defined the schema for our CustomResourceDefinition
we’ve made it easier for people to consume our api. CRDs hook into the kubectl describe
functionality
kubectl explain pizza
Gives us this helpful output
KIND: Pizza
VERSION: mykubernetes.acme.org/v1
DESCRIPTION:
A custom resource for making yummy pizzas (1)
FIELDS:
apiVersion <string>
APIVersion defines the versioned schema of this representation of an
object. Servers should convert recognized schemas to the latest internal
value, and may reject unrecognized values. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind <string>
Kind is a string value representing the REST resource this object
represents. Servers may infer this from the endpoint the client submits
requests to. Cannot be updated. In CamelCase. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata <Object>
Standard object's metadata. More info:
https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
sauce <string> (2)
The name of the sauce to use on our pizza
toppings <[]string> (3)
List of toppings for our pizza'
1 | Notice that this matches our overall description of the pizza |
2 | This is from the Schema section of the CRD for sauce. It says that it’s a string. The description comes from the description field |
3 | This is from the Schema section of the CRD for toppings. It says that it’s an array of strings. The description comes from the description field |
Deploying the Operator
Our CRD is not limited to a particular namespace, but we do need a namespace to put our operator that is going to operate on our pizza
CRs.
At its heart, an operator is just an application, like the myboot
application that we deployed previously. The difference is that the operator knows to to interact with the Kubernetes API and watch for resources that it cares about.
The Pizza operator that we’re about to deploy was written in Quarkus using the java operator sdk. The code for this operator is present in this repo. See the PizzaResourceWatcher.java
which is one of the key classes in the operator controller:
If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open |
package org.acme;
import io.fabric8.kubernetes.api.model.ContainerBuilder;
import io.fabric8.kubernetes.api.model.ObjectMetaBuilder;
import io.fabric8.kubernetes.api.model.Pod;
import io.fabric8.kubernetes.api.model.PodBuilder;
import io.fabric8.kubernetes.api.model.PodSpecBuilder;
import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientException;
import io.fabric8.kubernetes.client.Watcher;
import io.fabric8.kubernetes.client.dsl.NonNamespaceOperation;
import io.fabric8.kubernetes.client.dsl.Resource;
import io.quarkus.runtime.StartupEvent;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import javax.enterprise.event.Observes;
import javax.inject.Inject;
public class PizzaResourceWatcher {
@Inject
KubernetesClient defaultClient;
@Inject
NonNamespaceOperation<PizzaResource, PizzaResourceList, PizzaResourceDoneable, Resource<PizzaResource, PizzaResourceDoneable>> crClient;
void onStartup(@Observes StartupEvent event) {
System.out.println("Startup");
crClient.watch(new Watcher<PizzaResource>() { (1)
@Override
public void eventReceived(Action action, PizzaResource resource) {
System.out.println("Event " + action.name());
if (action == Action.ADDED) {
final String app = resource.getMetadata().getName();
final String sauce = resource.getSpec().getSauce();
final List<String> toppings = resource.getSpec().getToppings();
final Map<String, String> labels = new HashMap<>();
labels.put("app", app);
final ObjectMetaBuilder objectMetaBuilder = new ObjectMetaBuilder().withName(app + "-pod")
.withNamespace(resource.getMetadata().getNamespace()).withLabels(labels);
final ContainerBuilder containerBuilder = new ContainerBuilder().withName("pizza-maker")
.withImage("quay.io/lordofthejars/pizza-maker:1.0.0").withCommand("/work/application")
.withArgs("--sauce=" + sauce, "--toppings=" + String.join(",", toppings));
final PodSpecBuilder podSpecBuilder = new PodSpecBuilder().withContainers(containerBuilder.build())
.withRestartPolicy("Never");
final PodBuilder podBuilder = new PodBuilder().withMetadata(objectMetaBuilder.build())
.withSpec(podSpecBuilder.build());
final Pod pod = podBuilder.build();
defaultClient.resource(pod).createOrReplace();
}
}
@Override
public void onClose(KubernetesClientException e) {
}
});
}
}
1 | Notice that it’s watching for our custom resource of Pizza |
The creation of an operator controller is outside the scope of this tutorial. If you’d like to learn more about creating operators with Quarkus, watch this 20 minute tutorial |
kubectl apply -f apps/pizzas/pizza-deployment.yaml
Soon in your watch window (Terminal 2) you should see something like this
NAME READY STATUS RESTARTS AGE
quarkus-operator-example-5f5bf777bc-glfg9 1/1 Running 0 58s
Wait until the deployment |
Make some Pizzas
Once our operator is running, it will be on the lookout for information in our Pizza
Custom Resources and use it to (pretend to) make some pizzas by spinning up a pod configurated with information from the Custom Resource instance.
For example, consider this instance of the Pizza CustomResourceDefinition
:
Pay special attention to:
-
Sauce:
regular
-
Toppings:
mozzarella
Now let’s create this CustomResource
:
kubectl apply -f apps/pizzas/cheese-pizza.yaml
kubectl get pizzas
NAME AGE
cheesep 4s
kubectl describe pizza cheesep
Name: cheesep
Namespace: pizzahat
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"mykubernetes.acme.org/v1beta2","kind":"Pizza","metadata":{"annotations":{},"name":"cheesep","namespace":"pizzahat"},"spec":...
API Version: mykubernetes.acme.org/v1beta2
Kind: Pizza
...
And in our Terminal 2 we should see how the Operator responds…
NAME READY STATUS RESTARTS AGE
cheesep-pod 0/1 Completed 0 3s
quarkus-operator-example-5f5bf777bc-glfg9 1/1 Running 0 44m
And once the cheesep-pod
completes we should see the following in Terminal 3
+ cheesep-pod › pizza-maker
cheesep-pod pizza-maker __ ____ __ _____ ___ __ ____ ______
cheesep-pod pizza-maker --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
cheesep-pod pizza-maker -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
cheesep-pod pizza-maker --\___\_\____/_/ |_/_/|_/_/|_|\____/___/
cheesep-pod pizza-maker 2021-07-19 08:16:26,113 INFO [io.quarkus] (main) pizza-maker 1.0-SNAPSHOT (powered by Quarkus 1.4.0.CR1) started in 1.063s.
cheesep-pod pizza-maker 2021-07-19 08:16:26,114 INFO [io.quarkus] (main) Profile prod activated.
cheesep-pod pizza-maker 2021-07-19 08:16:26,114 INFO [io.quarkus] (main) Installed features: [cdi]
cheesep-pod pizza-maker Doing The Base
cheesep-pod pizza-maker Adding Sauce regular
cheesep-pod pizza-maker Adding Toppings [mozzarella]
cheesep-pod pizza-maker Baking
cheesep-pod pizza-maker Baked
cheesep-pod pizza-maker Ready For Delivery
cheesep-pod pizza-maker 2021-07-19 08:16:26,615 INFO [io.quarkus] (main) pizza-maker stopped in 0.000s
Notice that Sauce and Toppings matches what was specified in the pizza
CustomResource
Make more Pizzas
Take a look at meat-pizza.yaml
and veggie-lovers.yaml
to show the sauce and toppings options there
If you’re running this from within VSCode you can use CTRL+p (or CMD+p on Mac OSX) to quickly open |
apiVersion: mykubernetes.acme.org/v1
kind: Pizza
metadata:
name: meatsp
spec:
toppings:
- mozzarella
- pepperoni
- sausage
- bacon
sauce: extra
Now make the pizzas
kubectl apply -f apps/pizzas/meat-pizza.yaml
kubectl apply -f apps/pizzas/veggie-lovers.yaml
kubectl get pizzas --all-namespaces
Pod watch in the Terminal 2 should show
NAME READY STATUS AGE NODE
cheesep-pod 0/1 Completed 8m46s devnation
meatsp-pod 0/1 ContainerCreating 8s devnation
quarkus-operator-example-fdb76c946-cwmnq 1/1 Running 14m devnation
veggiep-pod 0/1 ContainerCreating 6s devnation
And this notice in our log terminal Terminal 3
+ meatsp-pod › pizza-maker
meatsp-pod pizza-maker __ ____ __ _____ ___ __ ____ ______
meatsp-pod pizza-maker --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
meatsp-pod pizza-maker -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
meatsp-pod pizza-maker --\___\_\____/_/ |_/_/|_/_/|_|\____/___/
meatsp-pod pizza-maker 2021-07-19 08:24:48,015 INFO [io.quarkus] (main) pizza-maker 1.0-SNAPSHOT (powered by Quarkus 1.4.0.CR1) started in 0.817s.
meatsp-pod pizza-maker 2021-07-19 08:24:48,016 INFO [io.quarkus] (main) Profile prod activated.
meatsp-pod pizza-maker 2021-07-19 08:24:48,016 INFO [io.quarkus] (main) Installed features: [cdi]
meatsp-pod pizza-maker Doing The Base
meatsp-pod pizza-maker Adding Sauce extra (1)
meatsp-pod pizza-maker Adding Toppings [mozzarella,pepperoni,sausage,bacon]
meatsp-pod pizza-maker Baking
meatsp-pod pizza-maker Baked
meatsp-pod pizza-maker Ready For Delivery
meatsp-pod pizza-maker 2021-07-19 08:24:48,517 INFO [io.quarkus] (main) pizza-maker stopped in 0.000s
+ veggiep-pod › pizza-maker
veggiep-pod pizza-maker __ ____ __ _____ ___ __ ____ ______
veggiep-pod pizza-maker --/ __ \/ / / / _ | / _ \/ //_/ / / / __/
veggiep-pod pizza-maker -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
veggiep-pod pizza-maker --\___\_\____/_/ |_/_/|_/_/|_|\____/___/
veggiep-pod pizza-maker 2021-07-19 08:24:55,289 INFO [io.quarkus] (main) pizza-maker 1.0-SNAPSHOT (powered by Quarkus 1.4.0.CR1) started in 0.869s.
veggiep-pod pizza-maker 2021-07-19 08:24:55,289 INFO [io.quarkus] (main) Profile prod activated.
veggiep-pod pizza-maker 2021-07-19 08:24:55,289 INFO [io.quarkus] (main) Installed features: [cdi]
veggiep-pod pizza-maker Doing The Base
veggiep-pod pizza-maker Adding Sauce extra (2)
veggiep-pod pizza-maker Adding Toppings [mozzarella,black olives]
veggiep-pod pizza-maker Baking
veggiep-pod pizza-maker Baked
veggiep-pod pizza-maker Ready For Delivery
veggiep-pod pizza-maker 2021-07-19 08:24:55,790 INFO [io.quarkus] (main) pizza-maker stopped in 0.000s
1 | Matches sauce and toppings on the meat-pizza CR |
2 | Matches sauce and toppings on the veggie-lovers CR |
Cleanup
Let’s cleanup everything in our namespace
kubectl delete all --all (1)
kubectl delete ns pizzahat
1 | Whilst namespaces do tend to automatically cleanup the resources within them, it’s usually good practice to empty them out first to ensure you don’t have any finalizer issues |
pizza.mykubernetes.acme.org "cheesep" deleted
pizza.mykubernetes.acme.org "meatsp" deleted
pizza.mykubernetes.acme.org "veggiep" deleted
pod "cheesep-pod" deleted
pod "meatsp-pod" deleted
pod "quarkus-operator-example-fdb76c946-cwmnq" deleted
pod "veggiep-pod" deleted
deployment.apps "quarkus-operator-example" deleted
namespace "pizzahat" deleted
And finally, let’s remove our CRD (which was not bound to a specific namespace like section-namespace
)
kubectl delete crd pizzas.mykubernetes.acme.org (1)
1 | When deleting a crd we need to refer to it by its fully qualified name |
customresourcedefinition.apiextensions.k8s.io "pizzas.mykubernetes.acme.org" deleted
Create some Kafka
Kafka for Minikube
Create a new namespace for this experiment:
kubectl create namespace franz
kubectl config set-context --current --namespace=franz
For minikube, the instructions for installation can be found here:
What follows were the instructions from a moment in time:
curl -sL https://github.com/operator-framework/operator-lifecycle-manager/releases/download/0.14.1/install.sh | bash -s 0.14.1
kubectl create -f https://operatorhub.io/install/strimzi-kafka-operator.yaml
Verify Install
kubectl get csv -n operators
kubectl get crds | grep kafka
Start a watch in another terminal:
watch kubectl get pods
Then deploy the resource requesting a Kafka cluster:
kubectl apply -f apps/kubefiles/mykafka.yml
NAME READY STATUS RESTARTS AGE
my-cluster-entity-operator-66676cb9fb-fzckz 2/2 Running 0 29s
my-cluster-kafka-0 2/2 Running 0 60s
my-cluster-kafka-1 2/2 Running 0 60s
my-cluster-kafka-2 2/2 Running 0 60s
my-cluster-zookeeper-0 2/2 Running 0 92s
my-cluster-zookeeper-1 2/2 Running 0 92s
my-cluster-zookeeper-2 2/2 Running 0 92s
And you can get all information from Kafka:
kubectl get kafkas
NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS
my-cluster 3 3