Go Operator

In this section we implement an Operator in Go which will define some Custom Resource to control and deploy a 3-tier app:

  • Frontend: React app from docker.io/jdob/visitors-webui:1.0.0

  • Backend: Python app from docker.io/jdob/visitors-service:1.0.0

  • DB: MySQL 5.7 from docker.io/library/mysql:5.7

Scaffold the new operator

Create a new directory on your machine, for instance : $HOME/visitors-operator

operator-sdk init --domain redhat.com --repo github.com/redhat-scholars/visitors-operator

Generate an API

Le’s create an API for our operator :

operator-sdk create api --group=app --version=v1 --kind=VisitorsApp --resource --controller

This will scaffold your operator’s resource API at $HOME/visitors-operator/api/v1/visitorsapp_types.go and the controller at $HOME/visitors-operator/controllers/visitorsapp_controller.go.

In general, it’s recommended to have one controller responsible for manage each API created for the project to properly follow the design goals set by controller-runtime.

API definition

To begin, we will represent our API by defining the VisitorApp type, which will have a VisitorAppSpec.Size field to set the quantity of memcached instances (CRs) to be deployed, and a Title

Open generated API: $HOME/visitors-operator/api/v1/visitorsapp_types.go

Fill these sections with the following:

type VisitorsAppSpec struct {
        // INSERT ADDITIONAL SPEC FIELDS - desired state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        Size  int32  `json:"size"`
        Title string `json:"title"`
}
// VisitorsAppStatus defines the observed state of VisitorsApp
type VisitorsAppStatus struct {
        // INSERT ADDITIONAL STATUS FIELD - define observed state of cluster
        // Important: Run "make" to regenerate code after modifying this file

        BackendImage  string `json:"backendImage"`
        FrontendImage string `json:"frontendImage"`
}

You can refer to the whole file in $TUTORIAL_HOME/apps/go/api/v1/visitorsapp_types.go.

After modifying the *_types.go file always run the following command to update the generated code for that resource type:

make generate

The above makefile target will invoke the controller-gen utility to update the api/v1/zz_generated.deepcopy.go file to ensure our API’s Go type definitons implement the runtime.Object interface that all Kind types must implement.

Generate CRDs

make manifests

This will invoke controller-gen utility to generate the CRD manifests at config/crd/bases/app.redhat.com_visitorsapps.yaml.

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  annotations:
    controller-gen.kubebuilder.io/version: v0.6.1
  creationTimestamp: null
  name: visitorsapps.app.redhat.com
spec:
  group: app.redhat.com
  names:
    kind: VisitorsApp
    listKind: VisitorsAppList
    plural: visitorsapps
    singular: visitorsapp
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        description: VisitorsApp is the Schema for the visitorsapps API
        properties:
          apiVersion:
            description: 'APIVersion defines the versioned schema of this representation
              of an object. Servers should convert recognized schemas to the latest
              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
            type: string
          kind:
            description: 'Kind is a string value representing the REST resource this
              object represents. Servers may infer this from the endpoint the client
              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
            type: string
          metadata:
            type: object
          spec:
            description: VisitorsAppSpec defines the desired state of VisitorsApp
            properties:
              size:
                format: int32
                type: integer
              title:
                type: string
            required:
            - size
            - title
            type: object
          status:
            description: VisitorsAppStatus defines the observed state of VisitorsApp
            properties:
              backendImage:
                type: string
              frontendImage:
                type: string
            required:
            - backendImage
            - frontendImage
            type: object
        type: object
    served: true
    storage: true
    subresources:
      status: {}
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

Controllers

Controllers are core components in Kubernetes and is where your operator logic takes place.

The reconcile function is responsible for enforcing the desired CR state on the actual state of the system. It runs each time an event occurs on a watched CR or resource, and will return some value depending on whether those states match or not.

In this way, every Controller has a Reconciler object with a Reconcile() method that implements the reconcile loop.

The Reconcile Loop
Figure 1. Hausenblas, Schimanski. Programming Kubernetes. O’Reilly, 2019.

Here’s our implementation:

//+kubebuilder:rbac:groups=app.redhat.com,resources=visitorsapps,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=app.redhat.com,resources=visitorsapps/status,verbs=get;update;patch
//+kubebuilder:rbac:groups=app.redhat.com,resources=visitorsapps/finalizers,verbs=update
func (r *VisitorsAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
        log := ctrllog.FromContext(ctx)

        log.Info("Reconciling VisitorsApp", "Request.Namespace", req.Namespace, "Request.Name", req.Name)

        // Fetch the VisitorsApp instance
        v := &appv1.VisitorsApp{}
        err := r.Client.Get(context.TODO(), req.NamespacedName, v)
        if err != nil {
                if errors.IsNotFound(err) {
                        // Request object not found, could have been deleted after ctrl req.
                        // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
                        // Return and don't requeue
                        return ctrl.Result{}, nil
                }
                // Error reading the object - requeue the req.
                return ctrl.Result{}, err
        }

        var result *ctrl.Result

        // == MySQL ==========
        result, err = r.ensureSecret(req, v, r.mysqlAuthSecret(v))
        if result != nil {
                return *result, err
        }

        result, err = r.ensureDeployment(req, v, r.mysqlDeployment(v))
        if result != nil {
                return *result, err
        }

        result, err = r.ensureService(req, v, r.mysqlService(v))
        if result != nil {
                return *result, err
        }

        mysqlRunning := r.isMysqlUp(v)

        if !mysqlRunning {
                // If MySQL isn't running yet, requeue the ctrl
                // to run again after a delay
                delay := time.Second * time.Duration(5)

                log.Info(fmt.Sprintf("MySQL isn't running, waiting for %s", delay))
                return ctrl.Result{RequeueAfter: delay}, nil
        }

        // == Visitors Backend  ==========
        result, err = r.ensureDeployment(req, v, r.backendDeployment(v))
        if result != nil {
                return *result, err
        }

        result, err = r.ensureService(req, v, r.backendService(v))
        if result != nil {
                return *result, err
        }

        err = r.updateBackendStatus(v)
        if err != nil {
                // Requeue the req if the status could not be updated
                return ctrl.Result{}, err
        }

        result, err = r.handleBackendChanges(v)
        if result != nil {
                return *result, err
        }

        // == Visitors Frontend ==========
        result, err = r.ensureDeployment(req, v, r.frontendDeployment(v))
        if result != nil {
                return *result, err
        }

        result, err = r.ensureService(req, v, r.frontendService(v))
        if result != nil {
                return *result, err
        }

        err = r.updateFrontendStatus(v)
        if err != nil {
                // Requeue the req
                return ctrl.Result{}, err
        }

        result, err = r.handleFrontendChanges(v)
        if result != nil {
                return *result, err
        }

        // == Finish ==========
        // Everything went fine, don't requeue

        return ctrl.Result{}, nil
}

// SetupWithManager sets up the controller with the Manager.
func (r *VisitorsAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
        return ctrl.NewControllerManagedBy(mgr).
                For(&appv1.VisitorsApp{}).
                Complete(r)
}

Copy the controllers logic and all dependencies such as common.go, backend.go, frontend.go and mysql.go into your operator controllers dir:

cp $TUTORIAL_HOME/apps/go/controllers/* $HOME/visitors-operator/controllers/

Download dependencies:

cd $HOME/visitors-operator
go get k8s.io/api/apps/v1@v0.21.2

Run your operator locally

Be sure to be connected to a Kubernetes cluster and then run

make install run
/home/bluesman/visitors-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go: creating new go.mod: module tmp
Downloading sigs.k8s.io/kustomize/kustomize/v3@v3.8.7
go get: added sigs.k8s.io/kustomize/kustomize/v3 v3.8.7
/home/bluesman/visitors-operator/bin/kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/visitorsapps.app.redhat.com created
/home/bluesman/visitors-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
api/v1/visitorsapp_types.go
go vet ./...
go run ./main.go
2021-10-28T00:20:13.875+0200    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2021-10-28T00:20:13.876+0200    INFO    setup   starting manager
2021-10-28T00:20:13.876+0200    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2021-10-28T00:20:13.876+0200    INFO    controller-runtime.manager.controller.visitorsapp       Starting EventSource    {"reconciler group": "app.redhat.com", "reconciler kind": "VisitorsApp", "source": "kind source: /, Kind="}
2021-10-28T00:20:13.877+0200    INFO    controller-runtime.manager.controller.visitorsapp       Starting Controller     {"reconciler group": "app.redhat.com", "reconciler kind": "VisitorsApp"}
2021-10-28T00:20:14.178+0200    INFO    controller-runtime.manager.controller.visitorsapp       Starting workers        {"reconciler group": "app.redhat.com", "reconciler kind": "VisitorsApp", "worker count": 1}

Apply a Custom Resource

You can now apply a custom resource

apiVersion: app.redhat.com/v1
kind: VisitorsApp
metadata:
  name: visitorsapp-sample
spec:
  size: 1
  title: "My First Operator in Go!"
kubectl apply -f $TUTORIAL_HOME/apps/cr/visitorsapp-go.yaml

Check the logs from the operator running locally:

2021-10-28T00:24:31.945+0200    INFO    controller-runtime.manager.controller.visitorsapp       Reconciling VisitorsApp {"reconciler group": "app.redhat.com", "reconciler kind": "VisitorsApp", "name": "visitorsapp-sample", "namespace": "demo", "Request.Namespace": "demo", "Request.Name": "visitorsapp-sample"}
2021-10-28T00:24:40.847+0200    INFO    controller_visitorsapp  Creating a new secret   {"Secret.Namespace": "rbc-demo", "Secret.Name": "mysql-auth"}
2021-10-28T00:24:41.337+0200    INFO    controller_visitorsapp  Creating a new Deployment       {"Deployment.Namespace": "demo", "Deployment.Name": "mysql"}
2021-10-28T00:24:41.873+0200    INFO    controller_visitorsapp  Creating a new Service  {"Service.Namespace": "rbc-demo", "Service.Name": "mysql-service"}
2021-10-28T00:24:42.061+0200    INFO    controller-runtime.manager.controller.visitorsapp       MySQL isn't running, waiting for 5s     {"reconciler group": "app.redhat.com", "reconciler kind": "VisitorsApp", "name": "visitorsapp-sample", "namespace": "demo"}
...

Check the pods getting created :

kubectl get pods

NAME                                 READY   STATUS    RESTARTS   AGE

mysql-86c559bb7f-kjjvt               1/1     Running   0          28h

visitors-backend-7489bb97dd-wggkt    1/1     Running   0          28h

visitors-frontend-86df47fffc-d2bgl   1/1     Running   0          28h

Check your newly create CR:

kubectl get visitorsapp
NAME                 AGE
visitorsapp-sample   1m

Get your VisitorApp status:

kubectl describe visitorsapp visitorsapp-sample
Name:         visitorsapp-sample
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  app.redhat.com/v1
Kind:         VisitorsApp
Metadata:
  Creation Timestamp:  2021-10-28T07:39:34Z
Spec:
  Size:   1
  Title:  My First Operator in Go!
Status:
  Backend Image:   jdob/visitors-service:1.0.0
  Frontend Image:  jdob/visitors-webui:1.0.0
Events:            <none>

Access the VisitorsApp! A kubernetes service for the frontend has been created (visitorsapp-sample-frontend-service) and it is exposed as a NodePort on port 30686.

On Minikube, get Minikube IP and access the app:

IP=$(minikube ip -p operators)
PORT=$(kubectl get service/visitorsapp-sample-frontend-service -o jsonpath="{.spec.ports[*].nodePort}")
curl $IP:$PORT

Or open it in the browser:

Visitors App

Build and Push the Operator

Your Makefile composes image tags either from values written at project initialization or from the CLI. In particular, IMAGE_TAG_BASE lets you define a common image registry, namespace, and partial name for all your image tags. Update this to another registry and/or namespace if the current value is incorrect. Afterwards you can update the IMG variable definition like so:

IMAGE_TAG_BASE ?= quay.io/redhat-scholars/visitors-operator
IMG ?= $(IMAGE_TAG_BASE):$(VERSION)

Be sure to be logged to your registry, then build and push your operator:

make docker-build docker-push
/home/bluesman/visitors-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
/home/bluesman/visitors-operator/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
KUBEBUILDER_ASSETS="/home/bluesman/.local/share/kubebuilder-envtest/k8s/1.21.4-linux-amd64" go test ./... -coverprofile cover.out
?       github.com/redhat-scholars/visitors-operator    [no test files]
?       github.com/redhat-scholars/visitors-operator/api/v1     [no test files]
ok      github.com/redhat-scholars/visitors-operator/controllers        7.019s  coverage: 0.0% of statements
docker build -t quay.io/redhat-scholars/visitors-operator:0.0.1 .
STEP 1: FROM golang:1.16 AS builder
STEP 2: WORKDIR /workspace
--> Using cache 4e85f8aa0ff71ec355dfc67058c21de8796486c2efa213525759565a4f872365
--> 4e85f8aa0ff
STEP 3: COPY go.mod go.mod
--> Using cache 339ec79ec7d2c2477ed2c6f5e6b45c931006d6678da422a6d60579d3fb2efac0
--> 339ec79ec7d
STEP 4: COPY go.sum go.sum
--> Using cache 34ac94423726a6554ccc30cf7ddcc276505fc07a2ae09f605acd515facbc2300
--> 34ac9442372
STEP 5: RUN go mod download
--> Using cache 852bba1954f63dca4fdf98a64032bbbcf98fe46e8ca98bf44a3edc1b1298f4e4
--> 852bba1954f
STEP 6: COPY main.go main.go
--> f82e4001343
STEP 7: COPY api/ api/
--> f606a197b22
STEP 8: COPY controllers/ controllers/
--> 163d54e246d
STEP 9: RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o manager main.go
--> f0e9262884f
STEP 10: FROM gcr.io/distroless/static:nonroot
STEP 11: WORKDIR /
--> Using cache db7823b88929278c7c33c8490f3b2b90332ceec8d5e80c676feb3cd73681c71a
--> db7823b8892
STEP 12: COPY --from=builder /workspace/manager .
--> 89c8123fcb3
STEP 13: USER 65532:65532
--> 7f2206d5f4f
STEP 14: ENTRYPOINT ["/manager"]
STEP 15: COMMIT quay.io/redhat-scholars/visitors-operator:0.0.1
--> 87823cdb316
87823cdb316360f9c1a0c9da11ebe414c1bfb513b03343fe579cf3dfae3a5ad4
docker push quay.io/redhat-scholars/visitors-operator:0.0.1
Getting image source signatures
Copying blob 8dd1e0231136 done
Copying blob c0d270ab7e0d skipped: already exists
Copying config 87823cdb31 done
Writing manifest to image destination
Copying config 87823cdb31 [--------------------------------------] 0.0b / 1.2KiB
Writing manifest to image destination
Writing manifest to image destination
Storing signatures

The version is incremental, first one is 0.0.1 so that your container image will look like similar to this:

quay.io/redhat-scholars/visitors-operator:0.0.1

Deploy to Kubernetes

make deploy
/home/bluesman/visitors-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /home/bluesman/visitors-operator/bin/kustomize edit set image controller=quay.io/redhat-scholars/visitors-operator:0.0.1
/home/bluesman/visitors-operator/bin/kustomize build config/default | kubectl apply -f -
I1028 00:36:45.121764   21692 request.go:645]
namespace/visitors-operator-system created
customresourcedefinition.apiextensions.k8s.io/visitorsapps.app.redhat.com configured
serviceaccount/visitors-operator-controller-manager created
role.rbac.authorization.k8s.io/visitors-operator-leader-election-role created
clusterrole.rbac.authorization.k8s.io/visitors-operator-manager-role created
clusterrole.rbac.authorization.k8s.io/visitors-operator-metrics-reader created
clusterrole.rbac.authorization.k8s.io/visitors-operator-proxy-role created
rolebinding.rbac.authorization.k8s.io/visitors-operator-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/visitors-operator-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/visitors-operator-proxy-rolebinding created
configmap/visitors-operator-manager-config created
service/visitors-operator-controller-manager-metrics-service created
deployment.apps/visitors-operator-controller-manager created

Check if operator is present:

kubectl get pods -n visitors-operator-system
NAME                                                    READY   STATUS    RESTARTS      AGE
visitors-operator-controller-manager-7f4b4cc4c7-z2xqm   2/2     Running   1 (29s ago)   2m43s

Deploy to Kubernetes with OLM

The Operator Lifecycle Manager (OLM) is a set of cluster resources that manage the lifecycle of an Operator. The Operator SDK supports both creating manifests for OLM deployment, and testing your Operator on an OLM-enabled Kubernetes cluster.

Install OLM with the Operator SDK CLI:

operator-sdk olm install

Bundle your operator, then build and push the bundle image. The bundle target generates a bundle in the bundle directory containing manifests and metadata defining your operator. bundle-build and bundle-push build and push a bundle image defined by bundle.Dockerfile.

This will create a new container image and will push it to your previously configured registry. e.g. quay.io/redhat-scholars/visitors-operator-bundle:0.0.1. The command below will prompt to fill info for your OLM managed operator:

make bundle bundle-build bundle-push
home/bluesman/visitors-operator/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
operator-sdk generate kustomize manifests -q

Display name for the operator (required):
> visitors-operator

Description for the operator (required):
> 3-tier app operator backed example

Provider's name for the operator (required):
> Red Hat

Any relevant URL for the provider name (optional):
> https://github.com/redhat-scholars/operators-sdk-tutorial

Comma-separated list of keywords for your operator (required):
> operatorsdk,devnation,redhat

Comma-separated list of maintainers and their emails (e.g. 'name1:email1, name2:email2') (required):
> Natale Vinto:nvinto@redhat.com
cd config/manager && /home/bluesman/visitors-operator/bin/kustomize edit set image controller=quay.io/redhat-scholars/visitors-operator:0.0.1
/home/bluesman/visitors-operator/bin/kustomize build config/manifests | operator-sdk generate bundle -q --overwrite --version 0.0.1
INFO[0000] Creating bundle.Dockerfile
INFO[0000] Creating bundle/metadata/annotations.yaml
INFO[0000] Bundle metadata generated suceessfully
operator-sdk bundle validate ./bundle
INFO[0000] All validation tests have completed successfully
docker build -f bundle.Dockerfile -t quay.io/redhat-scholars/visitors-operator-bundle:v0.0.1 .
...

Finally run the bundle:

operator-sdk run bundle  quay.io/redhat-scholars/visitors-operator-bundle:v0.0.1
INFO[0009] Successfully created registry pod: quay-io-redhat-scholars-visitors-operator-bundle-v0-0-1
INFO[0009] Created CatalogSource: visitors-operator-catalog
INFO[0009] OperatorGroup "operator-sdk-og" created
INFO[0009] Created Subscription: visitors-operator-v0-0-1-sub
INFO[0013] Approved InstallPlan install-9sdhh for the Subscription: visitors-operator-v0-0-1-sub
INFO[0013] Waiting for ClusterServiceVersion "default/visitors-operator.v0.0.1" to reach 'Succeeded' phase
INFO[0013]   Waiting for ClusterServiceVersion "default/visitors-operator.v0.0.1" to appear
INFO[0026]   Found ClusterServiceVersion "default/visitors-operator.v0.0.1" phase: Pending
INFO[0028]   Found ClusterServiceVersion "default/visitors-operator.v0.0.1" phase: Installing
INFO[0038]   Found ClusterServiceVersion "default/visitors-operator.v0.0.1" phase: Succeeded
INFO[0038] OLM has successfully installed "visitors-operator.v0.0.1"

Verify that a Subscription to your operator has been created:

kubectl get subscriptions
NAME                           PACKAGE             SOURCE                      CHANNEL
visitors-operator-v0-0-1-sub   visitors-operator   visitors-operator-catalog   alpha
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  labels:
    name: visitors-operator-v0-0-1-sub
spec:
  channel: alpha
  installPlanApproval: Manual
  name: visitors-operator
  source: visitors-operator-catalog
  sourceNamespace: default
  startingCSV: visitors-operator.v0.0.1

Verify that the operator installation has been successful:

kubectl get csv
NAME                       DISPLAY             VERSION   REPLACES   PHASE
visitors-operator.v0.0.1   visitors-operator   0.0.1                Succeeded

Get this complete example from this repo: Visitors Operator Example