Continuous Integration and Pipelines

In this lab you will learn about pipelines and how to configure a pipeline in OpenShift so that it will take care of the application lifecycle.

A continuous delivery (CD) pipeline is an automated expression of your process for getting software from version control right through to your users and customers. Every change to your software (committed in source control) goes through a complex process on its way to being released. This process involves building the software in a reliable and repeatable manner, as well as progressing the built software (called a "build") through multiple stages of testing and deployment.

OpenShift Pipelines is a cloud-native, continuous integration and delivery (CI/CD) solution for building pipelines using Tekton. Tekton is a flexible, Kubernetes-native, open-source CI/CD framework that enables automating deployments across multiple platforms (Kubernetes, serverless, VMs, etc) by abstracting away the underlying details.

Install OpenShift Pipelines from the Software Catalog

Those using the Developer Sandbox will already have the Pipelines Operator installed, feel free to skip this section. Installing an Operator also requires cluster administrator privileges.

OpenShift provides a marketplace of installable software packaged as Kubernetes Operator called OperatorHub . You can add Kubernetes-native CI/CD to your cluster installing OpenShift Pipelines directly from the embedded marketplace inside OpenShift.

Starting with OpenShift 4.21 the OperatorHub page has been merged into the unified Software Catalog, reachable from Ecosystem → Software Catalog. The install flow itself is unchanged.

From the left navigation, go to Ecosystem → Software Catalog and select the Operators catalog type. In the search box, search for pipelines, then click to Red Hat OpenShift Pipelines:

From the description view, click Install to review all installation settings.

Ensure Update Channel is set to stable , and click Install to start installing the Operator.

After few seconds, the installation should be completed with success and you can verify it looking at Status column, check if the status is Succeeded.

Understanding Tekton

Tekton defines a number of Kubernetes custom resources as building blocks in order to standardize pipeline concepts and provide a terminology that is consistent across CI/CD solutions.

The custom resources needed to define a pipeline are listed below:

Task: a reusable, loosely coupled number of steps that perform a specific task (e.g. building a container image)
Pipeline: the definition of the pipeline and the Tasks that it should perform
TaskRun: the execution and result of running an instance of task
PipelineRun: the execution and result of running an instance of pipeline, which includes a number of TaskRuns

In short, in order to create a pipeline, one does the following:

Create custom or install existing reusable Tasks
Create a Pipeline and PipelineResources to define your application’s delivery pipeline
Create a PersistentVolumeClaim to provide the volume/filesystem for pipeline execution or provide a VolumeClaimTemplate which creates a PersistentVolumeClaim
Create a PipelineRun to instantiate and invoke the pipeline

For further details on pipeline concepts, refer to the Tekton documentation that provides an excellent guide for understanding various parameters and attributes available for defining pipelines.

As pipelines provide the ability to promote applications between different stages of the delivery cycle, Tekton, which is our Continuous Integration server that will execute our pipelines, will be deployed on a project with a Continuous Integration role. Pipelines executed in this project will have permissions to interact with all the projects modeling the different stages of our delivery cycle.

A Pipeline is a user-defined model of a CD pipeline. A Pipeline’s code defines your entire build process, which typically includes stages for building an application, testing it and then delivering it.

A Task contains some steps to be executed. OpenShift Pipelines ships a set of ready-to-use Tasks in the openshift-pipelines namespace, and you can of course write your own custom ones in your project.

The cluster-scoped ClusterTask resource is deprecated and no longer shipped. The tasks provided by the Operator are now regular namespaced Tasks living in the openshift-pipelines namespace, and you reference them from any project through the cluster resolver instead of taskRef.kind: ClusterTask:

taskRef:
  resolver: cluster
  params:
    - name: kind
      value: task
    - name: name
      value: git-clone
    - name: namespace
      value: openshift-pipelines

You can explore all the Tasks provided by the Operator either from the Web Console (Pipelines → Tasks) or from the CLI:

oc get tasks -n openshift-pipelines

Create Your Pipeline

Ensure you installed OpenShift Pipelines Operator before proceeding.

Let’s now create a Tekton pipeline for Nationalparks backend. From the left navigation, click Pipelines → Pipelines, then click Create → Pipeline.

Here we can view the interactive Pipeline builder. We can add tasks to the pipeline by clicking on the Add task button. We can also add parameters to the pipeline by clicking on the created tasks. To save time here, we’ll use the YAML view to create the pipeline.

Now, depending on which language you are using, you’ll need to create the appropriate pipeline:

Java
.NET
Javascript
Python

Here, we can copy this Tekton Pipeline in the YAML text area. This pipeline will clone the source code from GitHub, build and test the application, build a container image and deploy it on OpenShift.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: nationalparks-pipeline
spec:
  params:
    - default: nationalparks
      name: APP_NAME
      type: string
    - default: 'https://github.com/openshift-roadshow/nationalparks.git'
      description: The application git repository url
      name: APP_GIT_URL
      type: string
    - default: master
      description: The application git repository revision
      name: APP_GIT_REVISION
      type: string
  workspaces:
    - name: app-source
    - name: maven-settings
  tasks:
    - name: git-clone
      params:
        - name: URL
          value: $(params.APP_GIT_URL)
        - name: REVISION
          value: $(params.APP_GIT_REVISION)
        - name: SUBMODULES
          value: 'true'
        - name: DEPTH
          value: '1'
        - name: SSL_VERIFY
          value: 'true'
        - name: DELETE_EXISTING
          value: 'true'
        - name: VERBOSE
          value: 'true'
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: git-clone
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: output
          workspace: app-source
    - name: build-and-test
      params:
        - name: GOALS
          value:
            - package
        - name: JAVA_VERSION
          value: '11'
      runAfter:
        - git-clone
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: maven
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: source
          workspace: app-source
        - name: maven_settings
          workspace: maven-settings
    - name: build-image
      params:
        - name: IMAGE
          value: image-registry.openshift-image-registry.svc:5000/$(context.pipelineRun.namespace)/$(params.APP_NAME):latest
        - name: DOCKERFILE
          value: ./Dockerfile
        - name: CONTEXT
          value: .
        - name: TLS_VERIFY
          value: 'true'
        - name: FORMAT
          value: oci
      runAfter:
        - build-and-test
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: buildah
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: source
          workspace: app-source
    - name: redeploy
      params:
        - name: SCRIPT
          value: oc rollout restart deployment/$(params.APP_NAME)
      runAfter:
        - build-image
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: openshift-client
          - name: namespace
            value: openshift-pipelines

Now, let’s head back to the Pipeline builder view to see it visually.

This pipeline has 4 Tasks defined:

git clone: this uses the git-clone Task to clone our source repository for nationalparks and store it into the app-source Workspace, which is backed by the PVC we are going to create for it
build-and-test: will build and test our Java application using the maven Task
build-image: this is the buildah Task that will build an image using a binary file as input in OpenShift, in our case a JAR artifact generated in the previous task
redeploy: it will use the openshift-client Task to deploy the created image on OpenShift using the Deployment named nationalparks we created in the previous lab

The Pipeline is parametric, with default values already preconfigured.

It is using two Workspaces:

app-source: linked to a PersistentVolumeClaim app-source-pvc previously created. This will be used to store the artifact to be used in different Task
maven-settings: an EmptyDir volume for the maven cache, this can be extended also with a PVC to make subsequent Maven builds faster

Here, we can copy this Tekton Pipeline in the YAML text area. This pipeline will clone the source code from GitHub, build and test the application, build a container image and deploy it on OpenShift.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: nationalparks-pipeline
spec:
  params:
    - default: nationalparks
      name: APP_NAME
      type: string
    - default: 'https://github.com/openshift-roadshow/nationalparks-dotnet.git'
      description: The application git repository url
      name: GIT_REPO
      type: string
    - default: master
      name: GIT_REVISION
      type: string
    - default: 'image-registry.openshift-image-registry.svc:5000/$(context.pipelineRun.namespace)/$(params.APP_NAME):latest'
      name: IMAGE_NAME
      type: string
    - default: .
      name: CONTEXT
      type: string
    - default: '8.0-ubi8'
      name: VERSION
      type: string
  workspaces:
    - name: workspace
  tasks:
    - name: fetch-repository
      params:
        - name: URL
          value: $(params.GIT_REPO)
        - name: REVISION
          value: $(params.GIT_REVISION)
        - name: SUBDIRECTORY
          value: ''
        - name: DELETE_EXISTING
          value: 'true'
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: git-clone
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: output
          workspace: workspace
    - name: build
      params:
        - name: IMAGE
          value: $(params.IMAGE_NAME)
        - name: VERSION
          value: $(params.VERSION)
        - name: CONTEXT
          value: $(params.CONTEXT)
        - name: TLS_VERIFY
          value: 'true'
      runAfter:
        - fetch-repository
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: s2i-dotnet
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: source
          workspace: workspace
    - name: deploy
      params:
        - name: SCRIPT
          value: oc rollout status deploy/$(params.APP_NAME)
      runAfter:
        - build
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: openshift-client
          - name: namespace
            value: openshift-pipelines

Now, let’s head back to the Pipeline builder view to see it visually.

This pipeline has 3 Tasks defined:

fetch-repository: this uses the git-clone Task to clone our source repository for nationalparks and store it into the workspace Workspace, which is backed by the PVC we are going to create for it
build: will build and test our .NET Core C# application, generate and push a container image automatically with compiled binaries inside OpenShift Container Registry
deploy: will deploy the created image on OpenShift using the Deployment named nationalparks we created in the previous lab

The Pipeline is parametric, with default values already preconfigured.

It is using one Workspace:

workspace: this needs to be linked to a PersistentVolumeClaim since it will be used to store the code and the compiled binary to be used in different Tasks

Here, we can copy this Tekton Pipeline in the YAML text area. This pipeline will clone the source code from GitHub, build and test the application, build a container image and deploy it on OpenShift.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: nationalparks-pipeline
spec:
  params:
    - default: nationalparks
      name: APP_NAME
      type: string
    - default: 'https://github.com/openshift-roadshow/nationalparks-js.git'
      description: The application git repository url
      name: GIT_REPO
      type: string
    - default: master
      name: GIT_REVISION
      type: string
    - default: 'image-registry.openshift-image-registry.svc:5000/$(context.pipelineRun.namespace)/$(params.APP_NAME):latest'
      name: IMAGE_NAME
      type: string
    - default: .
      name: CONTEXT
      type: string
    - default: '20-ubi9'
      name: VERSION
      type: string
  workspaces:
    - name: workspace
  tasks:
    - name: fetch-repository
      params:
        - name: URL
          value: $(params.GIT_REPO)
        - name: REVISION
          value: $(params.GIT_REVISION)
        - name: SUBDIRECTORY
          value: ''
        - name: DELETE_EXISTING
          value: 'true'
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: git-clone
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: output
          workspace: workspace
    - name: build
      params:
        - name: IMAGE
          value: $(params.IMAGE_NAME)
        - name: VERSION
          value: $(params.VERSION)
        - name: CONTEXT
          value: $(params.CONTEXT)
        - name: TLS_VERIFY
          value: 'true'
      runAfter:
        - fetch-repository
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: s2i-nodejs
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: source
          workspace: workspace
    - name: deploy
      params:
        - name: SCRIPT
          value: oc rollout status deploy/$(params.APP_NAME)
      runAfter:
        - build
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: openshift-client
          - name: namespace
            value: openshift-pipelines

Now, let’s head back to the Pipeline builder view to see it visually.

This pipeline has 3 Tasks defined:

fetch-repository: this uses the git-clone Task to clone our source repository for nationalparks and store it into the workspace Workspace, which is backed by the PVC we are going to create for it
build: will build and test our NodeJS application, generate and push a container image automatically with compiled binaries inside OpenShift Container Registry
deploy: will deploy the created image on OpenShift using the Deployment named nationalparks we created in the previous lab

The Pipeline is parametric, with default values already preconfigured.

It is using one Workspace:

workspace: this needs to be linked to a PersistentVolumeClaim since it will be used to store the code and the compiled binary to be used in different Tasks

Here, we can copy this Tekton Pipeline in the YAML text area. This pipeline will clone the source code from GitHub, build and test the application, build a container image and deploy it on OpenShift.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: nationalparks-pipeline
spec:
  params:
    - default: nationalparks
      name: APP_NAME
      type: string
    - default: 'https://github.com/openshift-roadshow/nationalparks-py.git'
      description: The application git repository url
      name: GIT_REPO
      type: string
    - default: master
      name: GIT_REVISION
      type: string
    - default: 'image-registry.openshift-image-registry.svc:5000/$(context.pipelineRun.namespace)/$(params.APP_NAME):latest'
      name: IMAGE_NAME
      type: string
    - default: .
      name: CONTEXT
      type: string
    - default: '3.9-ubi9'
      name: VERSION
      type: string
  workspaces:
    - name: workspace
  tasks:
    - name: fetch-repository
      params:
        - name: URL
          value: $(params.GIT_REPO)
        - name: REVISION
          value: $(params.GIT_REVISION)
        - name: SUBDIRECTORY
          value: ''
        - name: DELETE_EXISTING
          value: 'true'
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: git-clone
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: output
          workspace: workspace
    - name: build
      params:
        - name: IMAGE
          value: $(params.IMAGE_NAME)
        - name: VERSION
          value: $(params.VERSION)
        - name: CONTEXT
          value: $(params.CONTEXT)
        - name: TLS_VERIFY
          value: 'true'
      runAfter:
        - fetch-repository
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: s2i-python
          - name: namespace
            value: openshift-pipelines
      workspaces:
        - name: source
          workspace: workspace
    - name: deploy
      params:
        - name: SCRIPT
          value: oc rollout status deploy/$(params.APP_NAME)
      runAfter:
        - build
      taskRef:
        resolver: cluster
        params:
          - name: kind
            value: task
          - name: name
            value: openshift-client
          - name: namespace
            value: openshift-pipelines

Now, let’s head back to the Pipeline builder view to see it visually.

This pipeline has 3 Tasks defined:

fetch-repository: this uses the git-clone Task to clone our source repository for nationalparks and store it into the workspace Workspace, which is backed by the PVC we are going to create for it
build: will build and test our Python application, generate and push a container image automatically with compiled binaries inside OpenShift Container Registry
deploy: it will deploy the created image on OpenShift using the Deployment named nationalparks we created in the previous lab

The Pipeline is parametric, with default values already preconfigured.

It is using one Workspace:

workspace: this needs to be linked to a PersistentVolumeClaim since it will be used to store the code and the compiled binary to be used in different Tasks

Finally, make sure to click the Create button to create the Pipeline.

Exercise: Add Storage for your Pipeline

OpenShift manages Storage with Persistent Volumes to be attached to Pods running our applications through Persistent Volume Claim requests, and it also provides the capability to manage it at ease from the Web Console. From the left navigation, go to Storage → PersistentVolumeClaims.

Go to the top-right side and click the Create PersistentVolumeClaim button.

Inside PersistentVolumeClaim name insert app-source-pvc.

In Size section, insert 1 as we are going to create 1 GiB Persistent Volume for our Pipeline.

The Access mode is already set to Single user (RWO), which is what our Pipeline needs. Depending on the StorageClass available in your cluster, this field may be preset and not editable.

Leave all other default settings, and click Create.

The Storage Class is the type of storage available in the cluster.

Run the Pipeline

We can start now the Pipeline from the Web Console. From the left navigation click on Pipelines → Pipelines, then click on nationalparks-pipeline. From top-right Actions list, click on Start.

You will be prompted with parameters to add the Pipeline, showing default ones. Your appropriate values are already preconfigured depending on which language’s pipeline you chose. However, in Workspaces → select PersistentVolumeClaim from the list, and then select app-source-pvc. This is the share volume used by Pipeline Tasks in your Pipeline containing the source code and compiled artifacts. Click on Start to run your Pipeline.

You can follow the Pipeline execution at ease from Web Console. From the left navigation, click on Pipelines → Pipelines, then click on nationalparks-pipeline. Switch to the PipelineRuns tab to watch all the steps in progress:

Then click on the PipelineRun nationalparks-pipeline-run-:

Switch to the Logs tab and select a Task from the list on the left to check its logs:

Verify PipelineRun has been completed with success: