3.1 Creating a Kafka Instance

This section details creating a Kafka instance, Kafka topics, and connecting our application to stream data in a video form!

Due to Developer Sandbox limitations with support for Kafka, this will only work with a full dedicated OpenShift cluster.

Introduction

Now that we have the model working on pictures, let’s get this working with streaming data.

A Kafka instance in OpenShift Streams for Apache Kafka includes a Kafka cluster, bootstrap server, and the configurations needed to connect to producer and consumer services. We’ll be creating:

  • Kafka Instance - record your Bootstrap server

  • Service Account - record your Client ID and Client Secret

  • Topics - record the names

    • images

    • objects

    • notebook-test

OpenShift Kafka Operator

We’ll be creating this Kafka instance and related resources within our project, using the Kafka AMQ Streams Operator. The Kafka AMQ Streams Operator is a Kubernetes Operator that manages Kafka instances deployed to OpenShift. To install the Kafka AMQ Streams Operator, head to the Administrator, then select Operators → Operators Hub and search for Kafka to see the AMQ Streams Operator. Click Install to install the Kafka AMQ Streams Operator.

alt text

Create a Kafka Instance

  • Let’s create our a Kafka instance. In the Developer perspective, select +Add → All Services and search for Kafka to create a new Kafka instance.

alt text

  • Enter a unique Name for the Kafka instance, such as object-detection, and click Create instance.

alt text

Create Topics

After you create a Kafka instance, you can create two Kafka topics to start producing and consuming messages in your services. From the Developer perspective, select +Add → All Services and search for Kafka Topic to create a new Kafka topic. Repeat this process for these two topics.

alt text

  • Create the images topic.

    • Topic name: images

    • Partitions: 1

    • Replicas: 3 (default)

  • Create the objects topic.

    • Topic name: objects

    • Partitions: 1

    • Replicas: 3 (default)_

Great, your topics should have this same configuration (with the exception of the topic name):

alt text

When you’re done, you should have three topics listed, images, objects, and notebook-test. You can view this by selecting the Web Terminal at the top-right of the OpenShift Console, and running the command oc get kafkatopics.

alt text