3.2 Exploring Kafka from Jupyter Notebooks

Let’s head back to the Jupyter notebook and explore Kafka, and ensure that the Kafka producers and consumers are working as expected. Just like before let’s clone a git repo and bring in some sample code.

Clone the Git Repository

Select the root source directory to make sure you clone the git repository at the correct location. To do this click on the folder icon in the displayed filepath in the file browser.

On the left toolbar, click on the Git icon:
Then click on Clone a Repository:

Enter the Git https URL for the repository and click Clone:

https://github.com/blues-man/object-detection-kafka-consumer

Select the newly created folder, object-detection-kafka-consumer and double click to browse.

Explore the Notebooks

Now’s you can try out Kafka in an interactive environment. Two notebooks have been created for you to experiment with. Feel free to check out the Kafka documentation and the kafka-python docs.

1_kafka_consumer.ipynb - A sample consumer. Start here and leave the last cell running while you try out the next notebook.

2_kafka_producer.ipynb - A sample producer. Run this file to send messages you can consume.

Back to 1_kafka_consumer.ipynb - observe the messages consumed

Stop the kernel in both notebooks.

Exploring the Kafka Consumer App

In the next section, we’ll turn our git repo into a new application. Before we do, let’s stop to examine the application files.

Open the prediction.py file. What’s interesting here is that there is NO difference from the rest service. The data science prediction code did not change
Open the requirements.txt file. Again, the data science portion of the project as not changed. What has changed is the application dependencies. flask and gunicorn have been changed into our dependency for kafka-python.
Open the app.py file. This is the bulk of our change. Our application code changed from the rest service in the wsgi.py to a basic Kafka consumer/producer.

When you’ve seen all you want to see, head to the next section.