I will be using one of my previous blogs as a background for this one. Although the concept remains the same, the execution is different. We will leverage Oracle Event Hub cloud service as the managed Kafka broker. Our producer & consumer microservices run will run on Oracle Application Container Cloud and leverage its Service Binding feature for Oracle Event Hub cloud. CI/CD for both these applications will be handled by Oracle Developer Cloud service. Specific focus areas include

 

  • Overview of how to get started with Oracle Event Hub cloud including bootstrapping a cluster and topic
  • How to use the Oracle Event Hub Cloud service binding available in Application Container Cloud
  • Configuring Oracle Developer Cloud Service to achieve CI/CD to Application Container Cloud

 

 

Overview

Let's briefly look at what role do the individual cloud services play

 

Oracle Event Hub cloud

This is a fully managed Platform-as-a-Service which makes it dead simple to setup and work with an Apache Kafka cluster

  • You can easily setup clusters and scale them elastically
  • Quickly create topics and add/remove partitions
  • It also provides REST interface as well command line clients to work with your Kafka cluster & topics

 

Oracle Application Container cloud

We continue to use it as the platform to run our producer and consumer microservices. The good thing is that services running on Application Container cloud can easily connect with Oracle Event Hub cloud service using the Service Binding feature. We will this in action

 

Oracle Developer Cloud

This serves as a central hub for source code repository and DevOps pipeline. All we need to do is configure the build and deployment once and it will take care of seamless CI/CD to Application Container Cloud. Although Developer Cloud is capable of a lot more, this blog will focus on these features. Please refer to the documentation for more details

 

Code

The logic for our consumer and producer services is very much the same and the details available here. Let's focus on how to use the Oracle Event Hub service binding

 

Leveraging the Event Hub Service Binding in Application Container Cloud

The service bindings are utilized in the same way in both our services i.e. consumer and producer. The logic uses the Application Container Cloud environment variables (created as a result of the Service Binding) to fetch the location of our Event Hub Kafka cluster as well the topic we want to work with (in this case it’s just a single topic). You do not need to expose ports on the Kafka node(s) for the services on Application Container Cloud to access them. It's all taken care of by the Service Binding internally !

 

Here is a preview

 

Note the usage of OEHCS_TOPIC and OEHCS_EXTERNAL_CONNECT_STRING

public class Consumer implements Runnable {

    private static final Logger LOGGER = Logger.getLogger(Consumer.class.getName());
    private static final String CONSUMER_GROUP = "cpu-metrics-group";
    private final AtomicBoolean CONSUMER_STOPPED = new AtomicBoolean(false);
    private KafkaConsumer<String, String> consumer = null;
    private final String topicName;
    public Consumer() {
        Properties kafkaProps = new Properties();
        LOGGER.log(Level.INFO, "Kafka Consumer running in thread {0}", Thread.currentThread().getName());

        this.topicName = System.getenv().get("OEHCS_TOPIC");
        LOGGER.log(Level.INFO, "Kafka topic {0}", topicName);

        String kafkaCluster = System.getenv().get("OEHCS_EXTERNAL_CONNECT_STRING");
        LOGGER.log(Level.INFO, "Kafka cluster {0}", kafkaCluster);

        kafkaProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaCluster);
        kafkaProps.put(ConsumerConfig.GROUP_ID_CONFIG, CONSUMER_GROUP);
        kafkaProps.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
        kafkaProps.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");

        this.consumer = new KafkaConsumer<>(kafkaProps);
    }

 

 

  • The variable OEHCS_EXTERNAL_CONNECT_STRING allows us get the co-ordinates for the Kafka cluster. This is used in the Kafka configuration represented by a java.util.Properties object
  • OEHCS_TOPIC gives us the name of the topic which is then passed on to the subscribe method of the KafkaConsumer

 

 

 

Setting up Oracle Event Hub cloud service

Let’s see how to setup a managed Kafka cluster in Oracle Cloud

 

Bootstrap the cluster

We first create the Kakfa cluster itself. Use the wizard in the Oracle Event Hub Cloud Service – Platform section to get started

 

 

Enter the required details

 

 

In this case, we are choosing the following configuration

  • Basic deployment where the Kafka cluster and Zookeeper are co-located on the same node
  • A single Kafka node

 

 

 

You can choose from different options such as

  • changing the deployment type to Recommended
  • opting for 3 nodes with the Basic mode
  • deploying a REST proxy alongside your cluster etc.

 

Refer to the official product documentation for more details

 

Click Create to start the provisioning process for your Kafka cluster

 

 

Wait for the process to complete

 

 

Setup the Kafka topic

Once the Kafka cluster is created, you can now create individual topics. To do so, choose Oracle Event Hub Cloud Service from the Platform Services menu

 

 

Click on Create Service and fill in the required details in the subsequent page

 

 

Here we create a topic name cpu-metrics in the cluster kafka-cluster (we just created). The number of partitions is 10 and the retention period is one week (168 hours)

 

 

 

Click Create to conclude the process

 

 

 

Within a few seconds, you should see your newly created topic

 

 

 

Configuring Developer Cloud Service

 

Project & code repository creation

Please refer to the Project & code repository creation section in the Tracking JUnit test results in Developer Cloud service blog or check the product documentation for more details

 

Configure source code in Git repository

Push the project from your local system to your Developer Cloud Git repo you just created. We will do this via command line and all you need is Git client installed on your local machine. You can use Git or any other tool of your choice

 

Repeat this process for both your application (producer and consumer)

 

cd <project_folder> //where you unzipped the source code  
git init  
git remote add origin <developer_cloud_git_repo>  
//e.g. https://john.doe@developer.us.oraclecloud.com/developer007-foodomain/s/developer007-foodomain-project_2009/scm/sample.git//john.doe@developer.us.oraclecloud.com/developer007-foodomain/s/developer007-foodomain-project_2009/scm/sample.git   
git add .  
git commit -m "first commit"  
git push -u origin master  //Please enter the password for your Oracle Developer Cloud account when prompted

 

 

Configure build job

Repeat this process for both your application (producer and consumer)

 

Create a New Job

 

 

Basic Configuration

 

Select JDK

 

 

Source Control

 

Choose Git repository

 

 

Build Trigger (Continuous Integration)

 

Set build trigger - this build job will be triggered in response to updated within the Git repository (e.g. via git push)

 

 

Build steps

 

A Maven Build step – to produce the ZIP file to be deployed to Application Container Cloud

 

 

Post-Build actions

 

Activate a post build action to archive the zip file

 

 

   

 

Execute Build

 

Before configuring deployment, we need to trigger the build in order to produce the artifacts which can be referenced by the deployment configuration

 

 

 

After the build is complete, you can check the archived artifacts

 

 

 

Continuous Deployment (CD) to Application Container Cloud

 

Repeat this process for both your application (producer and consumer)

 

Create a New Configuration for deployment

 

 

 

 

  • Enter the required details and configure the Deployment Target
  • Configure the Application Container Cloud instance
  • Configure Automatic deployment option on the final confirmation page
  • Provide content for manifest.json and deployment.json

 

You’ll end up with the below configuration (the view has been split into two parts)

 

 

 

Application Container Cloud defines two primary configuration descriptors – manifest.json and deployment.json, and each of them fulfill a specific purpose (more details here)

 

 

 

Click Save, initiate your deployment and wait for it to finish

 

Confirmation screen

 

 

 

Check your application(s) in Application Container Cloud

 

 

 

In the Deployment sub-section of the application details screen, notice that the required Service Bindings have been automatically wired and the environment variables have been populated as well (only a couple of variables have been highlighted below)

 

 

 

Test the application

 

The details to test the application are the same as described in this section of the previous blog. It’s really simple and here are the high level steps

  • Start your producer application using its REST URL, and
  • Access your consumer application

 

You should see the real time metrics being sent by the producer component to the Event Hub cloud service instance and consumed by the Server-Sent event (SSE) client via the consumer microservice

 

Test the CI/CD flow

Make some code changes and push them to the Developer Cloud service Git repo. This should

  • Automatically trigger the build, which once successful will
  • Automatically trigger the deployment process, and
  • Redeploy the new application version to Application Container Cloud

 

 

**The views expressed in this post are my own and do not necessarily reflect the views of Oracle.