EC2 INSTANCE SETUP FOR KAFKA

This document guides you through the setup of an EC2 instance which has been set up
exclusively for Kafka. The following services have already been installed in the EC2 instance:

1. Zookeeper
2. Kafka
3. Anaconda (Jupyter Notebook)
4. Java 1.8

You don’t need to install all the above services again. You just have to install OS image into
your EC2 instance that’s been published as a public AMI.

The steps to set up the EC2 instance are as follows:

● Go to the AWS Management Console and click on “All Services” and then click on “EC2”
or search for the service in the search bar.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Once you click on that, you will be redirected to a new page, shown below. Click on the
“Launch Instance” button.

● You’ll be redirected the EC2 instance launch page.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● In the Application and OS Images section, you will need to select the Image to be used
for the EC2 instance.
Click on “Browse more AMIs”.

● In the search box that appears at the top, copy and paste the following AMI id and press
enter: ami-06c41d8b5a6ddd3c2
In the “Community AMIs” tab, you will find the AMI with the following AMI Name:
Kafka_Anaconda-New-2022
Click on the “Select” button to choose the image.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Once you hit “Select” you should get the screen shown below:

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Select the General Purpose t2.medium type EC2 instance, as shown in the image
below. Using any other instance type, such as t2.micro which has 1 vCPU and 1 GB
memory, may lead to memory errors while working with Kafka.
You also need to select the Key Pair to login to the instance via SSH.

● In the “Network Settings” section, go with the default security group. Make sure that the
option “Allow SSH traffic from Anywhere” is ticked. This will ensure that you’re able to
SSH into the instance from your SSH client. The settings for this will be modified in the
later steps.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● In the “Configure storage” settings, you need to enter the volume size as 30 GiB and
volume type as standard (magnetic), as shown in the image below. Once the settings
have been updated, click on the “Launch Instance” button to create the instance.

● You’ll need to wait for a while until the instance is created. Once finished, click on the
“View all Instances” button.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● You’ll be redirected to a new page where you can view all running EC2 instances. Click
on the security tab and the security group ID.

● As you can see, there’s only one inbound rule in the security group. Click on “Edit
Inbound Rules”

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 ● Click on Add Rule button and configure the security group as shown below. Enter the
following values as shown in the image below
Type: Custom TCP
Port Range: 8888
Source: Anywhere-IPv4

Similarly, you need to add the following port numbers:

2181, 9092, 9000, 8080
Make sure that it looks like this:

Note: This is a very important step. Double-check that all the port range and source are
identical as shown in the image.

Next click on the Save rules.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 ELASTIC IP CREATION INSTRUCTIONS
Whenever you start an EC2 instance, it generates a new IPv4 public IP. To keep the public IP
constant, an elastic IP is required. A constant public IP is crucial in Kafka as you are required to
create Kafka producers and consumers by writing code. You will understand the significance of
having a constant IP when you will actually write the codes.

To create an elastic IP, do the following:

● Go to your EC2 instance page and on the left-hand side scroll till you find the Network
and Security tab, on the left-hand side, under which you will see the Elastic IPs link, as
shown in the image below.

● Click on the Elastic IPs link. You will be redirected to a new page. Click on the Allocate
Elastic IP address button

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

Once you do so, you will be redirected to a new page as shown below.
Choose the settings and click on the “Allocate” button.

● Click on Allocate. You will be redirected back to the elastic IP screen. Please note that
only one Elastic IP address can’t be allotted to multiple EC2 instances.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Click on the checkbox against your newly created elastic IP. Then, click on the Actions
button, you will get a drop-down list. From this list, click on Associate Address.

● A new page will appear, as shown below. Click inside the Instance text box; you will get
a list of available instances, as shown below. From the list, choose the EC2 instance with
which you want to attach the elastic IP. If you have followed the steps in this document,
then its name would be kafka_instance. The Instance ID of that EC2 instance will
appear in the text box, as shown in the example below.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Next, click on the text-box next to Private IP and select the automatically generated
Private IP address and then click on Associate.

● You will be navigated to a new page where you will have confirmation that the elastic IP
has been attached to your desired EC2 instance, as shown below and click on Close.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Now, go to the Instances page using the Scroll bar on the left-hand side of the screen
and click on the tick box against the EC2 instance with which you associated the elastic
IP. You will see that the elastic IP has been attached to the EC2 instance, as shown
below.

● You can see that the elastic IP has been attached to the EC2 instance and your Private
IPv4 address has been updated as well.
Now that you have setup your EC2 instance, you will now need to SSH into the instance
using the Public IPv4 DNS of the EC2 instance.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

● Next, you need to login to the EC2 instance with your key pair. After logging in, go to the
/home/ec2-user/downloads directory where you will see the Kafka directory is already
present, as shown in the image below.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 KAFKA SERVER CONFIGURATION
The following steps list the process to configure the Kafka Server so that you can connect your
Kafka server with an IDE such as Eclipse.

To configure the Kafka server, you need to make changes to the server.properties file for
which you need to go through the following steps:

● Login to the EC2 instance and go inside the Kafka directory present there. The
command for the same is cd /home/ec2-user/downloads/kafka_2.12-2.3.0. Once you
are inside this directory, you need to go inside the config directory. The command for the
same is cd config/. Once you are inside the directory, enter the command ls. This will
list you the different files present inside the directory.

● Here you need to make changes to the server.properties file. To edit this file enter the
command vi server.properties. You would get a screen, as shown below.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 ● Here you need to make changes in the line which reads as follows:

#advertised.listeners=PLAINTEXT://your.host.name:9092

● You need to uncomment this line and in place of your.host.name you need to enter the
IPv4 Public IP of your EC2 Instance. This is the same as the Elastic IP associated with
your EC2 instance. So press i, and you will enter insert mode.

You can see the IP in my case is 52.21.15.133.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 ● Then go to the above line and uncomment it by removing the #. Next in place of
your.host.name enter the IPv4 Public IP of your EC2 instance. In this case, that line
would read as follows:

advertised.listeners=PLAINTEXT://52.21.15.133:9092

Your screen should look something as shown below:

Make sure that you enter the IP address of your EC2 instance.

Once you have done these steps, press esc and then type :wq to come out of the file.

With these steps, you have configured the Kafka server to connect it with an IDE such as
Eclipse.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 STEPS TO VERIFY THE INSTALLATION

Zookeeper
To verify the zookeeper installation, follow the steps listed below.
● You need to get inside the Kafka directory. Go to the Kafka directory using the cd
kafka_2.12-2.3.0/ command and then start the Zookeeper server using the
bin/zookeeper-server-start.sh config/zookeeper.properties command. You should
get the following output.

At the bottom of the screen, you would get something like:

INFO binding to port 0.0.0.0/0.0.0.0:2181
(org.apache.zookeeper.server.NIOServerCnxnFactory)

This means that your zookeeper server has successfully started.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

Kafka
Before going through this step, please ensure that the Zookeeper server is running.
To verify the Kafka installation, follow the steps listed below:
● Leave the previous terminal window as it is and login to your EC2 instance using another
terminal.
● Go to the Kafka directory using the cd downloads/kafka_2.12-2.3.0 command.
● Start the Kafka server using the bin/kafka-server-start.sh config/server.properties
command.
● You should get an output which displays a message something like “INFO [KafkaServer
id=0] started (kafka.server.KafkaServer)”

This means the instance creation is successful, and you can continue with the next steps.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 STARTING JUPYTER NOTEBOOKS
● Open a new Terminal window and log in to the EC2 instance.
● Make sure that the current directory is /home/ec2-user by running the pwd command.
Once you are inside the /home/ec2-user directory, follow the steps listed below.

● Enter the command cd .jupyter/ and then enter the ls command.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

 ● Enter the command vi jupyter_notebook_config.py. At the bottom you would see the
following two lines:
c.NotebookApp.allow_origin = '*'
c.NotebookApp.ip = '0.0.0.0'

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

If you see these two lines, which you most likely would, you do not need to do anything.
Just press :wq and come out the config file.

Optional: If these two lines are not present, then you need to copy the above two lines and then
write it into the jupyter_notebook_config.py file. For that press i and then enter these two lines
and then press esc and then type :wq to come save it and then come out of the file.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

Next you need to enter the cd .. to come to the /home/ec2-user directory.

Once you are in the /home/ec2-user directory, you need to enter the command nohup jupyter
notebook &.

Then you need to press Ctrl+C and then need to run the command cat nohup.out.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

Here you will get a line which says “The Jupyter Notebook is running at: http://0.0.0.0:8888/”.

Copy this url and replace 0.0.0.0 with your elastic ip associated with this EC2 instance. You also
need to make sure that the port number is always 8888 and not any other port number. Once
these are done, open the URL in your browser. So the url should be something like this:

52.21.15.133:8888 and the screen which you will get will look something as shown below.

This means that the Jupyter Notebook has started successfully.

Note: Always make sure that you always open the port number is 8888. Sometimes you might
need to run the nohup jupyter notebook & command and the cat nohup.out command again.
This might happen sometimes.

You can upload the Jupyter notebooks using the Upload button from your local machine to EC2
instance. Once you select the notebook, you need to upload you get a screen as shown below.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved

Click on the Upload button next to it and the notebook will get uploaded.

If you login to EC2 instance and enter the ls command you will see the notebook is present in
the /home/ec2-user directory.

© Copyright. upGrad Education Pvt. Ltd. All rights reserved