Kubernetes Monitoring with

Prometheus & Grafana

What is Prometheus ?
Prometheus is an open-source monitoring tool that collects metrics data,
and provide tools to visualize the collected data.

In addition, Prometheus allow you to generate alerts when metrics reach a

user specified threshold.

Prometheus collects metrics by scraping targets who expose metrics

through an HTTP endpoint.

Scraped metrics are then stored in a time series database which can be
queried using Prometheus’ build-in query language PromQL

Prometheus is designed to monitor time-series data that is numeric.

What kind of metrics can Prometheus Monitor ?

CPU/Memory Utilization

Disk Space

Service Uptime

Application Specific Data

Number of exceptions

Latency

Pending Requests

What kind of Data should Prometheus not Monitor?

Events

System Logs

Traces

Kubernetes Monitoring with Prometheus & Grafana 1

 Architecture of Prometheus

1. Prometheus Server
Main component that does most of the heavy lifting.

Collects metrics by pulling data from various sources (like applications or

infrastructure) at regular intervals.

Stores the metrics in a time-series database, which means data is stored

along with timestamps.

2. Exporters
These are small programs that collect metrics from your applications,
systems, or databases.

They convert this data into a format that Prometheus can understand.

For example, a Node Exporter collects system-level metrics (CPU, memory,

etc.), and a MySQL Exporter gathers database metrics.

3. Data Storage

Kubernetes Monitoring with Prometheus & Grafana 2

 Prometheus stores all the collected data in a local time-series database.
Each metric has a timestamp so you can track changes over time.

You can query this data using PromQL (Prometheus Query Language),
which helps in creating visualizations and setting up alerts.

4. PromQL (Prometheus Query Language)

This is the language used to query the stored metrics.

You can use PromQL to create custom graphs, set up alerts, and analyze
your system’s performance over time.

5. Alertmanager
When certain conditions are met (like high CPU usage), Prometheus can
trigger alerts.

These alerts are sent to the Alertmanager, which is responsible for

handling notifications (like sending emails or messages to Slack).

Alertmanager also manages alert routing, grouping, and silencing.

6. Pushgateway
Normally, Prometheus pulls data from applications. But sometimes, apps
might not be able to expose metrics at the right time (like short-lived batch
jobs).

The Pushgateway allows such jobs to push metrics to Prometheus instead.

7. Service Discovery
Prometheus can automatically discover targets to monitor, like containers or
microservices, without needing manual configuration.

This is done through service discovery mechanisms (like Kubernetes or

Consul).

8. Visualization
You can visualize metrics using Grafana, which integrates with Prometheus
and creates graphs and dashboards.

Grafana provides a user-friendly way to monitor the health and

performance of your system in real-time.

Kubernetes Monitoring with Prometheus & Grafana 3

 Collecting Metrics
Prometheus collects metrics by sending HTTP requests to /metrics
endpoint to each target

Prometheus can be configured to use a different path other then /metrics

Prometheus follows a pull based model

Prometheus needs to have a list of all targets it should scrape.

Other Pull based monitoring solutions include:

Zabix

nagios

Why Pull?
Some of the benefits of using a pull based system:

Easier to tell if a target is down

In a pushed based system we don’t know if its down or has been

decommissioned.

Push based systems could potentially overload metrics server if too many
incoming connections get flooded at same time

Pull based system have a definitive list of targets to monitor, creating a

central source of truth.

Practical Hand-on
Installing Prometheus
1. First we need to install Helm
choco install kubernetes-helm

Kubernetes Monitoring with Prometheus & Grafana 4

 2. Once Helm install successfully, Now we can Install Prometheus by following
few steps

Install using Helm

Add helm repo

helm repo add prometheus-community <https://prometheus-community.github.io/helm-

charts >

Update helm repo

helm repo update

Install helm
helm install prometheus prometheus-community/prometheus

Expose Prometheus Service

This is required to access prometheus-server using your browser.
kubectl expose service prometheus-server --type=NodePort --target-port=9090 --
name=prometheus-server-ext

3. After that we need to check the IP Address of minikube Cluster and check
the port of expose service Prometheus.

4. Now we can open Prometheus in browser by putting the minikube IP

Address along with PORT followed by Colon “:” symbol.

Installing Grafana
1. First we need to install Helm (Skip this if followed before)
choco install kubernetes-helm

2. Once Helm install successfully, Now we can Install Prometheus by following

few steps

Install using Helm

Add helm repo

helm repo add grafana <https://grafana.github.io/helm-charts

Update helm repo

Kubernetes Monitoring with Prometheus & Grafana 5

 helm repo update

Install helm
helm install grafana grafana/grafana

Expose Grafana Service

kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext

3. After that we need to check the IP Address of minikube Cluster and check
the port of expose service Grafana.

4. Now we can open Grafana in browser by putting the minikube IP Address

along with PORT followed by Colon “:” symbol.

5. Now we Integrate Grafana with Prometheus to see the Beautiful

Dashboards and Visualizations of data

Kubernetes Monitoring with Prometheus & Grafana 6