Observability - A complete

Guide: Part 2

Overview:
What is Metrics?

What is the difference between Metrics and Monitoring ?

Examples of Metrics

What is Prometheus ? Components

Install Prometheus on EKS

Login to Grafana

How metrics are scraped ? What are Exporter? Two commonly used Exporters

What is Metrics ?

Observability - A complete Guide: Part 2 1

 Metrics are your systems vital signs which provides a continuous pulse of its
health and performance. They are essentially values, that are aggregated over
time. Metrics can tell us the availability of a service, free CPU % of virtual
machines or nodes, pod status, deployment status. Metrics is a sub part of
monitoring.

What is the difference between Metrics and Monitoring ?

Metrics as discussed above are the values or data which is aggregated over time
to monitor the performance of the system. While monitoring is process of scraping
(pulling) the information from the metrics and representing these metrics in a well
readable format using the dashboards. Also it is capable of firing the alerts in case
of any issue with the system.

Examples of Metrics
Below are some of the most important metrics:
Infrastructure metrics: These metrics comes under the infrastructure level and
they are CPU usage, memory consumption, network bandwidth.
Kubernetes metrics: These are the metrics which is related to the kubernetes
cluster such as Pod status, deployment replicas.
Application metrics: These metrics are related to the application which are
http_request, error rate, user signups which will be a more application specific.

Once these metrics are collected, either they are feed to the monitoring system, or
scraped / pulled from the monitoring system.

What is Prometheus ?
Prometheus is one of the top open source time series database which is designed
for real time monitoring and also used for alerting.

Where time series means along with the time, we are storing a key value pair. Now
if you want to get these information from Prometheus, you can query http server
of Prometheus using PROMQL which is Prometheus query language. Then
Prometheus will give back these metrics which you requested for. You can also
get custom metrics for a kubernetes cluster using a service discovery mechanism.

Key Components of Prometheus

Observability - A complete Guide: Part 2 2

 Prometheus Server: This is the core component which scrapes and stores the
data.

Alertmanager: Sending alerts on certain thresholds when they are crossed.

Exporters: Services which exposes the metrics.

Pushgateway: Push metrics to Prometheus.

Service Discovery: Automatically detects services.

Install Prometheus on EKS

Pre-requisites:

1. AWS Account

2. AWS CLI

3. eksctl

4. kubectl

5. Helm

Steps:

1. Create cluster using the below command

eksctl create cluster --name=observability \

--region=us-east-1 \
--zones=us-east-1a,us-east-1b \
--without-nodegroup

2. Create the OIDC provider

eksctl utils associate-iam-oidc-provider \

--region ap-south-1 \

Observability - A complete Guide: Part 2 3

 --cluster observability \
--approve

3. Create the Node Group for your cluster, and update the context.

eksctl create nodegroup --cluster=observability \

--region=ap-south-1 \
--name=observability-ng-private \
--node-type=t3.medium \
--nodes-min=2 \
--nodes-max=3 \
--node-volume-size=20 \
--managed \
--asg-access \
--external-dns-access \
--full-ecr-access \
--appmesh-access \
--alb-ingress-access \
--node-private-networking

# Update ./kube/config file

aws eks update-kubeconfig --name observability

4. Install Kube prometheus stack.

Observability - A complete Guide: Part 2 4

 helm repo add prometheus-community https://prometheus-community
helm repo update

5. Create a new namespace.

kubectl create ns monitoring

6. Install the chart into the monitoring namespace.

cd day-2

helm install monitoring prometheus-community/kube-prometheus-sta

-n monitoring \
-f ./custom_kube_prometheus_stack.yml

7. Check for all the services if running in the monitoring namespace.

kubectl get all -n monitoring

Observability - A complete Guide: Part 2 5

 8. Forward the port for Prometheus, Grafana and Alertmanager

kubectl port-forward service/prometheus-operated -n monitoring 9

We have set up Prometheus successfully. In the same way we will set up Grafana
and Alertmanager.

kubectl port-forward service/monitoring-grafana -n monitoring 80

Login to Grafana: The username for the Grafana server is admin and password is
prom-operator .

Observability - A complete Guide: Part 2 6

 Alertmanager UI: Using the below command

kubectl port-forward service/alertmanager-operated -n monitoring

We have successfully installed all the 3 services in our kubernetes cluster. The
only difference in production server will be “ingress controller and ingress
resource”, instead of using the port-forward command.

Once done with hands-on demo, clean up the complete infrastructure.

9. Uninstall the Helm chart.

helm uninstall monitoring --namespace monitoring

10. Delete the namespace.

kubectl delete ns monitoring

11. Finally delete the cluster.

eksctl delete cluster --name observability

Observability - A complete Guide: Part 2 7

 How does Prometheus scrape metrics ?

How metrics are scraped ?

The first thing Prometheus needs is a target, target is the endpoint that supply the
metrics that Prometheus stores. Once Prometheus has the endpoints, it can begin
to retrieve the metrics from them. To scrape the metrics, Prometheus typically
uses the simple HTTP request.

What are Exporters ?

Exporters are small, built-in programs or plugins which stands between
prometheus and anything which you want to monitor that natively doesn’t support
prometheus. It acts as a translator between Prometheus and endpoint you want to
monitor.

Two most commonly used Exporters

Node Exporters: Node Exporters provide hardware and OS level system
metrics exposed by UNIX Kernels. Some of the metrics are CPU (CPU Load),
Memory (RAM Used, RAM Free) and Disk (Disk Space)

Kube State Metrics: It is a service that generates metrics about the state of
kubernetes objects through the Kubernetes API. It Exposes some of the
important information such as the Node Status, Available Deployment
Replicas, Pod Status.

Conclusion:
Monitoring is important. It helps identify when things have gone wrong, and it can
show when things are going right. Prometheus is a powerful Open Source metrics
tool. In this article we saw how to install Prometheus, Grafana and Alertmanager.
We also saw how the metrics are scraped and information about the exporters. In
the next article, we will deep dive into the Grafana and see how we can set the
Prometheus as data source and create a dashboard out of it.

Stay Tuned!

Observability - A complete Guide: Part 2 8

 👍
If you found this post useful, give it a like
Repost ♻️
Follow @Bala Vignesh for more such posts 💯🚀

Observability - A complete Guide: Part 2 9