The preferred monitoring solution for Kubernetes and Docker is gradually evolving towards Prometheus monitoring. This manual demonstrates how to use Prometheus for Kubernetes monitoring. You will learn how to build up Kube-state-metrics system, pull and collect metrics, deploy a Prometheus server and metrics exporters, configure alerts with Alertmanager, and create Grafana dashboards. We will go over both manual and automated deployment and installation techniques of Kubernetes Prometheus, including Prometheus operator. In the modern-day world, docker is something we hear day in and day out. Docker is used to run an application by containerizing and solving the major issue. It runs in the machine but not in prod. To learn more about docker, check out learning opportunities like Docker and Kubernetes Training.
An open-source monitoring tool called Prometheus is used to gather and aggregate metrics as time series data. Simply put, every item in a Kubernetes Prometheus store is a metric event that comes with a timestamp. The Cloud Native Computing Foundation now supports Prometheus, a community project that was first created at Soundcloud. It has gained popularity quickly over the past ten years as a result of being the best monitoring stack for contemporary applications because of its combination of querying features and cloud-native architecture.
Events are recorded in real time by Prometheus. Any event that is pertinent to your program can be included in this list, including memory usage, network activity, and specific inbound requests.
"Metrics" are the basic unit of data. Each measure has a name that can be used to refer to it as well as a number of labels. The metrics in your database can be filtered using labels, which are arbitrary key-value data pairs.
I want to highlight a few important points for your review.
The Kubernetes Prometheus object model is extremely adaptable and abstract; it consists of a collection of time series, each of which is identified by a distinct set of label-value pairs and a collection of timestamped values. The job label and the instance label are my favorites among these labels since they are more equitable than the others.
By defining these labels, we can arrange Prometheus' targets page to match the topology of our application appropriately and quickly identify any problems. In Prometheus, we decided to model Kubernetes services as Jobs and individual Pods as Instances.
You can find a variety of different items in the Kubernetes model using the Prometheus K8s service discovery module. The 'role' in the Prometheus configuration refers to the type of object. Each item of that type will be identified as a distinct scrape target, on which you can configure how to map your application into Prometheus using relabelling rules:
kubernetes_sd_configs: - api_servers: - 'https://kubernetes.default.svc.cluster.local' in_cluster: true role: endpoint
Before the emergence of DevOps, monitoring was made up of services. Since they can handle a lot of operations-debugging on their own, developers need the freedom to swiftly incorporate apps. Monitoring had to be made more accessible, democratized and expanded to include additional stack tiers.
The way we handle logging is fundamentally changing, and this includes monitoring. There are now a significant number of services, virtual network addresses, exposed metrics, and unstable software entities that can appear or go at any time.
These four characteristics helped Prometheus establish itself as the de facto benchmark for Kubernetes monitoring:
Similar to how Kubernetes labels infrastructure metadata, the model's structure is built on key-value pairs. It drives the query language of Prometheus, which makes time series data flexible and accurate.
The procedure of making the metrics of prometheus accessible is not too difficult. Metrics are published via a standard HTTP transport, are readable by humans, and have formats that are self-explanatory.
Because the Prometheus server regularly scrapes the targets, applications and services do not need to worry about creating data (metrics are pulled, not pushed). These Prometheus servers can locate scrape targets automatically in a variety of ways. Some of them may be configured to filter and match container metadata, making them perfect for ephemeral Kubernetes workloads.
Metric collecting, alerting, graphical visualization, and other functions are handled by various composable services. All of these services are intended to provide redundancy and sharding.
We will go over each of the parts that make up the K8s Prometheus architecture in this section. In Kubernetes, cAdvisor functions as a component of the Kubelet binary. In essence, cAdvisor is an open-source resource consumption and performance analysis agent library. It is mostly for containers and natively supports Docker containers. A straightforward service for listening to the Kubernetes-API is Kube-State metrics. It produces metrics describing the condition of things like deployments, nodes, and pods.
The following components make up the entire Kubernetes and Prometheus architecture:
The Kubernetes Prometheus servers want as many targets to automatically find as they can. Prometheus employs Prometheus in Kubernetes SD, Consul SD, Azure SD, or Azure VM, and GCE SD for GCE instances to do this. AWS VM and File SD for EC2 SD. The central processing unit of the system, the Prometheus servers, performs similar functions to the brain. The measurements are recorded by the server as multidimensional time series.
Additionally, Prometheus is capable of gathering data on the orchestration and service status of Kubernetes. Orchestration and cluster-level metrics are retrieved through the Kube-state-metrics. Additionally, Kubernetes is in charge of basic metrics like kubelet, etcd, DNS, scheduler, etc. Alertmanager controls inhibition, among other things. The rules can be set up in Prometheus to launch PromQl alerts. Grafana utilizes a dashboard using a UI to display the scraped metrics. If you want in-depth knowledge of Kubernetes, then I would recommend you check Courses on DevOps.
By utilizing YAML files to describe rights, configuration, and services, you can configure Prometheus' monitoring processes.
YAML files are used by Prometheus to access resources. Additionally, it aids in information retrieval while scraping components of the Kubernetes cluster. The documentation for Prometheus provides more information about YAML configuration.
You can install Prometheus as a container on a Kubernetes cluster after the configuration is finished. These Docker containers can be deployed using a variety of orchestration techniques, including StatefulSets, Kubernetes operators, and Helm charts.
You can use the following command in the Prometheus user interface to launch a Prometheus server within a container:
docker run \ -p 9090:9090 \ -v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \ prom/prometheus
Make sure you have administrator access to your Kubernetes cluster before connecting.
ACCOUNT=$(gcloud info --format='value(config.account)') kubectl create clusterrolebinding owner-cluster-admin-binding \ --clusterrole cluster-admin \ --user $ACCOUNT
On Github, you can find all the configuration files I described in this article. The command listed below can be used to clone the repository.
git clone https://github.com/techiescamp/kubernetes-prometheus
For a better understanding, as described in the stages, you can utilize the GitHub repo configuration files or make the files on the fly.
Let us begin by setting everything up.
For all of our monitoring components, we will first construct a Kubernetes namespace. All Prometheus kubernetes deployment objects are deployed on the default namespace if a dedicated namespace is not created.
To create a new namespace called monitoring, use the following command.
kubectl creates namespace monitoring
To read all the accessible metrics from Nodes, Pods, Deployments, etc., Prometheus uses the Kubernetes APIs. This calls for the creation of an RBAC policy that is bound to the monitoring namespace and has read access to the necessary API groups.
Step 1: Copy the following RBAC role into a file called clusterRole.yaml.
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus rules: - apiGroups: [""] resources: - nodes - nodes/proxy - services - endpoints - pods verbs: ["get", "list", "watch"] - apiGroups: - extensions resources: - ingresses verbs: ["get", "list", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: prometheus roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: prometheus subjects: - kind: ServiceAccount name: default namespace: monitoring
Step 2: Use the following command to create the role.
kubectl create -f clusterRole.yaml
The prometheus.yaml file contains all of Prometheus' configurations, while prometheus.rules contain all of the alert rules for Alertmanager.
prometheus.yaml: This file contains the main configuration for Prometheus and contains information on data scraping, service discovery, storage locations, data retention settings, etc.
prometheus.rules: The Prometheus alerting rules are all contained in this file.
You can avoid building the Prometheus image each time a configuration needs to be added or removed by externalizing Prometheus configs to a Kubernetes config map. To apply the updated configuration, you must update the config map and restart the Prometheus pods.
The prometheus.yaml and prometheus.rules files that include all of the Prometheus scrape configuration and alerting rules are mounted to the Prometheus container in /etc/prometheus.
kubectl create -f config-map.yaml
Inside the container, two files are created.
All of the configurations to dynamically find the running services and pods in the Kubernetes cluster are contained in the prometheus.yaml file. In our Prometheus scrape configuration, we have the following scrape jobs.
All of the alert rules for sending alerts to the Alertmanager are contained in prometheus.rules.
Step 1: Make a file called prometheus-deployment.yaml and copy the information below into it. In this configuration, as described in the previous section, the Prometheus configuration map is mounted as a file inside /etc/prometheus.
apiVersion: apps/v1
kind: Deployment
metadata:
name: prometheus-deployment
namespace: monitoring
labels:
app: prometheus-server
spec:
replicas: 1
selector:
matchLabels:
app: prometheus-server
template:
metadata:
labels:
app: prometheus-server
spec:
containers:
- name: prometheus
image: prom/prometheus
args:
- "--storage.tsdb.retention.time=12h"
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus/"
ports:
- containerPort: 9090
resources:
requests:
cpu: 500m
memory: 500M
limits:
cpu: 1
memory: 1Gi
volumeMounts:
- name: prometheus-config-volume
mountPath: /etc/prometheus/
- name: prometheus-storage-volume
mountPath: /prometheus/
volumes:
- name: prometheus-config-volume
configMap:
defaultMode: 420
name: prometheus-server-conf
- name: prometheus-storage-volume
emptyDir: {} Step 2: Utilizing the aforementioned file, create a deployment on the monitoring namespace.
kubectl create -f prometheus-deployment.yaml
Step 3: Use the following command to verify the deployment you established.
kubectl get deployments --namespace=monitoring
Method 1
You can reach a pod from your local workstation using a chosen port on your localhost by using kubectl port forwarding. This approach is mainly used for debugging.
Step 1: Get the name of the Prometheus pod.
kubectl get pods --namespace=monitoring
The result will appear as shown below.
kubectl get pods --namespace=monitoring
NAME READY STATUS RESTARTS AGE
prometheus-monitoring-3331088907-hm5n1 1/1 Running 0 5m
Step 2: Use the command below to connect to Prometheus using localhost port 8080 and your pod name.
Please substitute your pod name for prometheus-monitoring-3331088907-hm5n1.
kubectl port-forward prometheus-monitoring-3331088907-hm5n1 8080:9090 -n monitoring
Step 3: At this point, your browser will display the Prometheus home page when you reach http://localhost:8080.
Method 2:
You must expose the Prometheus dashboard as a Kubernetes service in order to access it through an IP address or DNS name.
First, make a file called prometheus-service.yaml and paste the information below into it. On port 30000, we will expose Prometheus to all IP addresses of Kubernetes nodes.
apiVersion: v1 kind: Service metadata: name: prometheus-service namespace: monitoring annotations: prometheus.io/scrape: 'true' prometheus.io/port: '9090' spec: selector: app: prometheus-server type: NodePort ports: - port: 8080 targetPort: 9090 nodePort: 30000
The aforementioned service's annotations YAML ensures that Prometheus deletes the service endpoint. Always use the target port specified in service YAML, prometheus.io/port.
Here are a few best practices that will enable you to integrate Prometheus into Kubernetes successfully.
Data is vital in general, but not all of it is required in every situation. As you create your consoles and dashboards, bear this in mind.
You should aim to display the most pertinent data rather than attempting to display all data within a single operational panel. Consider the most likely failure modes, and then portray each one in a useful visual presentation, to achieve this.
Labels can assist you in modifying and streamlining the data for your measurements. Resources such as RAM, disk space, bandwidth, and CPU are needed for each label set. Although this data is significant, producing labels on a wide scale requires a lot of resources.
By limiting the number of labels per meter to 10 or less, you can lower expenditures. Furthermore, you should only use labels for metrics that need them—not all metrics do. If you do need to label a lot of metrics, think about using specialized analysis tools to speed up the procedure.
Consider using timestamps, rather than the amount of time that has passed since an event happened, for tracking event timing. This can lessen errors and reduce the need for logic updates.
Some parts cannot be scrapped. The Prometheus for Kubernetes Pushgateway, which enables you to push time series data from transitory, service-level batch processes to intermediary jobs that may be scraped, can be used to monitor these components. You may combine this with the straightforward text-based explication format of Prometheus to make this instrumentation simple.
Pushgateway is excellent for recording the results of batch tasks at the service level. It is not intended for any further use cases. A single Pushgateway will turn into not only a single point of failure but also a potential bottleneck if you try to use it to monitor numerous instances, for example.
Limit the actions carried out in the inner loop when including metrics in code that are executed more than 100,000 times per second or is performance critical. Here are a few methods you can use to safeguard inner loops:
A contemporary approach to managing applications is Prometheus. Kubernetes is the main focus because that is the situation in which it performs best. Using Prometheus to deploy your Kubernetes cluster has a number of advantages, including a multidimensional data model and a DevOps culture. Following is a list of advantages of Kubernetes Prometheus:
Kubernetes and Prometheus interact without any issues. Projects from the Cloud Native Computing Foundation (CNCF), Kubernetes, and Prometheus, coexist harmoniously.
Prometheus offers APIs that make it simple to access monitoring metrics.
There are many libraries and exporters available through Prometheus for gathering application metrics.
All exporters are intended to increase Prometheus' coverage.
A standardized method that makes data collection easier is the pull-based model for time-series data collection.
Prometheus offers a simplified and constrained data model for pure telemetry monitoring, but this model does not fully contextualize events.
Prometheus only offers summarized data that exporters periodically scrape.
Prometheus was not made to monitor legacy infrastructure; it is best for Kubernetes.
Authentication and encryption are not included in Prometheus' data collection capabilities. Telemetry information is accessible to any user or component with network access.
Always essential to any application are metrics. If you want to successfully expand your software, you must check on them frequently. Kubernetes node and cluster monitoring are very simple with Prometheus. It offers various metrics, including counters, graphs, summaries, gauges, etc.
Using Prometheus, you can keep an eye on your Kubernetes cluster or services. A few other microservices that are older than Prometheus require exporters in order to use their metrics in Prometheus, despite the fact that many microservices offer metrics that can be directly converted to the Prometheus endpoint.
The undistributed storage layer in Prometheus is not designed for long-term storage and is in charge of maintaining data for months or years. Prometheus excels for alerting and short-term trends but falls short for more in-depth historical data requirements.
Prometheus has a simple UI for experimenting with PromQL queries, but it is not a dashboarding solution. However, it uses Grafana for dashboarding, which increases the setup's complexity.
We hope this article made you familiar with what is Prometheus in Kubernetes and how it can help you. To get certified in docker and Kubernetes, do check out.
To read all the available metrics from Nodes, Pods, Deployments, etc., Prometheus uses the Kubernetes APIs. This calls for the creation of an RBAC policy that is bound to the monitoring namespace and has read access to the necessary API groups.
The storage. local. path flag specifies the directory where Prometheus keeps its on-disk time series data. In terms of the working directory, the default path is ./data, which is fine for quick tests but probably not what you want for actual operations.
Service discovery is a method used by Prometheus to find targets for scraping. Labels, annotations, and a mechanism for tracking status and changes for various elements are all included in Kubernetes clusters. Prometheus needs to use the Kubernetes API to find targets.
YAML files are used by Prometheus to access resources. Additionally, it aids in information retrieval when scraping components of the Kubernetes cluster. The documentation for Prometheus contains more information on YAML configuration. You can install Prometheus as a container on a Kubernetes cluster after the configuration is finished.
Yes, you can monitor multiple Kubernetes clusters and providers using the Prometheus/Grafana combination.
| Name | Date | Fee | Know more |
|---|
