Monitor Spring Boot Apps using Prometheus on Kubernetes
This posts show how to monitor Spring Boot apps using Prometheus when running on Kubernetes. You should already have a running Kubernetes cluster.
Install Helm
I'm going to use Prometheus Operator to create and manage Prometheus on Kubernetes. Prometheus Operator is installed using Helm, so Helm needs to be installed first. On macOS, simply use Homebrew. This post uses Helm 2.14.0
.
$ brew install kubernetes-helm
Then you need to initialize Helm.
$ helm init
Install Prometheus Operator
Before installing Prometheus Operator, you need to make sure the current account has the permission to manage CustomResourceDefinitions
. A simple way is to assign the role cluster-admin
to current user, see here. Otherwise, you'll see error like below.
Error: customresourcedefinitions.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" is forbidden: User "system:serviceaccount:kube-system:default" cannot delete resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope
If you are using the default system service account, you can use the following command to create the role binding. Replace system:serviceaccount:kube-system:default
with the name of your own account.
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=system:serviceaccount:kube-system:default
Now you can install Prometheus Operator using Helm.
$ helm install stable/prometheus-operator --name prometheus-operator
After installation, you can use the following command to access the Prometheus web UI on http://localhost:9090. prometheus-operated
is the service to access Prometheus API and web UI.
$ kubectl port-forward svc/prometheus-operated 9090:9090
Create ServiceMonitor
By default, Prometheus Operator monitors various targets of the Kubernetes cluster. To add your own targets, you need to create ServiceMonitor
s.
You need to have a Service
for your Spring Boot application first.
The yaml file below creates a new ServiceMonitor
that matches the label app
with name my-service
. To expose metrics of the Spring Boot app, you need to add Spring Boot Actuator and have the prometheus
endpoint enabled. The path of the metrics endpoint is /actuator/prometheus
. The port api
is the port defined in the service. Use kubectl create
to create this ServiceMonitor
.
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: my-service
labels:
release: prometheus-operator
spec:
selector:
matchLabels:
app: my-service
endpoints:
- port: api
path: '/actuator/prometheus'
interval: 10s
honorLabels: true
Please note, the label release: prometheus-operator
is important for the ServiceMonitor
to work. If you run kubectl get servicemonitor prometheus-operator -o yaml
, you can see the following output. This means Prometheus Operator only selects ServiceMonitor
s with the label release: prometheus-operator
.
serviceMonitorSelector:
matchLabels:
release: prometheus-operator
Now you should see the target for the Spring Boot app in the list of http://localhost:9090/targets.