This posts show how to monitor Spring Boot apps using Prometheus when running on Kubernetes. You should already have a running Kubernetes cluster.

Install Helm

I'm going to use Prometheus Operator to create and manage Prometheus on Kubernetes. Prometheus Operator is installed using Helm, so Helm needs to be installed first. On macOS, simply use Homebrew. This post uses Helm 2.14.0.

$ brew install kubernetes-helm

Then you need to initialize Helm.

$ helm init

Install Prometheus Operator

Before installing Prometheus Operator, you need to make sure the current account has the permission to manage CustomResourceDefinitions. A simple way is to assign the role cluster-admin to current user, see here. Otherwise, you'll see error like below.

Error: customresourcedefinitions.apiextensions.k8s.io "alertmanagers.monitoring.coreos.com" is forbidden: User "system:serviceaccount:kube-system:default" cannot delete resource "customresourcedefinitions" in API group "apiextensions.k8s.io" at the cluster scope

If you are using the default system service account, you can use the following command to create the role binding. Replace system:serviceaccount:kube-system:default with the name of your own account.

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=system:serviceaccount:kube-system:default

Now you can install Prometheus Operator using Helm.

$ helm install stable/prometheus-operator --name prometheus-operator

After installation, you can use the following command to access the Prometheus web UI on http://localhost:9090. prometheus-operated is the service to access Prometheus API and web UI.

$ kubectl port-forward svc/prometheus-operated 9090:9090

Create ServiceMonitor

By default, Prometheus Operator monitors various targets of the Kubernetes cluster. To add your own targets, you need to create ServiceMonitors.

You need to have a Service for your Spring Boot application first.

The yaml file below creates a new ServiceMonitor that matches the label app with name my-service. To expose metrics of the Spring Boot app, you need to add Spring Boot Actuator and have the prometheus endpoint enabled. The path of the metrics endpoint is /actuator/prometheus. The port api is the port defined in the service. Use kubectl create to create this ServiceMonitor.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-service
  labels:
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: my-service
  endpoints:
  - port: api
    path: '/actuator/prometheus'
    interval: 10s
    honorLabels: true

Please note, the label release: prometheus-operator is important for the ServiceMonitor to work. If you run kubectl get servicemonitor prometheus-operator -o yaml, you can see the following output. This means Prometheus Operator only selects ServiceMonitors with the label release: prometheus-operator.

serviceMonitorSelector:
  matchLabels:
    release: prometheus-operator

Now you should see the target for the Spring Boot app in the list of http://localhost:9090/targets.