local-path-provisioner in Kubernetes, it's easy to configure Prometheus Operator to use the provisioner. Below is the
values.yml file to configure Prometheus Operator.
prometheus: prometheusSpec: storageSpec: volumeClaimTemplate: spec: storageClassName: local-path resources: requests: storage: 5G alertmanager: alertmanagerSpec: storage: volumeClaimTemplate: spec: storageClassName: local-path resources: requests: storage: 5G
However, the Prometheus pod keeps failing on
permission denied error when trying to create directory
/prometheus. I found out this is caused by
subPath: prometheus-db in Prometheus's stateful set definition. Although Alert Manger pod can start, it's actually not working properly. The
subPath of Prometheus is hardcoded in source code of Prometheus Operator. The
subPath of Alert Manger is also hardcoded to
services: kubelet: extra_args: containerized: "true" extra_binds: - "/:/rootfs:rshared"
Below is another workaround I used before. The drawback of this workaround is that when Prometheus Operator pod is recreated, the stateful sets will be reset to original states. Then you need to run
Because the stateful sets for Prometheus and Alert Manager are generated by Prometheus Operator, we need to patch them after Prometheus Operator is installed. The following code shows how to patch
kubectl patch command.
test is the name specified when installing Prometheus Operator using Helm.
/spec/template/spec/containers/0/volumeMounts/1/subPath is the JSON path to remove for
subPath of the
kubectl patch statefulset prometheus-test-prometheus-prometheus --type='json' -p ' - op: remove path: "/spec/template/spec/containers/0/volumeMounts/1/subPath" ' kubectl patch statefulset alertmanager-test-prometheus-alertmanager --type='json' -p ' - op: remove path: "/spec/template/spec/containers/0/volumeMounts/1/subPath"
After patching the stateful sets, both Prometheus and Alert Manager should work.