When configuring Prometheus Operator persistence with Rancher local path provisioner running on a Kubernetes cluster created with rke. I found an issue with subPath in volumeMount of Prometheus.

After configuring local-path-provisioner in Kubernetes, it's easy to configure Prometheus Operator to use the provisioner. Below is the values.yml file to configure Prometheus Operator.

prometheus:
  prometheusSpec:
    storageSpec:
       volumeClaimTemplate:
         spec:
           storageClassName: local-path
           resources:
             requests:
               storage: 5G

alertmanager:
  alertmanagerSpec:
    storage:
        volumeClaimTemplate:
          spec:
            storageClassName: local-path
            resources:
              requests:
                storage: 5G  

However, the Prometheus pod keeps failing on permission denied error when trying to create directory /prometheus. I found out this is caused by subPath: prometheus-db in Prometheus's stateful set definition. Although Alert Manger pod can start, it's actually not working properly. The subPath of Prometheus is hardcoded in source code of Prometheus Operator. The subPath of Alert Manger is also hardcoded to alertmanager-db.

There are already GitHub issues about this problem, see 1, 2, 3 and 4. The workaround is from this comment by modifying Rancher's cluster config.

services:
  kubelet:
    extra_args:
      containerized: "true"
    extra_binds: 
      - "/:/rootfs:rshared"

Below is another workaround I used before. The drawback of this workaround is that when Prometheus Operator pod is recreated, the stateful sets will be reset to original states. Then you need to run kubectl patch again.

Because the stateful sets for Prometheus and Alert Manager are generated by Prometheus Operator, we need to patch them after Prometheus Operator is installed. The following code shows how to patch statefulset using kubectl patch command. test is the name specified when installing Prometheus Operator using Helm. /spec/template/spec/containers/0/volumeMounts/1/subPath is the JSON path to remove for subPath of the volumeMount.

kubectl patch statefulset prometheus-test-prometheus-prometheus --type='json' -p '
- op: remove
  path: "/spec/template/spec/containers/0/volumeMounts/1/subPath"
'

kubectl patch statefulset alertmanager-test-prometheus-alertmanager --type='json' -p '
- op: remove
  path: "/spec/template/spec/containers/0/volumeMounts/1/subPath"

After patching the stateful sets, both Prometheus and Alert Manager should work.