Fix Context Deadline Exceeded Error in Prometheus Operator
After installing Prometheus Operator on Kubernetes cluster, node exporter runs on each cluster nodes to report metrics about the node. However, in the targets UI of Prometheus, I can only see one node exporter is UP
, all other node exporters are failing with context deadline exceeded
error. The only node exporter that works is running on the same node as Prometheus itself. So it's network issue that blocks the communication between node exporters and Prometheus itself.
The solution is very easy, you only need to edit the YAML file of daemon set prometheus-node-exporter
to delete hostNetwork: true
. If you are using Helm chart prometheus-operator to install Prometheus Operator, you can override the hostNetwork
setting using a values file with -f
option.
prometheus-node-exporter:
hostNetwork: false
After all pods are recreated, all node exporters should be UP
in Prometheus.