After installing Prometheus Operator on Kubernetes cluster, node exporter runs on each cluster nodes to report metrics about the node. However, in the targets UI of Prometheus, I can only see one node exporter is
UP, all other node exporters are failing with
context deadline exceeded error. The only node exporter that works is running on the same node as Prometheus itself. So it's network issue that blocks the communication between node exporters and Prometheus itself.
The solution is very easy, you only need to edit the YAML file of daemon set
prometheus-node-exporter to delete
hostNetwork: true. If you are using Helm chart prometheus-operator to install Prometheus Operator, you can override the
hostNetwork setting using a values file with
prometheus-node-exporter: hostNetwork: false
After all pods are recreated, all node exporters should be
UP in Prometheus.