After installing Prometheus Operator on Kubernetes cluster, node exporter runs on each cluster nodes to report metrics about the node. However, in the targets UI of Prometheus, I can only see one node exporter is UP, all other node exporters are failing with context deadline exceeded error. The only node exporter that works is running on the same node as Prometheus itself. So it's network issue that blocks the communication between node exporters and Prometheus itself.

The solution is very easy, you only need to edit the YAML file of daemon set prometheus-node-exporter to delete hostNetwork: true. If you are using Helm chart prometheus-operator to install Prometheus Operator, you can override the hostNetwork setting using a values file with -f option.

prometheus-node-exporter:
  hostNetwork: false

After all pods are recreated, all node exporters should be UP in Prometheus.