Getting error "Get http://localhost:9443/metrics: dial tcp 127.0.0.1:9443: connect: connection refused"
Since the targets are not running inside the prometheus container, they cannot be accessed through localhost. You need to access them through the host private IP or by replacing
Your prometheus container isn't running on host network. It's running on its own bridge (the one created by docker-compose). Therefore the scrape config for peer should point at the IP of the peer container.
Recommended way of solving this:
- Run prometheus and grafana in the same network as the fabric network. In you docker-compose for prometheus stack you can reference it like this:
networks: default: external: name: <your-hyperledger-network>
docker network ls to find the network name )
Then you can use
http://<peer_container_name>:9443 in your scrape config
The problem: On Prometheus you added a service for scraping but on
http://localhost:9090/targets the endpoint state is Down
with an error:
Get http://localhost:9091/metrics: dial tcp 127.0.0.1:9091: connect: connection refused
prometheus.yml you need to verify that
- scraping details pointing to the right endpoint.
- the yml indentation is correct.
curl -v http://<serviceip>:<port>/metricsshould prompt the metrics in plaintext in your terminal.
Note: If you pointing to some service in another docker container, your localhost might be represented not as localhost but as
servicename ( service name that shown in
docker ps ) or
docker.host.internal (the internal ip that running the docker container ).
for this example: I'll be working with 2 dockers containers prometheus and "myService".
sudo docker ps CONTAINER ID IMAGE CREATED PORTS NAMES abc123 prom/prometheus:latest 2 hours ago 0.0.0.0:9090->9090/tcp prometheus def456 myService/myService:latest 2 hours ago 0.0.0.0:9091->9091/tcp myService
and then edit the file
prometheus.yml (and rerun prometheus)
- job_name: myService scrape_interval: 15s scrape_timeout: 10s metrics_path: /metrics static_configs: - targets: // Presenting you 3 options - localhost:9091 // simple localhost - docker.host.internal:9091 // the localhost of agent that runs the docker container - myService:9091 // docker container name (worked in my case)