Relabel instance to hostname in Prometheus

I just came across this problem and the solution is to use a group_left to resolve this problem. You can't relabel with a nonexistent value in the request, you are limited to the different parameters that you gave to Prometheus or those that exists in the module use for the request (gcp,aws...).

So the solution I used is to combine an existing value containing what we want (the hostnmame) with a metric from the node exporter. Our answer exist inside the node_uname_info metric which contains the nodename value.

I used the answer to this post as a model for my request: https://stackoverflow.com/a/50357418 .

The solution is this one:

node_memory_Active_bytes
  * on(instance) group_left(nodename)
node_uname_info

With this, the node_memory_Active_bytes metric which contains only instance and job labels by default, gets an additional nodename label that you can use in the description field of Grafana.

Hope that this will help others.

This solution stores data at scrape-time with the desired labels, no need for funny PromQL queries or hardcoded hacks. It does so by replacing the labels for scraped data by regexes with relabel_configs.

By default, instance is set to __address__, which is $host:$port.

First attempt: In order to set the instance label to $host, one can use relabel_configs to get rid of the port of your scaping target:

  - job_name: 'whatever'
    static_configs:
      - targets: [
            'yourhost.lol:9001'
        ]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+)(:[0-9]+)?'
        replacement: '${1}'

But the above would also overwrite labels you wanted to set e.g. in the file_sd_configs:

[
    {
        "targets": ['yourhost.lol:9001'],
        "labels": {
            "instance": 'node42'
        }
    }
]

Solution: If you want to retain these labels, the relabel_configs can rewrite the label multiple times be done the following way:

  - job_name: 'whatever'
    metrics_path: /metric/rolf
    file_sd_configs:
      - files:
        - rolf_exporter_targets.yml
    relabel_configs:
      - source_labels: [instance]
        target_label: __tmp_instance
        regex: '(.+)'
        replacement: '${1};'
      - source_labels: [__tmp_instance, __address__]
        separator: ''
        target_label: instance
        regex: '([^:;]+)((:[0-9]+)?|;(.*))'
        replacement: '${1}'

Doing it like this, the manually-set instance in sd_configs takes precedence, but if it's not set the port is still stripped away.

I found hardcode solution:

    global:
      scrape_interval: 5s
      scrape_timeout: 5s
      external_labels:
        monitor: 'Prometheus'

    scrape_configs:

    - job_name: 'shelby'
      static_configs:
      - targets:
        - 10.100.0.01:9100
      relabel_configs:
      - source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: 'shelby'

    - job_name: 'camaro'
      static_configs:
      - targets:
        - 10.101.0.02:9100
      relabel_configs:
      - source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: 'camaro'

    - job_name: 'verona'
      static_configs:
      - targets:
        - 10.101.0.03:9100
      relabel_configs:
      - source_labels: [__address__]
        regex: '.*'
        target_label: instance
        replacement: 'verona'

Result:

    node_load15{instance="camaro",job="camaro"}    0.16
    node_load15{instance="shelby",job="shelby"}    0.4
    node_load15{instance="verona",job="verona"}    0.07