Prerequisites

In my previous article, we looked at how we can set up Prometheus and Node Exporter as systemd services on an Ubuntu instance.

But in the above setup, we were accessing the Prometheus expression browser and Node Exporter metrics endpoints via HTTP and there was no authentication enabled.
We are going to address these issues in this article.

HTTPS

Node Exporter

Create a new directory for storing the Node Exporter configuration file and change its ownership to "node_exporter" user

$ sudo mkdir -p /etc/node_exporter

$ sudo chown node_exporter:node_exporter /etc/node_exporter

Create a configuration file for Node Exporter and change its ownership to "node_exporter" user

$ sudo touch /etc/node_exporter/node_exporter.yml

$ sudo chown node_exporter:node_exporter /etc/node_exporter/node_exporter.yml

Generate a certificate and key using OpenSSL

$ sudo openssl req -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout prom.key -out prom.crt -subj "/C=US/ST=California/L=Oakland/O=MyOrg/CN=localhost" -addext "subjectAltName = DNS:localhost"

$ ls
prom.crt  prom.key

Copy the certificate and key files to the Node Exporter configuration directory and change their ownership to "node_exporter" user

$ sudo cp prom.* /etc/node_exporter

$ sudo chown node_exporter:node_exporter prom.crt

$ sudo chown node_exporter:node_exporter prom.key

Add the tls_server_config details to the configuration file

$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

Update the systemd unit file of node_exporter service to include the above configuration file

$ sudo vi /etc/node_exporter/node_exporter.yml

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter \
        --web.config.file /etc/node_exporter/node_exporter.yml

[Install]
WantedBy=multi-user.target

Restart the node_exporter service and verify its status

$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter

The metrics endpoint is now accessible via HTTPS

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down

Prometheus

Copy the certificate and key files to the Prometheus configuration directory and change its ownership to "prometheus" user

$ sudo cp prom.* /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prom.crt

$ sudo chown prometheus:prometheus /etc/prometheus/prom.key

Update the Prometheus configuration file to include scheme and tls_config for the "node_exporter" job

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
    static_configs:
      - targets: ["172.31.81.113:9100"]

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Restart the prometheus service to take effect the new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up

Now we have enabled secure communication between the Prometheus server and Node Exporter but still our Prometheus expression browser is using an HTTP connection

Create a new configuration file for configuring HTTPS connection and change its ownership to "prometheus" user

$ sudo touch /etc/prometheus/webconfig.yml

$ sudo chown prometheus:prometheus /etc/prometheus/webconfig.yml

Add the tls_server_config details to the newly created configuration file

$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

Update the systemd unit file of prometheus service to include the above configuration file

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries \
    --web.config.file /etc/prometheus/webconfig.yml

[Install]
WantedBy=multi-user.target

Now we can access the Prometheus expression browser using HTTPS

Authentication

Install the apache2 utils package to generate a password

$ sudo apt update

$ sudo apt install apache2-utils

Generate the password using the htpasswd tool

$ htpasswd -nBC 16 admin | tr -d ':\n'
New password:
Re-type new password:
admin$2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O

Node Exporter

Update the Node Exporter configuration file to include basic authentication

$ sudo vi /etc/node_exporter/node_exporter.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key
basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O

Restart the node_exporter service and verify its status

$ sudo systemctl restart node_exporter

$ sudo systemctl status node_exporter

Now access the Node Exporter metrics endpoint and it will show a login prompt

Open Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing down

Prometheus

Update the Prometheus configuration file to include basic authentication for the "node_exporter" job

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "node_exporter"
    scheme: https
    tls_config:
      ca_file: prom.crt
      insecure_skip_verify: true
      basic_auth:
        username: admin
        password: Password!
    static_configs:
      - targets: ["172.31.81.113:9100"]

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml
Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Restart the prometheus service to effect new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Open the Prometheus expression browser and navigate to Status -> Targets, we can see our node_exporter target is showing up

Now we have enabled basic authentication between the Prometheus server and Node Exporter but we need to enable authentication for the Prometheus server

Update the below configuration file to include basic authentication on the Prometheus server

$ sudo vi /etc/prometheus/webconfig.yml

tls_server_config:
  cert_file: prom.crt
  key_file: prom.key

basic_auth_users:
  admin: $2y$16$fMgRIvex1Rn67dHErc.Ft.CSI3ng5b457FOe9JIZkMB7k7p3PfS8O

Restart the prometheus service for the new changes and check its status

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Now access the Prometheus expression browser and it will show a login prompt

That's all for now

Reference

https://prometheus.io/docs/prometheus/latest/configuration/https/

https://kodekloud.com/courses/prometheus-certified-associate-pca/