Prometheus - Installation on Amazon EC2 (Ubuntu) - Part 3

In my previous article, we looked at how we can quickly set up Prometheus and Node Exporter.

But in that setup, Prometheus and Node Exporter processes are running in the foreground and it won't start the process after a reboot of the VM.
We are going to fix that issue in this article.

Prerequisites

• Setup an EC2 instance of type t2.small

• Ubuntu 22.04 LTS as AMI

• 30 GB of hard disk space

• Open ports 22 for SSH, 9090 for Prometheus and 9100 for Node Exporter

Installation

Prometheus

Login to your EC2 instance

$ ssh -i prometheus.pem ubuntu@3.80.133.119

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

Create a new user for managing the Prometheus process
Upon executing the command it will create a user called prometheus without a home directory and shell as /bin/false

$ sudo useradd --shell /bin/false --no-create-home prometheus

Create a new directory for storing our Prometheus configuration file and update the ownership of the directory

$ sudo mkdir -p /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus

Create another directory for storing Prometheus time series data and update the ownership of the directory

$ sudo mkdir -p /var/lib/prometheus

$ sudo chown prometheus:prometheus /var/lib/prometheus

Download and extract the latest Prometheus release from their GitHub page

$ wget https://github.com/prometheus/prometheus/releases/download/v2.46.0/prometheus-2.46.0.linux-amd64.tar.gz

$ tar -xzvf prometheus-2.46.0.linux-amd64.tar.gz

Change the directory and install Prometheus and Promtool binaries

$ cd prometheus-2.46.0.linux-amd64

$ sudo install prometheus /usr/local/bin

$ sudo install promtool /usr/local/bin

Change the permission of these binaries to prometheus user

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

$ sudo chown prometheus:prometheus /usr/local/bin/promtool

Check the version of prometheus or promtool binaries

$ prometheus --version
prometheus, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

$ promtool --version
promtool, version 2.46.0 (branch: HEAD, revision: cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0)
  build user:       root@42454fc0f41e
  build date:       20230725-12:31:24
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo,builtinassets,stringlabels

Copy the consoles and console_libraries directory which is used for dashboarding and visualization

$ sudo cp -rv consoles /etc/prometheus

$ sudo cp -rv console_libraries /etc/prometheus

Change the ownerships of the directories recursively to prometheus user

$ sudo chown -R prometheus:prometheus /etc/prometheus/consoles

$ sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries

Copy the configuration file to the above-mentioned directory and change its ownership to prometheus user

$ sudo cp prometheus.yml /etc/prometheus

$ sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Create a systemd unit file for the Prometheus service

$ sudo vi /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus \
    --web.console.templates /etc/prometheus/consoles \
    --web.console.libraries /etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Start the prometheus process and enable the process at boot

$ sudo systemctl daemon-reload

$ sudo systemctl start prometheus

$ sudo systemctl enable prometheus

Check the prometheus service status

$ sudo systemctl is-active prometheus
active

$ sudo systemctl is-enabled prometheus
enabled

Open http://3.80.133.119:9090 in the browser and you can see the Prometheus expression browser

Node Exporter

Create a new user for managing the node exporter process

$ sudo useradd --shell /bin/false --no-create-home node_exporter

Download and extract the latest Node Exporter release from their GitHub page

$ wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz

$ tar -xzvf node_exporter-1.6.1.linux-amd64.tar.gz

Change the directory and install the node_exporter binary

$ cd node_exporter-1.6.1.linux-amd64

$ sudo install node_exporter /usr/local/bin

Change the ownership of the binary to node_exporter user

$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

Check the version of the node_exporter binary

$ node_exporter --version
node_exporter, version 1.6.1 (branch: HEAD, revision: 4a1b77600c1873a8233f3ffb55afcedbb63b8d84)
  build user:       root@586879db11e5
  build date:       20230717-12:10:52
  go version:       go1.20.6
  platform:         linux/amd64
  tags:             netgo osusergo static_build

Create a systemd unit file for the Node Exporter service

$ sudo vi /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Start the node_exporter process and enable the process at boot

$ sudo systemctl daemon-reload

$ sudo systemctl start node_exporter

$ sudo systemctl enable node_exporter

Check the status of the node_exporter process

$ sudo systemctl is-active node_exporter
active

$ sudo systemctl is-enabled node_exporter
enabled

Open 3.80.133.119:9100/metrics in your browser to view all the metrics

Configuration

Prometheus

Now we have installed Prometheus and Node Exporter on the server and we need to modify the Prometheus configuration file to add new targets to scrape metrics

In the installation step, we already copied a default Prometheus configuration file to the /etc/prometheus location. You can edit this file or replace it with the below contents.

Here I have added a new job called "node_exporter" to scrape metrics from the same VM, instead of using localhost I have used the private IP of the instance

$ sudo vi /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  scrape_timeout: 10s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node_exporter"
    static_configs:
      - targets: ["172.31.93.117:9100"]

Validate the configuration file using the promtool

$ promtool check config /etc/prometheus/prometheus.yml

Checking /etc/prometheus/prometheus.yml
 SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Restart the Prometheus service to effect new configuration changes

$ sudo systemctl restart prometheus

$ sudo systemctl status prometheus

Open the expression browser and navigate to Status -> Targets to view the newly added "node_exporter" target

We can view an example metric called "node_memory_Active_bytes" using the expression browser

Select the Graph option to view it as a graph

That's all for now

Reference

https://prometheus.io/docs/prometheus/latest/getting_started/

https://kodekloud.com/courses/prometheus-certified-associate-pca/