Operators
Node Monitoring
Prometheus

Prometheus Setup

The following is a guide outlining the steps to setup Prometheus to monitor a Tangle node. If you do not have Tangle node setup yet, please review the Tangle Node Quickstart setup guide here. It is important to note that this guide's purpose is to help you get started with monitoring your Tangle node, not to advise on how to setup a node securely. Please take additional security and privacy measures into consideration.

In this guide we will configure the following modules to scrape metrics from the running Tangle node.

  • Prometheus is the central module; it pulls metrics from different sources to provide them to the Grafana dashboard and Alert Manager.
  • Node exporter provides hardware metrics of the dashboard.
  • Process exporter provides processes metrics for the dashboard (optional).

What is Prometheus?

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. It is now a standalone open source project and maintained independently of any company. To learn more about Prometheus, please visit the official docs site here (opens in a new tab).

Getting Started

Let's first start by downloading the latest releases of the above mentioned modules (Prometheus, Process exporter, and Node exporter).

This guide assumes the user has root access to the machine running the Tangle node, and following the below steps inside that machine.

1. Download Prometheus

AMD version:

AMD
wget https://github.com/prometheus/prometheus/releases/download/v2.40.3/prometheus-2.40.3.darwin-amd64.tar.gz

ARM version:

ARM
wget https://github.com/prometheus/prometheus/releases/download/v2.40.3/prometheus-2.40.3.darwin-arm64.tar.gz

2. Download Node Exporter

AMD version:

AMD
wget https://github.com/prometheus/node_exporter/releases/download/v1.40.0/node_exporter-1.4.0.darwin-amd64.tar.gz

ARM version:

ARM
wget https://github.com/prometheus/node_exporter/releases/download/v1.40.0/node_exporter-1.4.0.darwin-arm64.tar.gz

3. Download Process Exporter

AMD version:

AMD
wget https://github.com/ncabatoff/process-exporter/releases/download/v0.7.10/process-exporter-0.7.10.linux-amd64.tar.gz

ARM version:

ARM
wget https://github.com/ncabatoff/process-exporter/releases/download/v0.7.10/process-exporter-0.7.10.linux-arm64.tar.gz

For other linux distrubutions please visit official release page here (opens in a new tab).

4. Extract the Downloaded Files:

Run the following command:

tar
tar xvf prometheus-*.tar.gz &&
tar xvf node_exporter-*.tar.gz &&
tar xvf process-exporter-*.tar.gz

5. Copy the Extracted Files into /usr/local/bin:

Note: The example below makes use of the linux-amd64 installations, please update to make use of the target system you have installed.

We are first going to copy the prometheus binary:

cp
sudo cp ./prometheus-*.linux-amd64/prometheus /usr/local/bin/

Next, we are going to copy over the prometheus console libraries:

cp
sudo cp -r ./prometheus-*.linux-amd64/consoles /etc/prometheus &&
sudo cp -r ./prometheus-*.linux-amd64/console_libraries /etc/prometheus

We are going to do the same with node-exporter and process-exporter:

cp
sudo cp ./node_exporter-*.linux-amd64/node_exporter /usr/local/bin/ &&
sudo cp ./process-exporter-*.linux-amd64/process-exporter /usr/local/bin/

6. Create Dedicated Users:

Now we want to create dedicated users for each of the modules we have installed:

useradd
sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus &&
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter &&
sudo useradd --no-create-home --shell /usr/sbin/nologin process-exporter

7. Create Directories for Prometheus, and Process exporter:

mkdir
sudo mkdir /var/lib/prometheus &&
sudo mkdir /etc/process-exporter

8. Change the Ownership for all Directories:

We need to give our user permissions to access these directories:

prometheus:

chown
sudo chown prometheus:prometheus /etc/prometheus/ -R &&
sudo chown prometheus:prometheus /var/lib/prometheus/ -R &&
sudo chown prometheus:prometheus /usr/local/bin/prometheus

node_exporter:

chwon
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

process-exporter:

chown
sudo chown process-exporter:process-exporter /etc/process-exporter -R &&
sudo chown process-exporter:process-exporter /usr/local/bin/process-exporter

9. Finally, let's clean up these directories:

rm
rm -rf ./prometheus* &&
rm -rf ./node_exporter* &&
rm -rf ./process-exporter*

Great! You have now installed and setup your environment. The next series of steps will be configuring each service.

Configuration

If you are interested to see how we configure the Tangle Network nodes for monitoring check out the Tangle Network deployment README (opens in a new tab)

Prometheus

Let"s edit the Prometheus config file and add all the modules in it:

nano
sudo nano /etc/prometheus/prometheus.yml

Add the following code to the file and save:

promtheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
 
rule_files:
  - 'rules.yml'
 
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
 
scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "substrate_node"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9615"]
  - job_name: "node_exporter"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9100"]
  - job_name: "process-exporter"
    scrape_interval: 5s
    static_configs:
      - targets: ["localhost:9256"]
  • scrape_interval defines how often Prometheus scrapes targets, while evaluation_interval controls how often the software will evaluate rules.
  • rule_files set the location of Alert manager rules we will add next.
  • alerting contains the alert manager target.
  • scrape_configs contain the services Prometheus will monitor.

You can notice the first scrap where Prometheus monitors itself.

Process exporter

Process exporter needs a config file to be told which processes they should take into account:

nano
sudo touch /etc/process-exporter/config.yml
sudo nano /etc/process-exporter/config.yml

Add the following code to the file and save:

config.yml
process_names:
  - name: "{{.Comm}}"
    cmdline:
    - '.+'

Service Setup

Prometheus

Create and open the Prometheus service file:

promtheus.service
sudo tee /etc/systemd/system/prometheus.service > /dev/null << EOF
[Unit]
  Description=Prometheus Monitoring
  Wants=network-online.target
  After=network-online.target
 
[Service]
  User=prometheus
  Group=prometheus
  Type=simple
  ExecStart=/usr/local/bin/prometheus \
   --config.file /etc/prometheus/prometheus.yml \
   --storage.tsdb.path /var/lib/prometheus/ \
   --web.console.templates=/etc/prometheus/consoles \
   --web.console.libraries=/etc/prometheus/console_libraries
  ExecReload=/bin/kill -HUP $MAINPID
 
[Install]
  WantedBy=multi-user.target
EOF

Node exporter

Create and open the Node exporter service file:

node_exporter.service
sudo tee /etc/systemd/system/node_exporter.service > /dev/null << EOF
[Unit]
  Description=Node Exporter
  Wants=network-online.target
  After=network-online.target
 
[Service]
  User=node_exporter
  Group=node_exporter
  Type=simple
  ExecStart=/usr/local/bin/node_exporter
 
[Install]
  WantedBy=multi-user.target
EOF

Process exporter

Create and open the Process exporter service file:

process-exporter.service
sudo tee /etc/systemd/system/process-exporter.service > /dev/null << EOF
[Unit]
  Description=Process Exporter
  Wants=network-online.target
  After=network-online.target
 
[Service]
  User=process-exporter
  Group=process-exporter
  Type=simple
  ExecStart=/usr/local/bin/process-exporter \
   --config.path /etc/process-exporter/config.yml
 
[Install]
WantedBy=multi-user.target
EOF

Starting the Services

Launch a daemon reload to take the services into account in systemd:

deamon-reload
sudo systemctl daemon-reload

Next, we will want to start each service:

prometheus:

start serive
sudo systemctl start prometheus.service

node_exporter:

start serive
sudo systemctl start node_exporter.service

process-exporter:

start serive
sudo systemctl start process-exporter.service

And check that they are working fine:

prometheus:

status
systemctl status prometheus.service

node_exporter:

status
systemctl status node_exporter.service

process-exporter:

status
systemctl status process-exporter.service

If everything is working adequately, activate the services!

prometheus:

enable
sudo systemctl enable prometheus.service

node_exporter:

enable
sudo systemctl enable node_exporter.service

process-exporter:

enable
sudo systemctl enable process-exporter.service

Amazing! We have now completely setup our Prometheus monitoring and are scraping metrics from our running Tangle node.

You can view those metrics on the Prometheus dashboard by going to http://localhost:9090/metrics !