Monitoring with Prometheus

What is Monitoring ?

Monitoring tools are used to continuously keep track of the status of the system Where applications are deployed and running, in order to have the earliest warning of failures, defects or problems and to improve them. Also we can use this same for getting analytics about machine usage so that we can upgrade or downgrade to make performance and cost optimal.

With distributed and micro services architecture, which is general trend now a days to deploy application on multiple machines ,It is overhead to keep tracking all services are up and running and using all machine resources optimised way.   

This post series will walk through you setting up prometheus  ,Alert Manager and Grafana together.

Topics in this series will be covered as below -   

# Setting up Prometheus in 10 minutes

# Setting up Alert Manager and Alert Rules in 10 minutes

# Application monitoring

# Node js process monitoring

# Nginx Monitoring

# Setting up Grafana dashboards.

Setting up Prometheus in 5 minutes

Prometheus is an open-source systems monitoring and alerting toolkit with an active ecosystem.

Prometheus is widely used monitoring tool now days and it does support so many out of box solutions, So we at Crevise recommend to implement your monitoring system with Prometheus.    

In below diagram, We have setup prometheus ,Alert manager and Grafana on one machine and we have 4 machines where  S1 and S2 services are deployed (with replica 2 ).

for this series ,we are using ubuntu machines to install all services. lets get started

Sample Architecture

  

Installing Prometheus -

Download Prometheus binary from official download page.

wget https://github.com/prometheus/prometheus/releases/download/v1.7.1/prometheus-1.7.1.linux-amd64.tar.gz

This will download tar file , we can extract it using below command

tar -xzvf prometheus-1.7.1.linux-amd64.tar.gz

Now we can run directly this binary to start prometheus but wait for now. lets first install node exporter.

Install Node Exporter-

Node exporters will be installed on machines which we want to monitor ,Node exporter will send prometheus server all metrics about instances  e.g. CPU usage ,Memory Usage, IO ,File Disk Usage etc

Installing node exporter again is very simple.

download node exporter binary and run it

Wget https://github.com/prometheus/node_exporter/releases/download/v0.14.0/node_exporter-0.14.0.linux-amd64.tar.gz

Extract it using below command

tax -xzvf node_exporter-0.14.0.linux-amd64.tar.gz

You can run this binary as below

./node_exporter-0.14.0.linux-amd64 &  

Now go to your cloud provider console and open port 9100. Node exporter runs on this port.

So you can able to see running node exporter web on this port in your browser.

Run node exporter in same way on all machines which you want to monitor.

Configure prometheus

Now as ,Node Exporter is running , we can configure node exporter to push all Metrics data to our prometheus server. To do so, Go to Prometheus directory which we extract in first step.

You can see prometheus.yml file in that folder.

global:

 evaluation_interval: 15s

 external_labels:

   monitor: 'Prometheus_Alerts'

 scrape_interval: 15s

scrape_configs:

 -

   job_name: 'node'

   static_configs:

     - targets:

         - “xxx.xxx.xxx.xxx:9100”

         - “yyy.yyy.yyy.yyy:9100”

In above file , we can see we want to monitor 2 machines on which node exporter is installed.
We have configured these 2 machines in target block

scrape_interval - is interval time after which node exporter will push metrics to prometheus.

job_name - should be node in this car as we are fetching machine metrics.

Configuration done!

Run prometheus

Now we have installed prometheus and node exporters as well as prometheus is configured to get metrics from node exporters from diff machines, lets run prometheus now

Go to Prometheus folder and run as below

./Prometheus &

You need to open port 9090 to access prometheus on browser.

In next post we will see how to set up Alert manager and different Alert Rules so that you can receive instant alerts in your mailbox or slack channel.stay tuned.