Install iTop on Linux

Problem

  • Install itop on Linux with SELinux

Setting

  • Oracle Linux
  • iTop Version 2.3.4

Solution

Install MySQL

yum install mysql mysql-server

#Set root password
/usr/bin/mysql_secure_installation

mysql – u -p <>

mysql> create database itop;
mysql> create user ‘itop’@’%’ identified by ‘itop-password’;
mysql> grant all privileges on itop.* to ‘itop’@’%’;

mysql> quit;

Install Apache and PHP

yum install httpd php php-mysql php-xml php-soap

systemctl start httpd

Get iTop

wget https://downloads.sourceforge.net/project/itop/itop/2.3.4/iTop-2.3.4-3302.zip?r=https%3A%2F%2Fsourceforge.net%2Fprojects%2Fitop%2Ffiles%2Fitop%2F2.3.4%2F&ts=1502963636&use_mirror=10gbps-io

Files

mv ./web /var/www/itop

cd /var/www/html/itop

mkdir conf
mkdir env-production

chmod 777 conf
chmod 777 data
chmod 777 env-production
chmod 777 log

# If SELinux blocks access
chcon –user system_u –type httpd_sys_content_t -Rv /var/www/html/itop/
# or
sudo setenforce 0

Start install process

Go in your browser and type: http://localhost/itop

Fin

 

Advertisements

Restart failed program every 5 min

Problem

  • Your program fails regularly
  • You need to check if it is running and if not, restart it

Solution

Create a script:

nano restart.sh

Put in the code and replace htop with your program name:

#!/bin/sh

ps cax | grep htop > /dev/null
if [ $? -eq 0 ]; then
echo “Process running.”
else
echo “Process is down.”
fi

Make the script executable:

chmod +x restart.sh

Create a cron job:

crontab -e

Create a new line (replace /home/user/):

*/5 * * * * /home/user/restart.sh

Fin.

 

Icinga2 – Change layout of email notifications

Problem

  • You want to change the layout of the email notifications in Icinga2
  • You want to know, which variables Icinga2 provides

Solution

In Icinga2, emails are sent by a command line script (default is one for host and one for service notifications). In a default installation these scripts can be found under/etc/icinga2/scripts

The scripts are easy structured. At first, a variable template is set with a string which defines the mail body. After this template is “printed” and sent to mail. The -s parameter defines the subject of the email notification. (How to setup mail is described in an another post).

The default template looks like this:

#!/bin/sh
template=`cat <<TEMPLATE
***** Icinga *****

Notification Type: $NOTIFICATIONTYPE

Host: $HOSTALIAS
Address: $HOSTADDRESS
State: $HOSTSTATE

Date/Time: $LONGDATETIME

Additional Info: $HOSTOUTPUT

Comment: [$NOTIFICATIONAUTHORNAME] $NOTIFICATIONCOMMENT
TEMPLATE
`

/usr/bin/printf “%b” “$template” | mail -s “$NOTIFICATIONTYPE – $HOSTDISPLAYNAME is $HOSTSTATE” $USEREMAIL

This means, you can change the mail body and the subject of the email.

Variables

Unfortunately, a listing of the available variables in Icinga2 are difficult to find. I found them in the migration chapter of the Icinga2 docs.

If you need more information in your notification, pick them out of the list. To use a new variable for the template or subject, you have to register them in the environment.

The environment is set in the commands.conf under /etc/icinga2/conf.d/

object NotificationCommand “mail-service-notification” {
import “plugin-notification-command”

command = [ SysconfDir + “/icinga2/scripts/mail-service-notification.sh” ]

env = {
NOTIFICATIONTYPE = “$notification.type$”
SERVICEDESC = “$service.name$”
HOSTALIAS = “$host.display_name$”
HOSTADDRESS = “$address$”
SERVICESTATE = “$service.state$”
LONGDATETIME = “$icinga.long_date_time$”
SERVICEOUTPUT = “$service.output$”
NOTIFICATIONAUTHORNAME = “$notification.author$”
NOTIFICATIONCOMMENT = “$notification.comment$”
HOSTDISPLAYNAME = “$host.display_name$”
SERVICEDISPLAYNAME = “$service.display_name$”
USEREMAIL = “$user.email$”
}
}

In the “env = {” statement, a mapping between the Icinga2 variables and the command variables is build up. If you want to add the needed variable, just add a new line.

Fin

Disable ping as service for a host in Icinga2

Problem

  • You have a server which does not answer ping etc.
  • Icinga2 marks the host as critical and you want to get rid of it.

Solution

By default, Icinga2 applies to every host the service ping4 or ping6 (depending on the address). This is done with an apply rule in /etc/icinga2/conf.d/services.conf:
apply Service "ping4" {
import "generic-service"
check_command = "ping4"
assign where host.address
}

To disable it for two specific host you only have to add an ignore rule (ping4):
ignore where match("x.x.x.x", host.address) || match("y.y.y.y", host.address)

pnp4nagios

Performance Data in Icinga Web 2 (Update)

Problem

  • You have a running Icinga2 plus IcingaWeb2 and want to work with the recorded performance data.
  • You want use PNP4Nagios but the installation is a pain in the ass.

If you follow the instructions you will hopefully getting something like this:
pnp4nagios

Setting

  • Installed Icinga2
  • Configured hosts and services
  • Ubuntu Server 14.04.3

References

Install

~$ sudo apt-get update &amp;&amp; sudo apt-get upgrade
~$ sudo apt-get install --no-install-recommends pnp4nagios rrdcached libapache2-mod-wsgi
~$ sudo icinga2 feature enable perfdata

PNP4Nagios Configuration

This will be the worst part of it.
npcd.cfg
Edit /etc/pnp4nagios/npcd.cfg:

#perfdata_spool_dir = /var/spool/pnp4nagios/npcd/
perfdata_spool_dir = /var/spool/icinga2/perfdata

npcd daemon
Then, the npcd daemon needs to be configured. Edit /etc/default/npcd and change run to yes:

#Run=&quot;no&quot;
Run=&quot;yes&quot;

Restart the service:

~$ sudo service npcd start

rrdcached configuration

~$ sudo mkdir -p /var/cache/rrdcached
~$ sudo chown nagios: /var/cache/rrdcached

Edit /etc/default/rrdcached to add a new line:

OPTS=&quot;-w 1800 -z 1800 -j /var/cache/rrdcached -s nagios -m 0660 -l unix:/var/run/rrdcached.sock&quot;

Put www-data in the nagios group

~$ sudo usermod -G nagios www-data

pnp4nagios/config.php
Edit the file /etc/pnp4nagios/config.php:

#$conf['nagios_base'] = &quot;/cgi-bin/nagios3&quot;;
$conf['nagios_base'] = &quot;/cgi-bin/icinga&quot;;
...
#$conf['RRD_DAEMON_OPTS'] = '';
$conf['RRD_DAEMON_OPTS'] = 'unix:/var/run/rrdcached.sock';

pnp4nagios/process_perfdata.cfg
Edit /etc/pnp4nagios/process_perfdata.cfg and change RRD_DAEMON_OPTS:

RRD_DAEMON_OPTS = unix:/var/run/rrdcached.sock

Apache Configuration

cp /etc/pnp4nagios/apache.conf /etc/apache2/conf-available/pnp4nagios.conf

The default PNP4Nagios requires a htaccess file. Because IcingaWeb2 does not have one i deleted the login/password requirement in the configuration. If you want to use this in a real setup, you have to provide a valid htaccess.
Edit /etc/apache2/conf-available/pnp4nagios.conf and comment these lines:

# AuthType Basic
# AuthUserFile /etc/nagios3/htpasswd.users
# Require valid-user

Load config:

a2enconf pnp4nagios
service apache2 reload

Icinga2 Configuration (Update)

Update: You do not need to follow the instructions in this section. The PNP Module (see next section) will do the work for you.

To enable PNP4Nagios for your services and host you have to provide actions for every entity. To do this you simply add two new templates and import them in the host and the service template.
Edit /etc/icinga2/conf.d/templates.conf:

template Host “pnp-hst” {
action_url = “/pnp4nagios/graph?host=$HOSTNAME$’ class=’tips’ rel=’/pnp4nagios/popup?host=$HOSTNAME$&srv=_HOST_”

template Service “pnp-svc” {
action_url = “/pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$’ class=’tips’ rel=’/pnp4nagios/popup?host=$HOSTNAME$&srv=$SERVICEDESC$”
}

Import the templates in /etc/icinga2/conf.d/templates.conf:

template Host “generic-host” {
….
import “pnp-hst”
}

template Service “generic-service” {
….
import “pnp-svc”
}

IcingaWeb2 Configuration

Download module for PNP4Nagios:

wget https://exchange.icinga.org/icinga/PNP4Nagios/files/1002/icingaweb2-module-pnp4nagios.zip

Unzip it in /usr/share/icingaweb2/modules/
Fixing permissions

~$ sudo usermod -a -G icingacmd www-data
~$ sudo usermod -a -G icingaweb2 www-data

Restart Everything

~$ sudo service rrdcached restart
~$ sudo service npcd restart
~$ sudo service apache2 restart
~$ sudo service icinga2 reload
~$ sudo service icinga2 restart

Enable Module

In the last step you have to enable the pnp4nagios module:
Enable PNP4Nagios in IcingaWeb2

Fin

101 for Email Notifications in Icinga2

(How to change the layout of the notification is explained in an another post.)

Problem

  • You want to enable Email notifications for your hosts and services in Icinga2.
  • Install mail on Ubuntu.
  • You don’t have the time to read the very good Icinga2 documentation (because it’s a classic tl;dr).

Setting

  • Installed Icinga2 (see my HowTo)
  • Configured hosts and service (see my HowTo)
  • Ubuntu Server 14.04.3

Sending mails

In the default configuration Icinga2 uses the mail command:

/usr/bin/printf "%b" "$template" | mail -s "$NOTIFICATIONTYPE - $HOSTDISPLAYNAME - $SERVICEDISPLAYNAME is $SERVICESTATE" $USEREMAIL

For this tutorial i do not change it. If your setup does not allow mail, you have to change it in the /etc/icinga2/scripts/mail-host-notification.sh and /etc/icinga2/scripts/mail-service-notification.sh and stop reading this tutorial.
For a simple setup using mail in Ubuntu you need mailutils and sSMTP. sSMTP just sends mails to a SMTP server you have access to.
Install

sudo apt-get update && sudo apt-get upgrade -y
sudo apt-get install mailutils ssmtp -y

Configure sSMTP
Edit the /etc/ssmtp/ssmtp.conf. Uncomment FromLineOverride=YES and add your mailhub configuration. Here is a sample sSMTP config for a GMail SMTP:

#
# Config file for sSMTP sendmail
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
root=postmaster

# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
AuthUser=@gmail.com
AuthPass=Your-Gmail-Password
mailhub=smtp.gmail.com:587
UseSTARTTLS=YES

# Where will the mail seem to come from?
rewriteDomain=Icinga2.Host

# The full hostname
hostname=Icinga2

# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: address
FromLineOverride=YES

Test your setup

echo "Hello receiver" | mail -s "Test" someone@2342424422.com

If you don’t immediately receive an Email you either have greylisting enabled on your mail server or just have a wrong configuration.

icinga_custom_notification

Configure user (and usergroups)

In the default user configuration of Icinga2 you will find a UserGroup icingaadmins. If you add a user to this group he will get Emails, if the host of Icinga has some problems. (Please note that the user TestUser is not a IcingaWeb2 login user (which can be created in the configuration section)). A simple user looks like this:

object UserGroup "icingaadmins" {
display_name = "Icinga 2 Admin Group"
}

object User "TestUser" {
display_name = "A Name"
groups = [ "icingaadmins" ]
email = "Mail@gmail.com"
period = "24x7"
states = [ OK, Warning, Critical, Unknown ]
types = [ Problem, Recovery ]
vars.additional_notes = "Me"
}

If you need more information on the attributes just have a look at the documentation.

Assign Notification to Service and User

But if you want mails for hosts or services you have to use assign (see documentation). In the following example i assign the command mail-service-notification (defined in “/etc/icinga2/scripts/”) to every service, which runs on the host named “MySQL”. I only inform the newly created TestUser and he should only be informed once a state changes (“interval = 0”)

apply Notification "Alarm-Test" to Service {
command = "mail-service-notification"
users = ["TestUser"]
interval = 0
assign where service.host_name == "MySQL"
}

The assignment for hosts is very similar.

Fin.

Install Icinga2 and Icinga2Web in 10min

For more information how to configure Icinga2 please have a look at my 101.

Problem

  • Install Icinga2 and IcingaWeb2

Setting

  • Ubuntu Server 14.04.3

Basic Installation

Please note: Included in the instructions are password placeholders %BLABLA_PASSWORD% which have to be replaced and are referenced later in the setup.

sudo apt-get update && sudo apt-get upgrade -y
sudo add-apt-repository ppa:formorer/icinga
# Press enter
sudo apt-get update
sudo apt-get install mysql-server apache2 php5 php5-mysql imagemagick php5-imagick php5-gd php5-intl git -y
# Provide (twice) MySQL Password %MYSQLROOT%
sudo apt-get install icinga2 icinga2-ido-mysql -y
# Answer yes
# Answer yes
# Provide (twice) icinga-ido-mysql password %MYSQLIDO%
sudo icinga2 feature enable ido-mysql
sudo icinga2 feature enable command
sudo /etc/init.d/icinga2 restart
sudo usermod -a -G nagios www-data
mkdir icinga2web2_install
cd icinga2web2_install
git clone git://git.icinga.org/icingaweb2.git
sudo mv icingaweb2 /usr/share/icingaweb2
/usr/share/icingaweb2/bin/icingacli setup config webserver apache --document-root /usr/share/icingaweb2/public > icingaweb2.conf
sudo mv icingaweb2.conf /etc/apache2/sites-available/icingaweb2.conf
sudo chown root:root /etc/apache2/sites-available/icingaweb2.conf
sudo a2ensite icingaweb2.conf
sudo a2enmod rewrite
sudo /etc/init.d/apache2 restart
sudo addgroup --system icingaweb2
sudo usermod -a -G icingaweb2 www-data
mysql_secure_installation
# %MYSQLROOT%
# Answer n
# Answer Y
# Answer Y
# Answer Y
# Answer Y

Config Timezone and Database

mysql -u root -p
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, 
EXECUTE ON icinga2idomysql.* TO 'icingaweb'@'localhost'
IDENTIFIED BY '%ICINGA2IDODBPSW%'; 
CREATE DATABASE icingaweb;
GRANT SELECT, INSERT, UPDATE, DELETE, DROP, CREATE VIEW, INDEX, 
EXECUTE ON icingaweb.* TO 'icingaweb'@'localhost' 
IDENTIFIED BY '%ICINGAWEBDBPSW%'; 
FLUSH PRIVILEGES;
quit

Configure timezone in php.ini:

sudo nano /etc/php5/apache2/php.ini

Search for timezone and change ;date.timezone = to data.timezone = “Europe/Berlin” (or yours)

Restart and Token

sudo service mysql restart
sudo service apache2 restart
sudo service icinga2 restart

sudo /usr/share/icingaweb2/bin/icingacli setup config directory --group icingaweb2;
sudo /usr/share/icingaweb2/bin/icingacli setup token create;

Setup in Browser

1. Token
Open http://localhost/icingaweb2/setup in your browser (maybe change to your IP) and paste the token you generated with the last command:
icinga2_browser_1_token
2. Choose modules
icinga2_browser_2_modules
3. Requirements
The requirements should be ok
4. Authentication
We use database here:
icinga2_browser_3_auth
5. Database Resource
Provide %ICINGAWEBDBPSW%:
icinga2_browser_4_database
6. Schema
Provide %MYSQLROOT%:
icinga2_browser_5_schema
7. Authentication Backend
Type icingaweb (not icingaweb2):
icinga2_browser_6_auth_backend
8. Admin User
Set admin user and credentials:
icinga2_browser_7_admin_user
9. Push Next
10. Push Next
11. Push Next
12. Push Next
13. IDO Resource
Provide %ICINGA2IDODBPSW%
icinga2_browser_9_ido_resource
14. Push Next
15. Push Next
16. Finish

Fin

Icinga2 Hosts and Services 101

Config Location

In a default installation the configuration files for hosts and services can be found in /etc/icinga2/conf.d/. For every hostgroup I create a single config file.

Reload and Check

#Check if the configuration is parsable
user@localhost:~$ sudo service icinga2 checkconfig
#Reload the changed config
user@localhost:~$ sudo service icinga2 reload

Hostgroup; Host; Service

The Icinga2 logic of objects:

  • A Service can be monitored and parametrized.
  • A Service is assigned to a Host which has an address etc.
  • A Host can be a member of a HostGroup

Defining a HostGroup

object HostGroup "RaspberryPI" {
  display_name = "Raspberry PI Cluster"
}

​​
Defining a Host and templating
You can define templates in Icinga2 which allows you to set default values for all hosts. In the following example every host will inherit the hostgroup RaspberryPI and the values from the template generic-host:

template Host "RaspberryPI-hosts" {
 import "generic-host"
 groups += [ "RaspberryPI" ]
}
​​

The hosts import the RaspberryPI-hosts template and adds the address using an IP:

object Host "TTRSS" {
  import "RaspberryPI-hosts"
  address = "192.168.3.44"
}

object Host "Mail" {
  import "RaspberryPI-hosts"
  address = "192.168.3.64"
}
​​​​

Service
To define a service you need the host and a check_command (Documentation). Additionally the generic-service template is imported. In the following example an http service is defined.

object Service "Frontpage" {
import "generic-service"
host_name = "TTRSS"
check_command = "http"

vars.sla = "24x7"                             //define sla 
vars.http_uri = "https://192.168.3.44/ttrss"  //check login page
vars.http_ignore_body = 1                     //just getting header.
vars.http_ssl = 1                             //using https
vars.http_warn_time = 1000ms                  //Warning if loading takes more than 1s
vars.http_critical_time = 3000ms              //Critical if loading takes more than 3s
}
​​​​

Advanced HTTP Service Configuration

Escaping special character
If you want to search in the http body for em>H”allo you would have to define something like this:

vars.string = "H"allo"
​​​​

Because this isn’t a valid configuration you have to escape the special character . This is done with \. Thus the correct config is:

vars.string = "H\"allo"
​​​​

Check result size of a http call

vars.http_pagesize = "40000:70000" /between 40kB and 70kB
​​​​

Complex string matching
To search for complex strings which span over more than one line you have set the switch http_linespan:

vars.http_linespan = 1
​​​​

The expected string:

"Status: OK

Message: XtraServer is OK"
​​​​

To search for this string you use a regex:​​

vars.http_expect_body_regex = "Status:.OK.*Message:.XtraServer.is.OK" 
​​​​

In the regex a dot is the equivalent for a single arbitrary character. “is.OK” expresses the need to have one character between “is” and “ok”.
“.*” is the equivalent for a multiple (>=1) arbitrary characters. Because i do not know how many character are between “OK” and “Message” (and is also not important here) “OK.*Message” is the right choise.

Find a specific string somewhere in the body

vars.http_linespan = 1
vars.string = "OK"
​​​​

Allow different possible http results
Example: The body can either contain “{“servicestatus”:3}” or “{“servicestatus”:2}”:
​​

vars.http_linespan = 1
vars.http_expect_body_regex = ".*servicestatus.*[23]"
​​​​

The amount of characters between servicestatus and the status code is not important, but it is expected that a 2 or 3 follows. To express you use the brackets []. If you would expect 6, 7 or 8 you have to write [678].