Archive for the 'Cacti' Category

Jan 11 2012

Query the Tx SNR of an iDirect modem via SNMP

This is something for the Cacti, Nagios, WhatsUp and what-not users in the satellite business out there…

At work, we use Cacti to graph the SNMP values of A LOT of exotic devices, among them is the very popular family of iDirect modems. Now, we like to graph the Tx (Transmit) SNR (Signal to Noise Ratio) of certain iDirect modems. To do so, you usually open a telnet connection to an iDirect modem, login with the proper user credentials and execute the command

tx snr

Now, as you might know, monitoring and graphing servers like WhatsUp or Cacti cannot perform Telnet connections out of the box. They query SNMP devices through their IP addresses and request the values through their respective OIDs.

How did we solve this?

With a dedicated Linux server that runs an extended SNMP daemon.

Basically, that is a regular SNMP daemon that has some additional configuration lines in its /etc/snmp/snmpd.conf file, like this one:

extend UniqueiDirectModemName '/usr/local/bin/iDirectSNR.py -i 1.2.3.4 -u username -p password'

On this server, you can determine the assigned OID with the snmpwalk command using the -On switch:

snmpwalk localhost -c snmpcommunity -v1 .1.3.6.1.4.1.8072.1.3.2.4.1.2 -On

 

The following Python skript performs the telnet part to query the Tx SNR value; when it encounters an error, it returns the value “100.00000”, which is also returned when the modem is not locked for whatever reason.
Note: Usually the modem returns -100.000000 when it is not locked, but to make the graphs look a little nicer, we did not want to use negative values.

This is the file

/usr/local/bin/iDirectSNR.py

Make sure you perform

sudo chmod a+wxr /usr/local/bin/iDirectSNR.py

to make it executable!

#!/usr/bin/python
import sys
import telnetlib
from optparse import OptionParser

parser = OptionParser()
parser.add_option("-i", "--ipaddress", dest="ipaddress",
                  help="IP address of iDirect modem", metavar="")
parser.add_option("-u", "--username", dest="username",
                  help="IP address of iDirect modem", metavar="")
parser.add_option("-p", "--password", dest="password",
                  help="IP address of iDirect modem", metavar="")

(options, args) = parser.parse_args()

if options.username is None or options.password is None or options.ipaddress is None:
  print "100.000000"
  exit()

try:
  tn = telnetlib.Telnet(options.ipaddress,23,2)
except Exception, e:
  print "100.000000"
  exit()

tn.read_until("Username: ",1)
tn.write(options.username + "\n")
tn.read_until("Password: ",1)
tn.write(options.password + "\n")

tn.read_until("> ",1)

tn.write("rx snr\n")

tn.read_until("Rx SNR: ",1)

result = tn.read_until("\n",1).rstrip("\n")
result = result.lstrip("-")

if result == "":
  result = "100.000000"

print result

tn.write("exit\n")

If you don’t know how to setup a basic SNMP daemon:

sudo apt-get install snmp

Then you need to modify the file /etc/default/snmpd. Remove the reference to localhost in the SNMPDOPTS line, otherwise the server will ONLY listen to SNMP requests its localhost IP address 127.0.0.1. Make sure that the line looks somehow like this:

SNMPDOPTS='-Lsd -Lf /dev/null -u snmp -g snmp -I -smux -p /var/run/snmpd.pid -c /etc/snmp/snmpd.conf'

Now, create a simple /etc/snmp/snmpd.conf file:

rocommunity snmpcommunity
syslocation "MyCompany, MyLocation"
syscontact myname@my.domain
extend UniqueiDirectModemName '/usr/local/bin/iDirectSNR.py -i 1.2.3.4 -u username -p password'

Now restart the SNMP daemon:

sudo /etc/init.d/snmpd restart

And that’s it! Your Linux server will now respond to SNMP queries, and it will even return the Tx SNR value of the modems that you have added to its snmpd.conf file as if they belonged to the server itself. You can add as many devices as you want to your SNMP daemon, and with modified versions of the Python skript, of course, you can also return other values as well.

Comments Off on Query the Tx SNR of an iDirect modem via SNMP

Dec 21 2011

Installing Cacti on Ubuntu Server 10.04 LTS

I wish I could say that instaling Cacti, “the complete rrdtool-based graphing solution”, is a piece of cake, simple and straight forward. Well, it isn’t. Things have gotten better over the last couple of years, but if you have never done it before, it’s still one of those Open Source Software nightmares that the guy behind the Linux Hater’s Blog loves to rant about.

But that does not change the simple fact that Cacti provides a solution for problems that administrators of larger networks actually have, and the team behind Cacti does not charge thousands of Euros for a license for their software. If you need to keep an eye on the performance of thousands of network devices, then Cacti, bitchy, defiant and demanding as it might be, is your friend.

People with a stuffed wallet might rather look at Paessler PRTG Network Monitor and IPSwitch WhatsUp Gold; Paessler is much better in the graphing area than WhatsUp, but WhatsUp probably is a bit easier to use for sending alarms when devices go down.

With certain plugins, Cacti can also be used for monitoring and alarming, and with other plugins it can even be integrated into Nagios, which is -the- Open Source network monitoring solution. However, at my company, we use Cacti exclusively for graphing purposes.

Now what does that mean? It’s rather simple: Cacti uses SNMP (Simple Network Management Protocol) to query values from devices that support the SNMP protocol. This can be the current CPU load of a server or its currently used amount of system memory. This can also be the current amount of traffic that runs through an Ethernet interface of a router. It can be the running state of an Uninterruptable Power Supply or the Eb/N0 (energy per bit to noise power spectral density ratio) of a satellite modem.

Cacti queries such values at a defined interval and puts the results in graphs that are stored in a round robin database (RRD) so that you have nice performance statistics over periods of time. It’s pretty useful in our business, not only for us technicians, but also for our customers who use it as a reference for the quality of our and their own service. (Just for clarification, neither we nor our customers are end-users.)

You will find more information about Cacti and what it does on http://cacti.net. But since you are reading this, I assume that you already know what Cacti generally can do for you but that you, like myself, have a hard time getting it up and running on Ubuntu Linux Server 10.04 LTS.

I will provide some hints and information about this in the following. These are roughly the steps that I took to install Cacti 0.8.7i on a 64-Bit Ubuntu Server. 64.Bit Ubuntu 10.04 LTS is the standard Linux server operating system at my company, so going with something like “CactiEZ”,which is based upon CentOS, was not an option for me.

I will -not- cover the basic usage of Cacti here. To be honest, I am not even remotely qualified to write about that. I install Cacti and make sure that it is running and that my colleagues can use it. But adding data sources and graphs to the server is not what I do on a daily basis. I managed to get the software running on Ubuntu and I also managed to fix or work around some of the bugs, and I also found solutions for some of the things that do not work as advertised or documented by the Cacti team.

Don’t regard the following notes as a checklist that you can follow blindly. You can’t. I also don’t promise that this will work for you. I just pinned down the most important steps that help –ME– installing a new Cacti Server from scratch. If you need more details, you should pay me to write an exhaustive book on the subject. Since none of you is doing this, these free hints will have to do.

First, install a default Ubuntu LAMP (Linux, Apache, MySQL,  PHP) Server 10.04 LTS with OpenSSH.
Once you’re done, login to a shell on the server and perform a basic configuration.

 

sudo apt-get update
sudo apt-get upgrade

 

I use joe as my default text editor:

sudo apt-get install joe

 

Install compiler package ‘build-essential’

sudo apt-get install build-essential

 

Install command line utilities to add software repositories.

sudo apt-get install python-software-properties

 

Set server time zone to UTC:

sudo dpkg-reconfigure tzdata

sudo apt-get install ntp

I configure the ntp server to query the time servers of my company’s backbone routers. If you don’t have own time servers in your network, leave the configuration untouched.

sudo joe /etc/ntp.conf

# You do need to talk to an NTP server or two (or three).
server <aaa.bbb.ccc.ddd>
server <eee.fff.ggg.hhh>

 

If you have a Syslog server running in your environment, you probably want to make sure that it receives the syslog messages of this server.

sudo joe /etc/rsyslog.conf

 

Make sure that this file contains the following:

#  /etc/rsyslog.conf    Configuration file for rsyslogd.
*.*     @<ip.address.of.yoursyslogserver>:<port>

 

Configure the primary network interface by editing the file /etc/network/interfaces

Sample:

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.0.11
        netmask 255.255.255.0
        network 192.168.0.0
        broadcast 192.168.0.255
        gateway 192.168.0.1
        dns-nameservers <ip.address.of.yourdnsserver>
        dns-search <your.domain>
        pre-up iptables-restore < /etc/iptables.rules
        pre-down iptables-save > /etc/iptables.rules

 

Create the file /etc/iptables.rules for a simple iptables-based firewall.

Sample:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [122:8712]
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -s 192.168.0.0/255.255.255.0 -j ACCEPT
-A INPUT -j DROP
COMMIT

 

Enable iptables settings via

sudo iptables-restore < /etc/iptables.rules

 

Make sure that /etc/resolv.conf contains a nameserver reference to the Windows Domain controller:

nameserver <ip.address.of.yourdnsserver>

 

If you want to authenticate against a Microsoft Active Directory server (which we do at our company), you might want to install Centrify Express on your Linux server. Although Centrify Express is NOT Open Source, I found the free edition of their server to be BY FAR the best and most reliable tool to integrate Linux machines into a Microsoft Windows domain structure.

 

Add the repository for Centrify Express:

sudo add-apt-repository "deb http://archive.canonical.com/ lucid partner"
sudo apt-get update

 

Now install Centrify-Express

sudo apt-get install centrifydc

sudo adjoin –w <your.domain>

 

You should have a user group defined in your domain that is allowed to obtain root privileges on the Linux box.

sudo visudo

 

Add the group to the sudoers file:

%<yourWindowsDomainName>\\<WindowsUserGroupName> ALL=(ALL) ALL

 

In the next step, we usually disable the SSH login for non-Active Directory Users:

sudo joe /etc/ssh/sshd_config

 

Make sure that those configuration lines are in there:

PermitRootLogin no
DenyUsers <localUserAccount>

 

Note: All local user accounts that can gain root privileges should not be allowed to use ssh access! Local user accounts are only there for fallback purposes!

sudo apt-get dist-upgrade
sudo apt-get autoremove

 

Now reboot the server

sudo reboot

 

Now we begin with the actual Cacti installation.

 

Login to the server via ssh and execute

sudo -s
cd /var/www
rm index.html
wget http://www.cacti.net/downloads/cacti-0.8.7i-PIA-3.1.tar.gz
tar xvfz cacti-0.8.7i-PIA-3.1.tar.gz
cd cacti-0.8.7i-PIA-3.1/
cp -R * /var/www
cd /var/www
rm -Rf cacti-0.8.7i-PIA-3.1/
rm cacti-0.8.7i-PIA-3.1.tar.gz
apt-get install rrdtool
apt-get install snmp snmpd
apt-get install php5-cli
apt-get install php5-gd
apt-get install php5-curl
apt-get install unzip
mysql –u root -p
mysql> create database cacti;
mysql> grant all privileges on cacti.* to 'cacti'@'localhost' identified by 'cactidbpassword';
mysql> quit

mysql -u cacti -p -D cacti < cacti.sql
cd include
joe config.php

 

Change config.php so it looks like this:

$database_type = "mysql";
$database_default = "cacti";
$database_hostname = "localhost";
$database_username = "cacti";
$database_password = "cactidbpassword";
$database_port = "3306";
$database_ssl = false;
sudo chmod –R a+wr /var/www
sudo /etc/init.d/apache2 restart

 

On your client workstation/notebook, open a web browser and connect to the Linux server. The Cacti installation dialog should appear.

Click Next.
Select “New install” and click Next.
In the next dialog, everything should be marked as FOUND. If not, install the missing pieces and re-try.

Make sure that RRDTool 1.3 and Net-SNMP5 are selected.

Click “Finish” to proceed
Login to Cacti with the username admin and the password admin.
You are required to change the admin password. Do so now.
Go back to the console of the Linux server.

 

Create a new file named cati in /etc/cron.d:

joe /etc/cron.d/cacti

 

Add the following contents to it:

*/1 * * * * root /usr/bin/php -q /var/www/poller.php --force >> /var/log/poller.log 2>&1

 

Note: The Cacti documentation advises to use a five minute interval. However, at our company, that setting did not work for us when we applied a one minute polling interval in Cacti’s system console. Be advised that a one minute polling interval causes a lot of stress even on an eight core Xeon server with 16 GB RAM when you use the PHP poller. Unfortunately, for a couple of reasons, at my work place we have to use both the one minute polling interval AND the PHP poller instead of Spine. If you can avoid the PHP poller, by all means, use Spine and if you can live with bigger polling intervals, reduce the stress of your system and go for it.

 

Change /etc/snmp/snmpd.conf so that it contains the following lines:

com2sec paranoid  default         <yourSNMPcommunitykey>
com2sec readonly  default         <yourSNMPcommunitykey>
com2sec readwrite default         <yourSNMPcommunitykey>

 

In Cacti’s web interface, change the device localhost to use SNMPv1 and your community name <yourSNMPcommunitykey>.
Now reboot the Linux server, log back into Cacti’s web interface and check whether the graphs for the device localhost begin to show actual data.
Login back to the Linux server.
We will now install the Spine poller for Cacti.

 

In your home directory, perform

wget http://www.cacti.net/downloads/spine/cacti-spine-0.8.7i.tar.gz
tar xvfz cacti-spine-0.8.7i.tar.gz
cd cacti-spine-0.8.7i/
sudo apt-get install libmysqlclient16-dev
sudo apt-get install libsnmp-dev
./configure
make
sudo make install
sudo cp /usr/local/spine/etc/spine.conf.dist /usr/local/spine/bin/spine.conf

 

Change /usr/local/spine/bin/spine.conf so that it contains the following lines:

DB_Host         localhost
DB_Database     cacti
DB_User         cacti
DB_Pass         cactidbpassword
DB_Port         3306
DB_PreG         0

 

Now go the web interface of Cacti, login as admin and go to Console/Settings/Paths and change the Spine poller path to /usr/local/spine/bin/spine and save the changes.
In Console/Settings/Poller, select spine and save the changes. You now have to rebuild Cacti’s poller cache to make the poller changes work (you find this option in Cacti’s web interface under Console/System Utilites).

Select Console/User Management and select the admin user.
Check “Plugin Management” and click on Save.
Now we will install several Cacti plugins.

The website http://docs.cacti.net/plugins will be our main source for Cacti plugins.

 

Go back to your ssh session and go to the plugin directory of your Cacti installation.

cd /var/www/plugins
sudo -s
wget http://docs.cacti.net/_media/plugin:aggregate-v0.75.tgz -O aggregate.tgz
tar xvfz aggregate.tgz
rm aggregate.tgz

 

In your web browser, go to the Cacti website, login as admin (when you are not logged in already) and navigate to Console/Plugin Management.

It is possible that the freshly installed aggregate plugin does not appear immediately. Refresh the page a couple of times until your browser has caught up with the updated website.

Under “Actions”, first click on the icon to install the plugin, then click on the next icon to activate the plugin.

The first plugin should now be up and running.

wget http://docs.cacti.net/_media/plugin:boost-v5.1-1.tgz -O boost.tgz
tar xvfz boost.tgz
rm boost.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

sudo mkdir /var/www/imagecache
sudo chmod –R a+wr /var/www/imagecache

In Cacti’s web interface, go to Console/Settings/Boost. Find the “Image Caching” section, enable image caching and set the “location for image files” to /var/www/imagecache.

wget http://docs.cacti.net/_media/plugin:realtime-v0.5-1.tgz -O realtime.tgz
tar xvfz realtime.tgz
rm realtime.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

sudo mkdir /var/www/realtimegraphs
sudo chmod –R a+wr /var/www/realtimegraphs

In Cacti’s web interface, go to Console/Settings/Misc. Find the “Realtime Graphs” section and set the “Cache Directory” to /var/www/realtimegraphs.

Note: Your user account must have permissions to access this plugin.

wget http://docs.cacti.net/_media/plugin:rrdclean-v0.41.tgz -O rrdclean.tgz
tar xvfz rrdclean.tgz
rm rrdclean.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

sudo mkdir /var/www/rra/backup/
sudo mkdir /var/www/rra/archive/
sudo chmod -R a+wr /var/www/rra

wget http://www.nmid-plugins.de/attachments/download/110/nmid_039a_update.tgz
tar xvfz nmid_039a_update.tgz
rm nmid_039a_update.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

wget http://www.nmid-plugins.de/attachments/download/71/nmidWebService.tgz
tar xvfz nmidWebService.tgz
rm nmidWebService.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

wget http://redmine.nmid-plugins.de/attachments/download/314/nmidSmokeping_v1.01.zip
unzip nmidSmokeping_v1.01.zip
rm nmidSmokeping_v1.01.zip

Install and activate the plugin through Cacti’s plugin management interface as described above.

wget http://docs.cacti.net/_media/userplugin:fix64bit-0.3.tar.gz -O fix64bit.tgz
tar xvfz fix64bit.tgz
rm fix64bit.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

Change /var/www/include/global.php so that the variable url_path is defined:

/* set URL path */
if (! isset($url_path)) {
        $url_path = "http://your.cactiserver.url/";
}

Important: the url_path must end with a slash!

cd /var/www
wget http://redmine.nmid-plugins.de/attachments/download/269/nmidWeb2_files_087g.tgz -O nmidWeb2.tgz
tar xvfz nmidWeb2.tgz
rm nmidWeb2.tgz
mysql –u root –D cacti –p < nmidweb2.sql
rm nmidweb2.sql

 

For Spine to work cleanly, we also need to install the “settings” plugin and the “syslog” plugin.

wget http://docs.cacti.net/_media/plugin:settings-v0.71-1.tgz -O settings.tgz
tar xvfz settings.tgz
rm settings.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

wget http://docs.cacti.net/_media/plugin:syslog-v1.22-2.tgz -O syslog.tgz
tar xvfz syslog.tgz
rm syslog.tgz

Install and activate the plugin through Cacti’s plugin management interface as described above.

 

Now we install Smokeping.

sudo mkdir /var/nmid
sudo chmod –R a+wr /var/nmid
sudo apt-get install fping
sudo apt-get install echoping
sudo apt-get install sendmail
sudo apt-get install smokeping
apt-get install libsoap-lite-perl
sudo chmod –R a+wr /usr/share/smokeping
sudo chmod -R a+wr /var/lib/smokeping

 

The URL http://your.cactiserver.url/cgi-bin/smokeping.cgi should now show you the smokeping website.

cd /usr/lib/cgi-bin
wget http://redmine.nmid-plugins.de/attachments/download/219/getSmokePingPath.pl.txt
mv getSmokePingPath.pl.txt getSmokePingPath.pl
chmod a+wxr getSmokePingPath.pl

 

Copy the following to a file named /etc/cron.d/nmid:

*/5 * * * * root /etc/smokeping/nmid

 

Copy the following to a file named /etc/cron.d/smokeping:

*/5 * * * * root /etc/smokeping/smokecron

 

Copy the following to a file named /etc/smokeping/nmid:

cat /etc/smokeping/config_header > /etc/smokeping/config.d/Targets
cat /var/nmid/smokeping_nmid_spserver1_config.txt >> /etc/smokeping/config.d/Targets
/etc/init.d/smokeping reload > /dev/null 2>&1

 

Copy the following to a file named /etc/smokeping/smokecron:

#!/bin/sh
logfile=/var/log/smokeping
# Stellt sicher, dass smokeping immer wieder gestartet wird.
if [ $(ps -A | grep -c smokeping) = 0 ];
then
echo .$(date) Smokeping wiederbeleben. >> $logfile
/etc/init.d/smokeping restart > /dev/null 2>&1
fi

 

To make the nmidSmokeping plugin 1.01 work, we had to add the following code to the file /var/www/plugins/nmidSmokeping/getSmokepingImage.php:

function readConfigOption( $option )
{
    $query = "select value from settings where name='".$option."'";
    $result = mysql_query($query);
    $config_option = mysql_fetch_assoc($result);
    // Free the result set
    mysql_free_result($result);
    return $config_option['value'];
}

If you use brackets and colons in your device or network names, you must change the file /var/www/plugins/nmidSmokeping/setup.php and make sure that “::”, “[“ and “]” are replaced in the respective host names.

Make sure that occurrences like this are added to the source code where the original nmid source code processes similar replacements for umlauts and other special characters:

            $treeName = preg_replace("@\[@","_",$treeName);
            $treeName = preg_replace("@\]@","_",$treeName);
            $treeName = preg_replace("/::/","__",$treeName);

In the same file, you also have to fix a bug in the source code:

In

function plugin_nmidSmokeping_device_action_execute(

make sure that the following lines look like this:

if ( preg_match("/^s1/",$current_nwmgmt_settings) == 0)
          {
                  /* Smokeping not set, yet */
                  $current_nwmgmt_settings = preg_replace("/^s\d/","s1",$current_nwmgmt_settings);
                  db_execute("UPDATE host SET nwmgmt_settings = \"$current_nwmgmt_settings\" WHERE id=".$data["host_id"]);
                  db_execute("UPDATE host SET nwmgmt_smokeping_server=\"".read_config_option("nmid_spserver".$matches[1])."\" WHERE id=".$data["host_id"]);
          }

The important part is the $data[“host_id”] code at the end of the line; the original source code ends with “$host_id”, which will not work.

cd /var/www
wget http://redmine.nmid-plugins.de/attachments/download/269/nmidWeb2_files_087g.tgz
sudo tar xvfz nmidWeb2_files_087g.tgz
tar xvfz nmidWeb2_files_087g.tgz

I know, hardcore Unix security fanatics will hate me for this, but I don’t care. I simply want this thing to work:

sudo chmod -R a+wr /var/www

Now reboot the server. Again, I know, this sounds like typical Windows administration, but I like to be sure that all systems come up properly after a restart or power failure. So just reboot the box and when it is up again, open a web browser on your client and check if your new Cacti server is actually working.

2 responses so far