Author: Winni

Posted on

iSCSItargets on Ubuntu Server 12.04/14.04

At work, I’m currently building a software-based SAN on “commodity hardware” for our forthcoming VMWare cluster. In translation that means that we want to use a regular server instead of an overly expensive, proprietary hardware solution from HP or EMC² or whoever else builds such things.

We’re going to connect the storage server via Gigabit-Ethernet and we are going to use the iSCSI protocol to connect it to the VMWare ESXi hosts.

We first tried the Windows-based StarWind solution, and it worked great. It was easy to install, easy to configure, easy to manage and the HA feature also “just worked”. But somewhere down the road we might want to turn our SAN into a high availability solution with two or three redundant nodes, and StarWind’s price list immediately killed the idea of using their software. At the time of this writing, the StarWind software for a two-node HA cluster with 4TB storage capacity starts at around EUR 4,800. While the software is headache-free and thus absolutely worth it if you have the budget, we decided that it was not for us.

Hence, once again, we began looking for an Open Source-based solution.

I’ve spent the last days testing three options that are available for Ubuntu Linux 12.04/14.04:

#1 SCST

#2 iscsitarget

#3 LIO

SCST takes you back to the early days of Linux: You actually need to patch the Linux kernel sources and then you need to compile a custom Linux kernel in order to get it running. I did this on an older dual core Intel Xeon server with 6 GB RAM and a RAID 5 with 6 15k USCSI hard disks. The procedure took something between 6 and 8 hours on this machine. To add insult to injury, I had to do it twice because something didn’t work the first time. Once SCST was compiled, it would be a lie to say that it was easy to configure and that it installed itself properly on the Ubuntu server. It didn’t. I eventually got it to publish an iSCSItarget, but I had to launch the necessary daemons manually after each reboot. I didn’t spend any additional time to fix this. SCST worked, yes, it felt quite fast, yes, but we decided not to pursue this option any further. A solution that requires custom Linux kernels to run is not really maintainable when you are a user and not an OEM who distributes its own operating system with its own hardware. When you choose SCST on Ubuntu, I think you are actively choosing to NEVER update your server again unless you really want to build your own new kernel every single time when Canonical roll out a new operating system kernel. That certainly was not what we had in mind for our storage server, so SCST was out.

iscsitarget didn’t work on Ubuntu 12.04.4. It didn’t work on a daily build of Ubuntu 14.04 either. An “iscsi_trgt” kernel module was reported missing and the iscsitarget-dkms package did NOT deliver a proper fix for this. On Ubuntu 12.04 it was impossible to build the required kernel module for various reasons that have been published on several blogs on the web. On Ubuntu 14.04, I did not even try to fix this anymore. It shouldn’t be the user’s job to work around known problems in operating systems, and I would have expected from Canonical that their forthcoming flag ship release 14.04 LTS will come with a software repository that actually WORKS. So iscsitarget was also out.

LIO. It’s built into the kernel. Unless Linus Torvalds and his hackers screw things up, no new Linux kernel in Ubuntu should ever have the chance of breaking this solution. To get LIO to work, you only need to install a command line administration tool to create and publish your iSCSI targets. targetcli does this job and it responds to commands like create lun0 /var/mystoragefile 10g, which creates a file-backed LUN. The tool lets you navigate through a hierarchical structure of the available configuration options; you use the cd command to move in the hierarchy, ls shows you what is available at the current position and with commands like create or set you can create or modify settings. That sounds manageable. I followed a short guide that I found on www.linuxclustering.net and two minutes later I had an iSCSI target published and accessible from my VMWare hosts. I cannot yet say if LIO is faster, slower or as fast as SCST. I’m not even sure if it really matters – it’s easy to set up and by no means a maintenance nightmare. “It just works.” And it works out of the box. That is the important aspect.

My next challenge will be to use LIO in an HA environment (DRBD and Heartbeat or Pacemaker/Corosync will be involved in this).

Posted on

Remotely restart the SNMP daemon on Ubuntu 12.04 LTS via a Python skript

It can happen that the SNMP daemon on a server crashes or stops, sometimes without leaving any hints in the system’s logfiles. If you’re running a monitoring server like WhatsUp, you can define a monitor for the SNMP service and should it be reported as down, you can define an action that launches the following Python skript to restart the daemon:

#!/usr/bin/python
import paramiko

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy( paramiko.AutoAddPolicy())
ssh.connect('yourserver.yourdomain.net', username='pythonuser', password='password')

stdin, stdout, stderr = ssh.exec_command( "sudo service snmpd restart")

ssh.close()

In order for this skript to work properly, the user “pythonuser” needs to be able to execute sudo commands without having to type in a password again. The used Python ssh-library, paramiko, does not easily support the use of sudo commands, so it is easier to work around the problem by allowing the user “pythonuser” to run certain commands without having to provide a password again.

In order to allow “pythonuser” to run the “service” command via “sudo” without entering an additional password, this line has to be added to the end of /etc/sudoers on the server on which you need to restart the snmp daemon:

pythonuser ALL=NOPASSWD: /usr/sbin/service

IMPORTANT: This line MUST be at the end of the file, otherwise it will NOT override any settings defined for the “sudo” or other user groups (whose members must provide passwords).