At work, I’m currently building a software-based SAN on “commodity hardware” for our forthcoming VMWare cluster. In translation that means that we want to use a regular server instead of an overly expensive, proprietary hardware solution from HP or EMC² or whoever else builds such things.
We’re going to connect the storage server via Gigabit-Ethernet and we are going to use the iSCSI protocol to connect it to the VMWare ESXi hosts.
We first tried the Windows-based StarWind solution, and it worked great. It was easy to install, easy to configure, easy to manage and the HA feature also “just worked”. But somewhere down the road we might want to turn our SAN into a high availability solution with two or three redundant nodes, and StarWind’s price list immediately killed the idea of using their software. At the time of this writing, the StarWind software for a two-node HA cluster with 4TB storage capacity starts at around EUR 4,800. While the software is headache-free and thus absolutely worth it if you have the budget, we decided that it was not for us.
Hence, once again, we began looking for an Open Source-based solution.
I’ve spent the last days testing three options that are available for Ubuntu Linux 12.04/14.04:
#1 SCST
#2 iscsitarget
#3 LIO
SCST takes you back to the early days of Linux: You actually need to patch the Linux kernel sources and then you need to compile a custom Linux kernel in order to get it running. I did this on an older dual core Intel Xeon server with 6 GB RAM and a RAID 5 with 6 15k USCSI hard disks. The procedure took something between 6 and 8 hours on this machine. To add insult to injury, I had to do it twice because something didn’t work the first time. Once SCST was compiled, it would be a lie to say that it was easy to configure and that it installed itself properly on the Ubuntu server. It didn’t. I eventually got it to publish an iSCSItarget, but I had to launch the necessary daemons manually after each reboot. I didn’t spend any additional time to fix this. SCST worked, yes, it felt quite fast, yes, but we decided not to pursue this option any further. A solution that requires custom Linux kernels to run is not really maintainable when you are a user and not an OEM who distributes its own operating system with its own hardware. When you choose SCST on Ubuntu, I think you are actively choosing to NEVER update your server again unless you really want to build your own new kernel every single time when Canonical roll out a new operating system kernel. That certainly was not what we had in mind for our storage server, so SCST was out.
iscsitarget didn’t work on Ubuntu 12.04.4. It didn’t work on a daily build of Ubuntu 14.04 either. An “iscsi_trgt” kernel module was reported missing and the iscsitarget-dkms package did NOT deliver a proper fix for this. On Ubuntu 12.04 it was impossible to build the required kernel module for various reasons that have been published on several blogs on the web. On Ubuntu 14.04, I did not even try to fix this anymore. It shouldn’t be the user’s job to work around known problems in operating systems, and I would have expected from Canonical that their forthcoming flag ship release 14.04 LTS will come with a software repository that actually WORKS. So iscsitarget was also out.
LIO. It’s built into the kernel. Unless Linus Torvalds and his hackers screw things up, no new Linux kernel in Ubuntu should ever have the chance of breaking this solution. To get LIO to work, you only need to install a command line administration tool to create and publish your iSCSI targets. targetcli does this job and it responds to commands like create lun0 /var/mystoragefile 10g, which creates a file-backed LUN. The tool lets you navigate through a hierarchical structure of the available configuration options; you use the cd command to move in the hierarchy, ls shows you what is available at the current position and with commands like create or set you can create or modify settings. That sounds manageable. I followed a short guide that I found on www.linuxclustering.net and two minutes later I had an iSCSI target published and accessible from my VMWare hosts. I cannot yet say if LIO is faster, slower or as fast as SCST. I’m not even sure if it really matters – it’s easy to set up and by no means a maintenance nightmare. “It just works.” And it works out of the box. That is the important aspect.
My next challenge will be to use LIO in an HA environment (DRBD and Heartbeat or Pacemaker/Corosync will be involved in this).