Tell them, my song.

At work, I’m currently building a software-based SAN on “commodity hardware” for our forthcoming VMWare cluster. In translation that means that we want to use a regular server instead of an overly expensive, proprietary hardware solution from HP or EMC² or whoever else builds such things.

We’re going to connect the storage server via Gigabit-Ethernet and we are going to use the iSCSI protocol to connect it to the VMWare ESXi hosts.

We first tried the Windows-based StarWind solution, and it worked great. It was easy to install, easy to configure, easy to manage and the HA feature also “just worked”. But somewhere down the road we might want to turn our SAN into a high availability solution with two or three redundant nodes, and StarWind’s price list immediately killed the idea of using their software. At the time of this writing, the StarWind software for a two-node HA cluster with 4TB storage capacity starts at around EUR 4,800. While the software is headache-free and thus absolutely worth it if you have the budget, we decided that it was not for us.

Hence, once again, we began looking for an Open Source-based solution.

I’ve spent the last days testing three options that are available for Ubuntu Linux 12.04/14.04:

#1 SCST

#2 iscsitarget

#3 LIO

SCST takes you back to the early days of Linux: You actually need to patch the Linux kernel sources and then you need to compile a custom Linux kernel in order to get it running. I did this on an older dual core Intel Xeon server with 6 GB RAM and a RAID 5 with 6 15k USCSI hard disks. The procedure took something between 6 and 8 hours on this machine. To add insult to injury, I had to do it twice because something didn’t work the first time. Once SCST was compiled, it would be a lie to say that it was easy to configure and that it installed itself properly on the Ubuntu server. It didn’t. I eventually got it to publish an iSCSItarget, but I had to launch the necessary daemons manually after each reboot. I didn’t spend any additional time to fix this. SCST worked, yes, it felt quite fast, yes, but we decided not to pursue this option any further. A solution that requires custom Linux kernels to run is not really maintainable when you are a user and not an OEM who distributes its own operating system with its own hardware. When you choose SCST on Ubuntu, I think you are actively choosing to NEVER update your server again unless you really want to build your own new kernel every single time when Canonical roll out a new operating system kernel. That certainly was not what we had in mind for our storage server, so SCST was out.

iscsitarget didn’t work on Ubuntu 12.04.4. It didn’t work on a daily build of Ubuntu 14.04 either. An “iscsi_trgt” kernel module was reported missing and the iscsitarget-dkms package did NOT deliver a proper fix for this. On Ubuntu 12.04 it was impossible to build the required kernel module for various reasons that have been published on several blogs on the web. On Ubuntu 14.04, I did not even try to fix this anymore. It shouldn’t be the user’s job to work around known problems in operating systems, and I would have expected from Canonical that their forthcoming flag ship release 14.04 LTS will come with a software repository that actually WORKS. So iscsitarget was also out.

LIO. It’s built into the kernel. Unless Linus Torvalds and his hackers screw things up, no new Linux kernel in Ubuntu should ever have the chance of breaking this solution. To get LIO to work, you only need to install a command line administration tool to create and publish your iSCSI targets. targetcli does this job and it responds to commands like create lun0 /var/mystoragefile 10g, which creates a file-backed LUN. The tool lets you navigate through a hierarchical structure of the available configuration options; you use the cd command to move in the hierarchy, ls shows you what is available at the current position and with commands like create or set you can create or modify settings. That sounds manageable. I followed a short guide that I found on www.linuxclustering.net and two minutes later I had an iSCSI target published and accessible from my VMWare hosts. I cannot yet say if LIO is faster, slower or as fast as SCST. I’m not even sure if it really matters – it’s easy to set up and by no means a maintenance nightmare. “It just works.” And it works out of the box. That is the important aspect.

My next challenge will be to use LIO in an HA environment (DRBD and Heartbeat or Pacemaker/Corosync will be involved in this).

Over the last few years, Canonical has earned the reputation of “trying to be another Apple”, in both the good and the bad meaning. Good, because it shows that Canonical has a vision and follows it. Bad, because Ubuntu is not the community-driven product that so many believed that it was or maybe should have been.
In any case, I’m certain that Valve had very good reasons not to go with a Linux distribution that is “owned” by another company – they are already in such a situation with the OS X and Windows versions of their products. They probably wanted to have full control over what they are doing and no longer wanted to rely on the choices of another party and possible competitor – after all, Steam and the Ubuntu Store -are- competing solutions. Even Valve’s “Big Picture” and Canonical’s “Ubuntu Touch” or “Unity” are competing solutions that have no place on the same system; one environment tries to lock the user into Valve’s ecosystem, the other environment tries to lock him or her into Canonical’s ecosystem.
Debian is not “owned” by a company and does not contain any proprietary extensions that might conflict with Valve’s interests or that have the potential to put them into a problematic legal situation. For Valve, Debian is a good choice. FreeBSD would have been an even better choice for licensing reasons – which is why Apple chose it as the foundation for Darwin – but Debian is close enough and Linux has a generally better hardware support than FreeBSD. Debian is also close to Ubuntu, so most of Valve’s development efforts were not in vain and can easily be re-used.
For the user, none of this actually matters. Most SteamOS users will never notice the difference, simply because they will never leave Steam’s “Big Picture” mode and fall back to the desktop environment.
I’m not sure if Ubuntu ever had a real chance of winning this: Valve ported Steam to Ubuntu not because Ubuntu is such a great environment, but because Ubuntu is the leading Linux distribution for the desktop with the largest user base — a gaming company has to go where the users are.
But does it make sense to base your own system software on a desktop distribution when you want to build a system that is not a desktop but a console? Do you really want to bundle your own store and ecosystem with the store and ecosystem of somebody else? No.
Just like they would not use Windows RT, OS X, iOS or Android with bundled Google Apps and Google Play Store as the foundation of their own operating system with an integrated distribution channel, they certainly did not want to have the Ubuntu Store and Ubuntu One on their SteamOS.
At a first glimpse, Valve’s decision might seem surprising and disappointing for us Ubuntu users. But when you give it some thought, it makes perfect sense. After all, Valve is still a company with commercial interests.