{"id":1402,"date":"2014-03-20T15:26:49","date_gmt":"2014-03-20T14:26:49","guid":{"rendered":"http:\/\/www.wmaus.net\/?p=1402"},"modified":"2020-11-29T18:13:09","modified_gmt":"2020-11-29T17:13:09","slug":"setting-up-a-high-availability-storage-cluster-for-vmware-with-ubuntu-14-04-lts","status":"publish","type":"post","link":"https:\/\/www.wmaus.net\/?p=1402","title":{"rendered":"Setting up a high availability storage cluster for VMWare with Ubuntu 14.04 LTS"},"content":{"rendered":"<p style=\"padding-left: 30px; text-align: left;\"><em>Double, double toil and trouble,<\/em><br \/>\n<em>Fire burn, and cauldron bubble.<\/em><br \/>\n&#8211; Macbeth<\/p>\n<p>As I&#8217;ve mentioned before, I&#8217;m currently working on a virtualization project. VMWare vSphere Essentials Plus is the virtualization software of choice, and the goal is to build an affordable redundant\/clustered storage solution for the VMWare cluster.<\/p>\n<p>After a lot of experimenting with Ubuntu 12.04 and Ubuntu 14.04, DRBD, Heartbeat, LIO and NFS, I finally managed to configure a WORKING high availability storage cluster based upon a daily development build of the forthcoming Ubuntu Server 14.04 LTS x64, DRBD, Heartbeat and NFS.<\/p>\n<p>Why NFS and not a LIO-based iSCSI solution? Because it was much easier to set up and, when you believe certain benchmarks on the web, an NFS-based solution is even a bit faster and more responsive than an iSCSI-based one. At this point, I have not run any benchmarks myself and all I can say is that they <em>feel<\/em> almost the same.<\/p>\n<p>The following hardware is involved in this test environment:<\/p>\n<ul>\n<li>A Cisco Gigabit switch with &#8220;system mtu jumbo 9000&#8221; configured, so that we can use jumbo packets to improve the data transfer between the storage units and the VMWare hosts. We just have to make sure to use a good data management service (<a href=\"https:\/\/www.couchbase.com\/products\/cloud\">read more<\/a>)<\/li>\n<li>Two non-identical x64 servers with at least three Gigabit Ethernet ports.<\/li>\n<li>Two identical x64 servers for the VMWare cluster, each with 24 GB of RAM and 6 Gigabit Ethernet ports.<\/li>\n<\/ul>\n<p>Like I&#8217;ve said, this is a TEST environment. When you can AFFORD (financially, I mean) to be really serious about this, then you would of course have redundant 10 Gig Ethernet switches and 10 Gig Ethernet NICs in place. But in the real world where I live, there are always financial constraints and I have to make do with what I have.<\/p>\n<p>I have three networks in place, each assigned to one NIC on the storage servers:<\/p>\n<ul>\n<li>eth0: 192.168.0.0\/24 &#8212; This is the &#8220;management&#8221; network<\/li>\n<li>eth1: 10.99.99.0\/24 &#8212; This is the actual storage\/data transfer network<\/li>\n<li>eth2: 172.16.99.0\/24 &#8212; This is the synchronization network for the storage servers and the DRBD daemon running on them<\/li>\n<\/ul>\n<p>The first storage server has the hostname <em>storage01<\/em> and uses the IP addresses 192.168.0.34, 10.99.99.31 and 172.16.99.1.<\/p>\n<p>The second storage server has the hostname <em>storage02<\/em> and uses the IP addresses 192.168.0.35, 10.99.99.32 and 172.16.99.2.<\/p>\n<p>I won&#8217;t cover the actual setup on the VMWare hosts in this little post. I have one VMWare machine using the IP address 10.99.99.11 and the other one uses 10.99.99.12 to communicate with the storage servers. Both VMWare hosts communicate with the virtual\/floating IP address of the storage machines that is generated and assigned by the Heartbeat daemon running on the storage servers. That floating IP address is 10.99.99.30.<\/p>\n<p>Both storage servers are using a default Ubuntu Server 14.04 LTS x64 installation with only the OpenSSH service installed on them. I used a &#8220;guided use entire disk&#8221; partition layout on both machines, in case if you wonder. For testing purposes, the only thing that matters is that both servers have sufficient free disk space available to mirror the data between them.<\/p>\n<p>Once Ubuntu is up and running on both machines, open a terminal on both of them and run the following commands (unless explicitly stated otherwise, you will always execute the SAME commands on BOTH machines):<\/p>\n<p>Being the superuser will make life much easier, so enter a superuser shell on both storage servers:<\/p>\n<pre># sudo -s<\/pre>\n<p>I grew up with Wordstar on CP\/M and MS-DOS, so <em>joe<\/em> with its Wordstar-compatible commands is my preferred console editor. I install it now along with a few other missing packages:<br \/>\n# apt-get install joe traceroute python-software-properties build-essential ntp<\/p>\n<p>Configure NTP to use the network time servers of your own network. Yes, you should have two of those running in your network. If not, let&#8217;s say I didn&#8217;t hear that and you are now quietly installing NTP servers before you go on with this setup.<\/p>\n<pre># joe \/etc\/ntp.conf\n\nserver 91.151.144.1\nserver 91.151.144.2<\/pre>\n<p>Now we configure static IP addresses on each node respectively &#8211; and make sure that the data and sync NICs use Jumbo frames:<\/p>\n<p>On storage01:<\/p>\n<pre>#  joe \/etc\/network\/interfaces\n\n# The primary network interface\nauto eth0\niface eth0 inet static\naddress 192.168.0.34\nnetmask 255.255.255.0\nnetwork 192.168.0.0\nbroadcast 192.168.0.255\ngateway 192.168.0.1\n# dns-* options are implemented by the resolvconf package, if installed\ndns-nameservers 192.168.0.41\ndns-search ce-tel.net\nauto eth1\niface eth1 inet static\naddress 10.99.99.31\nnetmask 255.255.255.0\nnetwork 10.99.99.0\nbroadcast 10.99.99.255\nmtu 9000\nauto eth2\niface eth2 inet static\naddress 172.16.99.1\nnetmask 255.255.255.0\nnetwork 172.16.99.0\nbroadcast 172.16.99.255\nmtu 9000<\/pre>\n<p>On storage02:<\/p>\n<pre>#  joe \/etc\/network\/interfaces\n\n# The primary network interface\nauto eth0\niface eth0 inet static\naddress 192.168.0.35\nnetmask 255.255.255.0\nnetwork 192.168.0.0\nbroadcast 192.168.0.255\ngateway 192.168.0.1\n# dns-* options are implemented by the resolvconf package, if installed\ndns-nameservers 192.168.0.41\ndns-search ce-tel.net\nauto eth1\niface eth1 inet static\naddress 10.99.99.32\nnetmask 255.255.255.0\nnetwork 10.99.99.0\nbroadcast 10.99.99.255\nmtu 9000\nauto eth2\niface eth2 inet static\naddress 172.16.99.2\nnetmask 255.255.255.0\nnetwork 172.16.99.0\nbroadcast 172.16.99.255\nmtu 9000<\/pre>\n<p>On both storage servers, the following modifications need to be done to \/etc\/hosts:<\/p>\n<pre># joe \/etc\/hosts\n\n192.168.0.34    storage01.ce-tel.net\n\n192.168.0.35  storage02.ce-tel.net\n\n10.99.99.31     storage01\n10.99.99.32     storage02\n172.16.99.1     storage01-sync\n172.16.99.2     storage02-sync<\/pre>\n<p>We won&#8217;t be using physical disk drives for DRBD in our setup. Instead, we will be using loopback devices that point to disk image files. In this scenario, I will be using a 300 GB image for our NFS storage. Modify that to your own needs and possibilities:<\/p>\n<pre># mkdir -p \/var\/mystorage\/img \/var\/mystorage\/mnt\n# dd if=\/dev\/zero of=\/var\/mystorage\/img\/meta.img bs=1024 count=250000\n# dd if=\/dev\/zero of=\/var\/mystorage\/img\/data.img bs=1 seek=300G count=0<\/pre>\n<p>This script (found somewhere on the web) will be used to bind the image files to the loopback devices after each reboot of the servers:<\/p>\n<pre># joe \/etc\/init.d\/drbdloopbacks\n\n#!\/bin\/bash\n#\n#Startup script to create LOFSs for drbd on ubuntu \/ vps.net\n#\n#Author: Sid Sidberry &lt;greg@halfgray.com&gt; http:\/\/himynameissid.com\n#\n#Description: This script attaches files from the file system to loopback\n#       devices for use as drbd partitions.\n#       Two files are required, 1 for drbd meta-data and 1 for drbd data\n#\n#your partion files\nDRBD_METADATA_SRC=\"\/var\/mystorage\/img\/meta.img\"\nDRBD_FILEDATA_SRC=\"\/var\/mystorage\/img\/data.img\"\n#loopback devices\nDRBD_METADATA_DEVICE=\"\/dev\/loop6\"\nDRBD_FILEDATA_DEVICE=\"\/dev\/loop7\"\n#losetup\nLOSETUP_CMD=\/sbin\/losetup\n#make sure the src files exist\n[ -x $LOSETUP_CMD ] || exit 0\n[ -e \"$DRBD_METADATA_SRC\" ] || exit 0;\n[ -e \"$DRBD_FILEDATA_SRC\" ] || exit 0;\n#includes lsb functions\n. \/lib\/lsb\/init-functions\nfunction connect_lofs\n{\nlog_daemon_msg \"Connecting loop devices $DRBD_METADATA_DEVICE, $DRBD_FILEDATA_DEVICE\"\n$LOSETUP_CMD $DRBD_METADATA_DEVICE $DRBD_METADATA_SRC\n$LOSETUP_CMD $DRBD_FILEDATA_DEVICE $DRBD_FILEDATA_SRC\n}\nfunction release_lofs\n{\nlog_daemon_msg \"Releasing loop devices $DRBD_METADATA_DEVICE, $DRBD_FILEDATA_DEVICE\"\n$LOSETUP_CMD -d $DRBD_METADATA_DEVICE\n$LOSETUP_CMD -d $DRBD_FILEDATA_DEVICE\n}\ncase \"$1\" in\nstart)\nconnect_lofs\n;;\nrelease)\nrelease_lofs\n;;\nstop)\nrelease_lofs\n;;\n*)\necho \"Usage: \/etc\/init.d\/drbdloopbacks {start|release}\"\nexit 1\n;;\nesac\nexit 0<\/pre>\n<p>Now we&#8217;re going to make that script executable and configure it to be automatically launched on each system startup:<\/p>\n<pre># chmod +x \/etc\/init.d\/drbdloopbacks\n# update-rc.d drbdloopbacks defaults 15 15<\/pre>\n<p>Now that we have a foundation for DRBD, let&#8217;s install and configure DRBD:<\/p>\n<pre># apt-get install drbd8-utils<\/pre>\n<p>Make sure that the DRBD kernel module is loaded when the machine boots:<\/p>\n<pre># echo 'drbd' &gt;&gt; \/etc\/modules\n<\/pre>\n<p>We&#8217;ll load the module now in our currently running servers:<\/p>\n<pre># modprobe drbd<\/pre>\n<p>Now create a configuration file for DRBD:<\/p>\n<pre># joe \/etc\/drbd.conf\n\nglobal {\nusage-count no;\n}\n\ncommon {\nprotocol C;\nsyncer {\nrate 100M;\n}\n       startup {\n                wfc-timeout  10;\n                degr-wfc-timeout 8;\n                outdated-wfc-timeout 5;\n        }\n\n}\n\nresource mystorageres {\ndevice   \/dev\/drbd0;\ndisk     \/dev\/loop7;\nmeta-disk \/dev\/loop6[0];\non storage01 {\naddress  172.16.99.1:7789;\n}\non storage02 {\naddress  172.16.99.2:7789;\n}\nnet {\nafter-sb-0pri   discard-younger-primary;\nafter-sb-1pri   consensus;\nafter-sb-2pri   disconnect;\n}\n}<\/pre>\n<p>In the following step, we will create the meta-data for our resource and then bring the resource &#8220;live&#8221;:<\/p>\n<pre># drbdadm create-md mystorageres\n# drbdadm up all<\/pre>\n<p>This commands is to be run on storage01 ONLY. It will promote storage01 to be the primary node and it will push its data to the secondary node by overwriting all the data on storage02 (which is currently an empty shell anyway):<\/p>\n<pre># drbdadm -- --overwrite-data-of-peer primary mystorageres<\/pre>\n<p>Verify that it&#8217;s syncing:<\/p>\n<pre>#  cat \/proc\/drbd<\/pre>\n<p>This should yield something like this:<\/p>\n<pre>version: 8.4.3 (api:1\/proto:86-101)\nsrcversion: F97798065516C94BE0F27DC\n0: cs:SyncSource ro:Primary\/Secondary ds:UpToDate\/Inconsistent C r-----\nns:107516 nr:0 dw:0 dr:108244 al:0 bm:6 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:314465284\n[&gt;....................] sync'ed:  0.1% (307092\/307200)Mfinish: 4:52:15 speed: 17,916 (17,916) K\/sec<\/pre>\n<p>I recommend waiting for the initial synchronization to complete.<\/p>\n<p>Once the synchronization process has finished, we will format the drbd0 device and mount it:<\/p>\n<pre># mkfs.ext4 \/dev\/drbd0\n# mount \/dev\/drbd0 \/var\/mystorage\/mnt<\/pre>\n<p>It is time to configure NFS.<\/p>\n<p>On both nodes:<\/p>\n<pre># apt-get install nfs-kernel-server<\/pre>\n<p>Since Heartbeat will be controlling NFS, we need to remove the launcher scripts from our startup configuration.<\/p>\n<pre># update-rc.d -f nfs-kernel-server remove\n# update-rc.d nfs-kernel-server stop 20 0 1 2 3 4 5 6 .<\/pre>\n<p>We will move the NFS configuration information to the redundant DRBD device. This prepares storage01 for this:<\/p>\n<pre># mount \/dev\/drbd0 \/var\/mystorage\/mnt\n# mv \/var\/lib\/nfs\/ \/var\/mystorage\/mnt\n# ln -s \/var\/mystorage\/mnt\/nfs\/ \/var\/lib\/nfs\n# mv \/etc\/exports \/var\/mystorage\/mnt\n# ln -s \/var\/mystorage\/mnt\/exports \/etc\/exports\n\n# mkdir \/var\/mystorage\/mnt\/export\n# joe \/etc\/exports\n           \/var\/mystorage\/mnt\/export 10.99.99.0\/24(rw,async,no_root_squash,no_subtree_check,fsid=1)<\/pre>\n<p>On storage02, we only need to prepare this:<\/p>\n<pre># rm -rf \/var\/lib\/nfs\n# ln -s \/var\/mystorage\/mnt\/nfs\/ \/var\/lib\/nfs\n# rm \/etc\/exports\n# ln -s \/var\/mystorage\/mnt\/exports \/etc\/exports<\/pre>\n<p>Now we will configure Heartbeat on both servers:<\/p>\n<pre># apt-get install heartbeat<\/pre>\n<p>Heartbeat needs three configuration files to work properly, of which identical copies need to be placed on both servers.<\/p>\n<p>We will begin with ha.cf:<\/p>\n<pre># joe \/etc\/ha.d\/ha.cf\n\nuse_logd yes\nautojoin none\nbcast eth1\nwarntime 5\ndeadtime 10\ninitdead 30\nkeepalive 2\nlogfacility local0\nnode storage01\nnode storage02\nauto_failback on<\/pre>\n<p>Then we will configure the authkeys file:<\/p>\n<pre># joe \/etc\/ha.d\/authkeys\n\nauth 1\n1 sha1 thisisourlittlesecret<\/pre>\n<p>Important: authkeys requires special permission settings:<\/p>\n<pre># chmod 600 \/etc\/ha.d\/authkeys<\/pre>\n<p>Finally, we will configure the Heartbeat resources in haresources:<\/p>\n<pre># joe \/etc\/ha.d\/haresources\n\n      storage01 IPaddr::10.99.99.30\/24 drbddisk::mystorageres Filesystem::\/dev\/drbd0::\/var\/mystorage\/mnt::ext4 nfs-kernel-server<\/pre>\n<p>In my test environment, after the NFS changes were made, the loopback devices were no longer created after a system restart. So I recommend to apply these settings again:<\/p>\n<pre># update-rc.d drbdloopbacks defaults<\/pre>\n<p>This will start Heartbeat:<\/p>\n<pre># \/etc\/init.d\/heartbeat start<\/pre>\n<p>But, maybe, you rather want to reboot the machines:<\/p>\n<pre># reboot<\/pre>\n<p>If everything works well, you can now configure the VMWare ESXi hosts to mount the NFS folder \/var\/mystorage\/mnt\/exports on server 10.99.99.30.<\/p>\n<p>Good luck!<\/p>\n<p><strong>UPDATE, April 24, 2014:<\/strong><\/p>\n<p>I had to make the experience that using protocol C in DRBD only makes sense when both servers have equally powerful hardware. In my environment, one machine used 15k SAS disks and the other one had 7.2k SATA disks. The result was that the server with the SATA disks brutally thwarted the SAS-machine. In such cases, it might make more sense to use protocol A or B, because they don&#8217;t wait for the actual disk write on the target to be finished. I also could observe that DRBD is a very CPU intensive process; even the new 8-core Xeon server was running at load averages between 4.5 and 6.0, the 8-core target server with SATA disks went beyond 8.0.<\/p>\n<p>This resulted in a very simple decision: We no longer use DRBD. High availability is nice to have, but let&#8217;s face it, in most cases it&#8217;s not a show stopper if you don&#8217;t have it. After all, we don&#8217;t run life support systems here. So we now use both servers as regular standalone storage servers and use VMWare Replication to replicate the VMs every 24 hours from the main server to the backup server. That has nothing to do with high availability, it does not even replace a backup, but at least it makes a disaster recovery much simpler if that should ever be required.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Double, double toil and trouble, Fire burn, and cauldron bubble. &#8211; Macbeth As I&#8217;ve mentioned before, I&#8217;m currently working on a virtualization project. VMWare vSphere Essentials Plus is the virtualization software of choice, and the goal is to build an affordable redundant\/clustered storage solution for the VMWare cluster. After a lot of experimenting with Ubuntu &hellip; <\/p>\n<p class=\"link-more\"><a href=\"https:\/\/www.wmaus.net\/?p=1402\" class=\"more-link\">Continue reading<span class=\"screen-reader-text\"> &#8220;Setting up a high availability storage cluster for VMWare with Ubuntu 14.04 LTS&#8221;<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_links_to":"","_links_to_target":""},"categories":[49],"tags":[],"class_list":["post-1402","post","type-post","status-publish","format-standard","hentry","category-ubuntu-linux"],"_links":{"self":[{"href":"https:\/\/www.wmaus.net\/index.php?rest_route=\/wp\/v2\/posts\/1402","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wmaus.net\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wmaus.net\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wmaus.net\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wmaus.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1402"}],"version-history":[{"count":16,"href":"https:\/\/www.wmaus.net\/index.php?rest_route=\/wp\/v2\/posts\/1402\/revisions"}],"predecessor-version":[{"id":1810,"href":"https:\/\/www.wmaus.net\/index.php?rest_route=\/wp\/v2\/posts\/1402\/revisions\/1810"}],"wp:attachment":[{"href":"https:\/\/www.wmaus.net\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1402"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wmaus.net\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1402"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wmaus.net\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1402"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}