How to Build a Diskless Cluster

Building a diskless cluster can be tricky. Most guides I found rely on mounting the filesystem over NFS, but I’d rather persist as little of the system as possible between boots. I want to manage the configuration completely via FreeIPA and Puppet. That way, I can easily reset the system to a known state and be confident that no custom configuration is persisted. In my final configuration, I have the Puppet SSL directory and user home directories mounted over NFS, but the rest of the system is prepared dynamically via Puppet.

The configuration below is based on Red Hat 6.5.

Build a minimal install image

This will later be compressed into the initramfs image for the PXE Boot. Customize the package list to fit your needs.

su - root
mkdir ~/diskless_image
cd ~/diskless_image/
yum install --installroot=/root/diskless_image redhat-release-server
cp /etc/yum.repos.d/* ~/diskless_image/etc/yum.repos.d/
rm ~/diskless_image/etc/yum.repos.d/r*  # get rid of the rhel-source repo.

# install base binaries.  If you're not using puppet and IPA, this list can be trimmed.
yum --installroot=/root/diskless_image/ install basesystem filesystem bash kernel passwd dhclient yum openssh-server openssh-clients puppet nfs-utils ipa-client cronie-anacron selinux-policy-targeted vim-minimal

# Set the root password in the image
chroot ~/diskless_image/
chroot #> passwd
chroot #> exit

cd ~/diskless_image/
ln -s ./sbin/init ./init

# Enable networking
echo NETWORKING=yes > etc/sysconfig/network
chmod 644 etc/sysconfig/network

Set up DHCP networking in ~/diskless_image/etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=dhcp
USERCTL=no

Yum Fixes

By default, Yum checks for disk space during the install process. Since we don’t have any true disks, we need to turn off this feature. In ~/diskless_image/etc/yum.conf, add the following in the main section:

diskspacecheck=0

The above setting affects only the install phase of Yum. Unfortunately, Yum also won’t download packages if it thinks there is no disk space. Dirty hack to get Yum working in initramfs:

sed -i -e “s/^([ \t])if (dirstat.f_bavail.$/\1if False:/” ~/diskless_image/usr/lib/python2.6/site-packages/yum/init.py

Puppet & FreeIPA Configuration

Once it’s running, I use Puppet and FreeIPA to automatically configure the diskless server for it’s role in the cluster. If you’re not using these tools, skip this section.

Set up initial puppet configuration.

cp /etc/puppet/puppet.conf ~/diskless_image/etc/puppet/puppet.conf

Set up Puppet SSL directory so that the certificates are persisted across boots.
~/diskless_image/etc/fstab:

nfs:/home/puppet_certs  /var/lib/puppet/ssl     nfs     defaults 0 0

Create Puppet Certs mountpoint:

mkdir ~/diskless_image/var/lib/puppet/ssl

Add to ~/diskless_image/etc/rc.local to install IPA and run puppet on boot. Replace user, domain, and password as needed.

ipa-client-install –force-join –principal setup_user@DOMAIN.LOCAL -w setup_user_password –unattended
puppet agent -t

Create PXE Images

Create binaries in tftp root:

mkdir -p /var/lib/tftpboot/pxelinux/images/diskless
chmod 755 /var/lib/tftpboot/pxelinux/images/diskless

cd ~/diskless_image/ find | cpio -ocv | gzip -9 > /var/lib/tftpboot/pxelinux/images/diskless/diskless.img chmod 644 /var/lib/tftpboot/pxelinux/images/diskless/diskless.img

cp ~/diskless_image/boot/vmlinuz-* /var/lib/tftpboot/pxelinux/images/diskless/vmlinuz chmod 644 /var/lib/tftpboot/pxelinux/images/diskless/vmlinuz

TFTP Configuration:

/etc/xinetd.d/tftp:

service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -v -s /var/lib/tftpboot
        disable                 = no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

Copy PXE Binaries into the TFTP directories:

mkdir /var/lib/tftpboot/pxelinux
chmod 755 /var/lib/tftpboot/pxelinux

cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/pxelinux/pxelinux.0 chmod 644 /var/lib/tftpboot/pxelinux/pxelinux.0

cp /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/pxelinux/vesamenu.c32 chmod 644 /var/lib/tftpboot/pxelinux/vesamenu.c32

PXE Boot Menu. I’ve culled this down to the portions relevant for diskless booting. Add other configurations as needed. /var/lib/tftpboot/pxelinux/pxelinux.cfg/default:

default vesamenu.c32
prompt 0
timeout 150
ONTIMEOUT diskless
MENU MARGIN 10
MENU ROWS 16
MENU TABMSGROW 21
MENU TIMEOUTROW 26
MENU COLOR BORDER 30;44         #20ffffff #00000000 none
MENU COLOR SCROLLBAR 30;44              #20ffffff #00000000 none
MENU COLOR TITLE 0              #ffffffff #00000000 none
MENU COLOR SEL   30;47          #40000000 #20ffffff
MENU BACKGROUND redhat.jpg
MENU TITLE PXE Menu

LABEL local menu label Boot from ^local drive localboot 0xffff

LABEL diskless MENU LABEL Diskless boot RHEL 6 x86_64 KERNEL images/diskless/vmlinuz APPEND initrd=images/diskless/diskless.img ip=dhcp lang=en_US keymap=us

DHCPD Configuration

I statically define IP Addresses in my DHCP configuration. You can also assign them from a pool, but since I’m running headless servers, I find changing IP Addresses to be un-helpful.

/etc/dhcp/dhcpd.conf:

option space PXE;
option PXE.mtftp-ip     code 1 = ip-address;
option PXE.mtftp-cport  code 2 = unsigned integer 16;
option PXE.mtftp-sprot  code 3 = unsigned integer 16;
option PXE.mtftp-tmout  code 4 = unsigned integer 8;
option PXE.mtftp-delay  code 5 = unsigned integer 8;
option arch             code 93 = unsigned integer 16; #RFC4578

option domain-name “DOMAIN.LOCAL”; # REPLACE WITH CORRECT DOMAIN option domain-name-servers 192.168.0.2; # REPLACE WITH CORRECT DNS SERVER default-lease-time 600; max-lease-time 7200;

authoritative;

subnet 192.168.0.0 netmask 255.255.255.0 { # REPLACE WITH CORRECT SUBNET option broadcast-address 192.168.0.255; # REPLACE WITH CORRECT BROADCAST }

group { next-server 192.168.0.10; # REPLACE WITH CORRECT PXE SERVER

if option arch = 00:07 { # Detect if this is booting via EFI or BIOS filename “efi/bootx64.efi”; } else { filename “/pxelinux/pxelinux.0”; }

host diskless-1 {
option host-name “HOSTNAME.DOMAIN.LOCAL”; # REPLACE WITH CORRECT HOSTNAME AND DOMAIN hardware ethernet aa:bb:cc:dd:ee:ff; # REPLACE WITH CORRECT MAC fixed-address 192.168.0.101; # REPLACE WITH CORRECT IP } host diskless-2 {
option host-name “HOSTNAME.DOMAIN.LOCAL”; # REPLACE WITH CORRECT HOSTNAME AND DOMAIN hardware ethernet aa:bb:cc:dd:ee:ff; # REPLACE WITH CORRECT MAC fixed-address 192.168.0.102; # REPLACE WITH CORRECT IP } }

After all of this, you should be able to PXE boot a diskless virtual or physical server.