Building a diskless cluster can be tricky. Most guides I found rely on mounting the filesystem over NFS, but I’d rather persist as little of the system as possible between boots. I want to manage the configuration completely via FreeIPA and Puppet. That way, I can easily reset the system to a known state and be confident that no custom configuration is persisted. In my final configuration, I have the Puppet SSL directory and user home directories mounted over NFS, but the rest of the system is prepared dynamically via Puppet.
The configuration below is based on Red Hat 6.5.
Build a minimal install image
This will later be compressed into the initramfs image for the PXE Boot. Customize the package list to fit your needs.
su - root mkdir ~/diskless_image cd ~/diskless_image/ yum install --installroot=/root/diskless_image redhat-release-server cp /etc/yum.repos.d/* ~/diskless_image/etc/yum.repos.d/ rm ~/diskless_image/etc/yum.repos.d/r* # get rid of the rhel-source repo. # install base binaries. If you're not using puppet and IPA, this list can be trimmed. yum --installroot=/root/diskless_image/ install basesystem filesystem bash kernel passwd dhclient yum openssh-server openssh-clients puppet nfs-utils ipa-client cronie-anacron selinux-policy-targeted vim-minimal # Set the root password in the image chroot ~/diskless_image/ chroot #> passwd chroot #> exit cd ~/diskless_image/ ln -s ./sbin/init ./init # Enable networking echo NETWORKING=yes > etc/sysconfig/network chmod 644 etc/sysconfig/network
Set up DHCP networking in ~/diskless_image/etc/sysconfig/network-scripts/ifcfg-eth0:
DEVICE=eth0 TYPE=Ethernet ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=dhcp USERCTL=no
Yum Fixes
By default, Yum checks for disk space during the install process. Since we don’t have any true disks, we need to turn off this feature. In ~/diskless_image/etc/yum.conf, add the following in the main section:
diskspacecheck=0
The above setting affects only the install phase of Yum. Unfortunately, Yum also won’t download packages if it thinks there is no disk space. Dirty hack to get Yum working in initramfs:
sed -i -e "s/^\([ \t]*\)if (dirstat.f_bavail.*$/\1if False:/" ~/diskless_image/usr/lib/python2.6/site-packages/yum/__init__.py
Puppet & FreeIPA Configuration
Once it’s running, I use Puppet and FreeIPA to automatically configure the diskless server for it’s role in the cluster. If you’re not using these tools, skip this section.
Set up initial puppet configuration.
cp /etc/puppet/puppet.conf ~/diskless_image/etc/puppet/puppet.conf
Set up Puppet SSL directory so that the certificates are persisted across boots.
~/diskless_image/etc/fstab:
nfs:/home/puppet_certs /var/lib/puppet/ssl nfs defaults 0 0
Create Puppet Certs mountpoint:
mkdir ~/diskless_image/var/lib/puppet/ssl
Add to ~/diskless_image/etc/rc.local to install IPA and run puppet on boot. Replace user, domain, and password as needed.
ipa-client-install --force-join --principal setup_user@DOMAIN.LOCAL -w setup_user_password --unattended puppet agent -t
Create PXE Images
Create binaries in tftp root:
mkdir -p /var/lib/tftpboot/pxelinux/images/diskless chmod 755 /var/lib/tftpboot/pxelinux/images/diskless cd ~/diskless_image/ find | cpio -ocv | gzip -9 > /var/lib/tftpboot/pxelinux/images/diskless/diskless.img chmod 644 /var/lib/tftpboot/pxelinux/images/diskless/diskless.img cp ~/diskless_image/boot/vmlinuz-* /var/lib/tftpboot/pxelinux/images/diskless/vmlinuz chmod 644 /var/lib/tftpboot/pxelinux/images/diskless/vmlinuz
TFTP Configuration:
/etc/xinetd.d/tftp:
service tftp
{
socket_type = dgram
protocol = udp
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -v -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}
Copy PXE Binaries into the TFTP directories:
mkdir /var/lib/tftpboot/pxelinux chmod 755 /var/lib/tftpboot/pxelinux cp /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot/pxelinux/pxelinux.0 chmod 644 /var/lib/tftpboot/pxelinux/pxelinux.0 cp /usr/share/syslinux/vesamenu.c32 /var/lib/tftpboot/pxelinux/vesamenu.c32 chmod 644 /var/lib/tftpboot/pxelinux/vesamenu.c32
PXE Boot Menu. I’ve culled this down to the portions relevant for diskless booting. Add other configurations as needed.
/var/lib/tftpboot/pxelinux/pxelinux.cfg/default:
default vesamenu.c32 prompt 0 timeout 150 ONTIMEOUT diskless MENU MARGIN 10 MENU ROWS 16 MENU TABMSGROW 21 MENU TIMEOUTROW 26 MENU COLOR BORDER 30;44 #20ffffff #00000000 none MENU COLOR SCROLLBAR 30;44 #20ffffff #00000000 none MENU COLOR TITLE 0 #ffffffff #00000000 none MENU COLOR SEL 30;47 #40000000 #20ffffff MENU BACKGROUND redhat.jpg MENU TITLE PXE Menu LABEL local menu label Boot from ^local drive localboot 0xffff LABEL diskless MENU LABEL Diskless boot RHEL 6 x86_64 KERNEL images/diskless/vmlinuz APPEND initrd=images/diskless/diskless.img ip=dhcp lang=en_US keymap=us
DHCPD Configuration
I statically define IP Addresses in my DHCP configuration. You can also assign them from a pool, but since I’m running headless servers, I find changing IP Addresses to be un-helpful.
/etc/dhcp/dhcpd.conf:
option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sprot code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16; #RFC4578
option domain-name "DOMAIN.LOCAL"; # REPLACE WITH CORRECT DOMAIN
option domain-name-servers 192.168.0.2; # REPLACE WITH CORRECT DNS SERVER
default-lease-time 600;
max-lease-time 7200;
authoritative;
subnet 192.168.0.0 netmask 255.255.255.0 { # REPLACE WITH CORRECT SUBNET
option broadcast-address 192.168.0.255; # REPLACE WITH CORRECT BROADCAST
}
group {
next-server 192.168.0.10; # REPLACE WITH CORRECT PXE SERVER
if option arch = 00:07 { # Detect if this is booting via EFI or BIOS
filename "efi/bootx64.efi";
} else {
filename "/pxelinux/pxelinux.0";
}
host diskless-1 {
option host-name "HOSTNAME.DOMAIN.LOCAL"; # REPLACE WITH CORRECT HOSTNAME AND DOMAIN
hardware ethernet aa:bb:cc:dd:ee:ff; # REPLACE WITH CORRECT MAC
fixed-address 192.168.0.101; # REPLACE WITH CORRECT IP
}
host diskless-2 {
option host-name "HOSTNAME.DOMAIN.LOCAL"; # REPLACE WITH CORRECT HOSTNAME AND DOMAIN
hardware ethernet aa:bb:cc:dd:ee:ff; # REPLACE WITH CORRECT MAC
fixed-address 192.168.0.102; # REPLACE WITH CORRECT IP
}
}
After all of this, you should be able to PXE boot a diskless virtual or physical server.