RPi 4 Cluster - Part 2

Back again for the next installment. In Part 1 we talked about the hardware and some other preliminary things necessary for successful beginnings. Whenever possible, I will not attempt to duplicate the references I mentioned I am using for this.

Now, we will get into the first task for setting up static IPs for all the nodes. As no nodes will show up with an IP at this point (remember, no router), except the default 192.254 or some such, you will have to connect a keyboard and monitor to each individually, then edit the /etc/dhcpcd.conf file. In the “example static” setup area, uncomment the lines:

interface eth0
static ip_address=192.168.10.1/24

then edit the second line for the network address for each node address you wish to use. For this example, I will use an IP of 192.168.1.x for the nodes.

The second thing I want to do is allow the master node to be able to login to each node without a password. This is not really necessary for cluster operation, but is handy for routine maintenance and perform the setup procedure for the various nodes. This is accomplished using public/private key pairs for security.

Perform this first step only on the master node. Here are the steps:

ssh-keygen -t rsa

Do this with no password when asked. Perform the following two steps for each node.

ssh pi@[IP_address] mkdir -p .ssh
cat .ssh/id_rsa.pub | ssh pi@[IP_address] 'cat >> .ssh/authorized_keys'

Test by logging in at each node. No password should be required.

At this point, I returned to the procedure mentioned in Part 1 for setting up the individual Pi nodes and involves setting the hostname and editing the /etc/hostname and /etc/hosts files for the individual nodes. Reboot after this step.

In Part 1: The Basics of his series, Garrett Mills uses “ntp” and “ntpdate” in his procedures. “ntpdate” is deprecated, and I had no success getting the nodes to update, especially after a time when the nodes were powered down. My particular setup has a RTC on the master node only and I want to update all the nodes from that time source, which itself gets updated from the Net. After trying several things, I removed “ntp” and “ntpdate” from the system and installed “chrony.” It can work as both a server and client for the nodes, so seems ideal for my purposes. If things get really out of wack, “chronyc” can be used to manually update the time for the nodes. The configuration file, /etc/chrony/chrony.conf needs only a line added telling each node where to look for the time signal. You can place it anywhere, but I suggest near the top. It can simply look like this:

server 192.168.1.1

Edit the above for your particular address.

For the master nodes configuration, you want the server to be able to listen to itself. That sounds a bit counter-intuitive, but it is simple to do and it works. You can add (or uncomment) the lines to do that. For the master node add the lines to the /etc/chrony/chrony.conf.

#Allow NTP client access from local network (this is only a comment)
allow 192.168.1.0/24

For the client, add the line near the top somewhere to tell the node who to listen for.

server 192.168.1.1 iburst prefer

That’s all it should take to set things up for time updates.

You should reboot everything after this to avoid hiccups.

Next we will get into setting up the shared storage for the cluster. I am using a 128GB micro-SD card formatted as ext4 for that. I refer to the process referenced above in Garrett’s Part 1. And, this is where I ran into a problem the first time I tried his procedure. The process is pretty straight forward, so I think the issue is where the /etc/fstab entry is, the system tries to mount the NFS share before the network is fully up. Using his reference, this is “Step 4: Shared Storage.” The process works okay, but errors refer to “network not available.” Completing the procedure, I am able to use “sudo mount -a” on each node with complete success, so I know the process is fine. My workaround is a small file that sequentially enters each node and performs the “mount -a” command. I am currently doing it manually for now until I complete all the setup. The file is very simple and is:

ssh pi@node2 sudo mount -a
ssh pi@node3 sudo mount -a
ssh pi@node4 sudo mount -a

Node 1, of course is the master and doesn’t need the share mounted. So, what is the real fix for this problem. Pretty simple once I realized the issue. Bringing up “raspi-config” and selecting the boot options, there is an option to boot after the network is available. Select this on all the client nodes. Problem solved, and the above script is not needed.

Another issue is a small error where initial directory creation is with “rwx” permissions for everyone (Step 4.1.3, “chmod -R 777 /clusterfs”) but later redefined as “766” in Step 4.1.5. This has the unfortunate effect of removing execute access to the directory for all but root. This could cause some angst for someone if they can’t figure why access is suddenly gone.

Next, “Configuring the Master Node” where I will not duplicate what steps Garrett indicated. The only issue is related to where, in Step 6.3, copying the configuration files won’t work unless the NFS share is mounted on each node manually. So we will stop for now until I have performed those steps again.

For your reference, Garrett’s Reference I am using as a baseline is listed here: Part 1: The Basics; Part II: Some Simple Jobs; Part III: OpenMPI, Python, and Parallel Jobs.