Setting up a virtual cluster can be a complex, time-consuming process, so the best way to get your footing is to...
start small. In this article, I will show you how to create a virtual cluster by first creating an example test cluster. Ideally, this article will give you a better understanding of virtual clustering on Red Hat.
In order to proceed, you should have two Red Hat Enterprise Linux (RHEL) or CentOS 7 nodes set up with correct forward and reverse domain name system to work with, as there are major changes between Red Hat versions 6 and 7 that drastically reduce cluster management complexity.
There are essentially two technologies that lie underneath the clustering setup: Pacemaker and Corosync. Pacemaker runs the cluster management side, whereas Corosync manages the lower level clustering items to ensure connectivity.
Install them both using the command shown below:
yum install –y pcs fence-agents-all
This command will download all the dependencies and requirements for the clustering setup. It must be repeated on both clusters.
Out of the box you will also need to add a firewall rule to allow required traffic. Doing this is as simple as entering the following commands:
firewall-cmd --permanent --add-service=high-availability
firewall-cmd --permanent --add-service=http
Starting the cluster service
The next step is to enable and start the cluster service. Use the commands below to configure it:
systemctl enable pcsd.service
systemctl start pcsd.service
The first line enables the clustering service at boot up; the second starts it up for the current session.
If you're interested, you can check the logs in /var/logs/cluster/ to see what happens. At this point, we have started the cluster services, but we haven't actually built a cluster setup.
With the latest version of RHEL or CentOS you can directly configure from the command line with the pcs command -- short for "Pacemaker/Corosync configuration system" -- making it much easier than previous installations.
When the cluster setup is installed, it creates a user called hacluster that manages the virtual clustering. In order to be able to use this account, you must change the password, so reset the password to something you know -- on both nodes -- using this command:
Managing the virtual cluster
Once that is done, we can start to manage the virtual cluster and the nodes. The first step is to authorize the nodes that will be in the virtual cluster. The command is as following:
pcs cluster auth node1 node2
If everything goes according plan, your screen should look similar to Figure B.
Creating cluster resources
The next step is to create the cluster resources on top. This can be accomplished using the pcs command along with the cluster setup command and substituting your node names and application names.
pcs cluster setup --start --name myapp node1 node2
At this point, the cluster resource controller will flush the existing configuration, sync new configuration data and build a new configuration incorporating the two nodes specified. All that's left to get the virtual cluster up and running is to enable it:
pcs cluster enable --all
This should show both your nodes as enabled. To check the full cluster status, we can use the command pcs cluster status, which will give an overview of the current cluster status.
Understanding the role of shared storage
Let's take a moment to talk about shared storage. There are many approaches to shared storage -- Network File System, iSCSI, among others- but in our case, we won't be setting up any storage.
One important item to remember when dealing with virtualized cluster nodes is to ensure any file locking system at the hypervisor level is turned off. Not doing so can create many issues, including causing the disk in question to go read-only. Ignore this at your peril. The locking should be done at the OS level. To override this setting, check the documentation for your hypervisor.
In this example, we'll just be setting up the configuration.
One thing we need to do is manage how the servers decide what is "alive" and what is "dead," so to speak, within the virtual cluster. This is done through a process called fencing. Fencing gives each node the power to stop the other one in order to preserve the integrity of the cluster in the event that a node doesn't fail fully and causes issues.
Without a valid fencing configuration, no client facing services will start. The fencing command is pcs stonith. STONITH is an acronym for "Shoot The Other Node In The Head," which is a Linux service capable of shutting down a node that is not working correctly.
There are several fence methods available, but in our basic example we will use the built-in virtual fencing method. From either node in our example, use the following command:
pcs stonith create newfence fence_virt pcmk_host_list=f1
At this point, we can re-enable the cluster components by using the command pcs cluster start --all.
If you use the pcs status command, it should show that the services are now running properly.
Enabling the virtual IP
Once this is complete, we need to enable the virtual IP. The virtual IP is the address that doesn't have a physical connection. Its purpose is to act as a fault tolerant front end for the service that virtual clustering provides. For example, if one node fails, all traffic will be routed to the alternative cluster node without any manual configuration or noticeable downtime.
Use the command below, substituting your IP address as needed:
pcs resource create Cluster_VIP ocf:heartbeat:IPaddr2 ip=192.168.0.100 cidr_netmask=24 op monitor interval=20s
The cluster should now be live and active. You will need to add a cluster resource before you begin working with your cluster.
Setting up virtual clusters in ESXi with shared disks
Tips for resolving common clustering problems
Exploring the relationship between VM clustering and HA