Main Page
From FC4GFS
| Table of contents |
Introduction
I started working with GFS before FC4 came out and, well, let's just say I got it working but it took a while. When FC4 came out and was said to include GFS, I was overjoyed. I expected to have some GUI and it would "just work". Unfortunatly that was not to be.
Setting up GFS turns out to be fairly simple on FC4 if you know what to do. That's where this guide comes in. If you follow the following steps you will have GFS installed, you can then set it up the details based on other guides.
Install FC 4
Set up Fedora as you normally would, ignoring the space you intend to use as shared storage. I suggest using some form of raid redundancy (raid 1 or 5) for your system disks. I always use the minimal install and then install things using yum as needed.
Note: if you ever tried to install GFS from source, you will need to format/reinstall rather than upgrade fedora. The rpms put files in different locations, which causes things not to work in unexpected ways.
Install required rpms from yum
- run yum update
- reboot to use new kernel
- install the rpms from yum as follows (if you are using some kernel other than smp or uni (eg. xen) you'll need to install modules for the right type of kernel)
yum install GFS-kernel-smp GFS-kernheaders GFS-kernel gnbd-kernel-smp gnbd-kernel GFS lvm2-cluster cman-kernel cman-kernheaders cman-kernel-smp cman ccs ccs-devel fence dlm-kernel dlm-kernheaders dlm-kernel-smp dlm dlm-devel magma-plugins
Configure GFS
Almost all of the configuration for GFS is contained in /etc/cluster/cluster.conf
Here is a very basic config for a two node cluster. It really should use better methods to fence (block out disconnected nodes.) But this seems to work ok to get things up and running.
<?xml version="1.0"?>
<cluster name="clust" config_version="1">
<cman two_node="1" expected_votes="1">
</cman>
<clusternodes>
<clusternode name="cap.bar.com" votes="1">
<fence>
<method name="single">
<device name="human" ipaddr="10.1.1.86"/>
</method>
</fence>
</clusternode>
<clusternode name="flux.bar.com" votes="1">
<fence>
<method name="single">
<device name="human" ipaddr="10.1.1.47"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fence_devices>
<fence_device name="human" agent="fence_manual"/>
</fence_devices>
</cluster>
Open up your firewall
The cluster nodes must be able to talk to each other on ports which are blocked by default.
You need to open up the following ports for the other computer.
udp/6809 udp/5510 tcp/636 tcp/21064
Turn on the required services
Turn on the required services with the following commands:
chkconfig ccsd on chkconfig gfs on chkconfig cman on chkconfig fenced on chkconfig clvmd on
Once you reboot both nodes, /proc/cluster/nodes should list the nodes. /proc/cluster/status has other useful information.
Create logical volumes in CLVM
(from usage.txt) Creating CLVM logical volumes
Use standard LVM commands (see LVM documentation on using pvcreate, vgcreate, lvcreate.) A node must be running the CLVM system to use the LVM commands. Running the CLVM system means successfully running the commands above up through starting clvmd.
Creating file systems
The syntax for creating file systems is: gfs_mkfs -p lock_dlm -t <ClusterName>:<FSName> -j <Journals> <Device>
So, for example, you might do: gfs_mkfs -p lock_dlm -t clust:bigsharddisk -j 8 /dev/volGroup/bigvolume
You need one journal (the <Journals> argument) for each node. Each journal takes up 128M.
Mounting the file systems
You can mount the filesystem by running
mount -t gfs /dev/volGroup/bigvolume /mnt
just as any other filesystem.
Conclusion
GFS is fairly complicated, and you'll probably want to read some other documentation to understand what's going on, so that when things don't work you'll have some idea of where to look.
If you have any questions or comments email me at gfs@ethanet.com
