Centos5: OCFS2 cluster FS on dual primary DRBD: part 3 – setup OCFS2 cluster filesystem

Now for the final part – configuring OCFS2 on our dual primary DRBD.

Requirements: internet connection
Time: about 30 min – depends on disk size.

1. Visit this page at Oracle web site. There is detailed OCFS2 documentation available for download in PDF format – a lot of details that are out of scope of this tutorial, but might come in handy if you have some specific requirements for your cluster storage. Worth reading before formatting the partition. Follow download link to the kernel module that matches your kernel version – Oracle is pretty good at following RHEL kernel versions. Download RPM for your architecture. Do not install it just yet.
2. Switch to ocfs2-tools page. From there follow download link and get ocfs2-tools RPM for your architecture. Install both packages on each node..

rpm -ivh ocfs2-*
warning: ocfs2-2.6.18-194.32.1.el5-1.4.7-1.el5.i686.rpm: Header V3 DSA signature: NOKEY, key ID 1e5e0159
Preparing...                ########################################### [100%]
   1:ocfs2-tools            ########################################### [ 50%]
   2:ocfs2-2.6.18-194.32.1.e########################################### [100%]

2. Cluster configuration – 2 files. First /etc/sysconfig/o2cb

#
# This is a configuration file for automatic startup of the O2CB
# driver.  It is generated by running /etc/init.d/o2cb configure.
# Please use that method to modify this file
#
# O2CB_ENABLED: 'true' means to load the driver on boot.
O2CB_ENABLED=false
# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
O2CB_BOOTCLUSTER=ocfs2

Just change O2CB_ENABLED to true and save.
/etc/ocfs2/cluster.conf – you will need to create it from scratch.

node1#
 vi /etc/ocfs2/cluster.conf
node:
        ip_port=7700
        ip_address = 10.0.0.10
        number=0
        name=node1
        cluster=ocfs2

node:
        ip_port=7700
        ip_address = 10.0.0.11
        number=1
        name=node2
        cluster=ocfs2
cluster:
        node_count = 2
        name=ocfs2

Configuration file contents are pretty self explanatory. Both nodes have to have identical configuration files.
3.Now we need to configure Distributed lock manager (DLM), that comes bundled with OCFS2. For this, start service script with the command line argument “configure”. You will need to do it on each node.

node1#
/etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
<ENTER> without typing an answer will keep that current value.  Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [y]:
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading filesystem "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK

node2#
 /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
<ENTER> without typing an answer will keep that current value.  Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [y]:
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Loading filesystem "configfs": OK
Mounting configfs filesystem at /sys/kernel/config: OK
Loading filesystem "ocfs2_dlmfs": OK
Creating directory '/dlm': OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK
node1#
/etc/init.d/o2cb status
Driver for "configfs": Loaded
Filesystem "configfs": Mounted
Driver for "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Heartbeat dead threshold = 31
  Network idle timeout: 30000
  Network keepalive delay: 2000
  Network reconnect delay: 2000
Checking O2CB heartbeat: Not active

4. All preliminaries covered, now you are ready to format and mount cluster filesystem.

mkfs.ocfs2 /dev/drbd1
mkfs.ocfs2 1.4.4
Cluster stack: classic o2cb
Label:
Features: sparse backup-super unwritten inline-data strict-journal-super
Block size: 1024 (10 bits)
Cluster size: 4096 (12 bits)
Volume size: 304283648 (74288 clusters) (297152 blocks)
Cluster groups: 10 (tail covers 5168 clusters, rest cover 7680 clusters)
Extent allocator size: 2097152 (1 groups)
Journal size: 16777216
Node slots: 2
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 0 block(s)
Formatting Journals: done
Growing extent allocator: done
Formatting slot map: done
Writing lost+found: done
mkfs.ocfs2 successful

node1#
 mount -t ocsf2 -orealtime /dev/drbd1 /shared
node1#
mount
[...]
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
configfs on /sys/kernel/config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/drbd1 on /shared type ocfs2 (rw,_netdev,heartbeat=local)

node1#
dmesg
[...]
OCFS2 Node Manager 1.4.7 Thu Jan  6 01:02:32 EST 2011 (build dca58e8b7045f2d9340a05da754165a9)
OCFS2 DLM 1.4.7 Thu Jan  6 01:02:33 EST 2011 (build 42d6d7419b63e8f3ef85980b4ed113bf)
OCFS2 DLMFS 1.4.7 Thu Jan  6 01:02:33 EST 2011 (build 42d6d7419b63e8f3ef85980b4ed113bf)
OCFS2 User DLM kernel interface loaded
OCFS2 1.4.7 Thu Jan  6 01:02:28 EST 2011 (build 45eaa8a7b7f9cf913f8f9cb33ec017f9)
o2net: accepted connection from node cos-vz (num 1) at 192.168.8.7:7777
ocfs2_dlm: Nodes in domain ("A2240798B5FA46F78E10B8C8205BB6E5"): 0 1
JBD: Ignoring recovery information on journal
kjournald starting.  Commit interval 5 seconds
ocfs2: Mounting device (147,1) on (node 0, slot 1) with ordered data mode.

That’s it. Now you have minimal (2nodes) cluster files system. If you plan to mount partition at boot time, update /etc/fstab with the proper configuration for that. The easiest way to do it (with OCFS2 partition mounted manually), will be

node1#
grep ocfs2 /etc/mtab >> /etc/fstab

Note: If you do have softRAID layer under your OCFS2 partition you will need to get rid of /etc/cron.weekly/99-raid-rebuild script – for some reasion raid rebuild brings huge performance problems to the top OCFS2 layer.

Share Button

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">