Oracle Grid Infrastructure Installation Issues on Linux

This page discusses a multicasting issue encountered installing Oracle on a two-node x86-64 cluster running Oracle Enterprise Linux 5 Update 5.

The Grid Infrastructure installation went smoothly until we tried to run on the second node. The script failed with the following error:

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS 
  daemon on node <nodename>, number 1, and is terminating
  An active cluster was found during exclusive startup, restarting to join the 
  Failed to start Oracle Clusterware stack
  Failed to start Cluster Synchorinisation Service in clustered mode at 
  /u01/app/ line 1016
  /u01/app/ execution failed

We decided to try installing Oracle and then upgrading to Oracle This process requires the latest Grid Infrastructure PSU to prevent a failure when the script is executed on the first node. When this patch has been applied, succeeded on the first node, but failed with the error described above on the second node.

This issue is described in MOS Note 1212703.1 "Grid Infrastructure install or upgrade may fail due to Multicasting"

If multicasting is not enabled on the private network, then will be successful on the first node, but will fail on the second and subsequent nodes when attempting to start CSSD. This affects both installations and upgrades.

Multicasting is required to enable the new HAIP interconnect feature. If multicast is not enabled, the node will not be able to join the cluster.

According to the note, the only solution is to enable multicasting on the private network (interconnect). This could be difficult on a production system, particularly for an out-of-hours upgrade where the relevant network specialists may not available to modify the switch configurations.

However, we did some research and it appears that multicasting is already enabled by default in OEL5U5. Each network interface described in ifconfig already had MULTICAST enabled. However past experience tells us that just because something is configured at operating system level, we cannot assume it is configured at switch level - remember jumbo frames?

Since we originally discovered this problem, Oracle have released a utility to test the availability of multicast addresses. The utility is called mcasttest and can be downloaded from MOS Note 1212703.1 Grid Infrastructure install or upgrade may fail due to Multicasting.

In the environment discussed here the mcasttest utility returned the following output:

$ ./ -n server23,server24 -i bond1
###########  Setup for node server23  ##########
Checking node access 'server23'
Checking node login 'server23'
Checking/Creating Directory /tmp/mcasttest for binary on node 'server23'
Distributing mcast2 binary to node 'server23'
###########  Setup for node server24  ##########
Checking node access 'server24'
Checking node login 'server24'
Checking/Creating Directory /tmp/mcasttest for binary on node 'server24'
Distributing mcast2 binary to node 'server24'
###########  testing Multicast on all nodes  ##########

Test for Multicast address
Nov 19 11:29:11 | Multicast Failed for bond1 
  using address

Test for Multicast address
Nov 19 11:29:12 | Multicast Succeeded for bond1 
   using address

The mcasttest utility first attempts to use which is the default address. It then repeats the test for If the first test fails, but the second test succeeds as shown in the above example then Oracle recommends that the patch for bug Bug 9974223 - "Grid Infrastructure needs multicast communication on address working" is installed on each node in the cluster after installation of the Oracle binaries, but before running or

I have subsequently successfully installed Oracle Grid Infrastructure at another site without any issues. The second site is also a 2-node Linux x86-64 cluster, this time running Red Hat Enterprise Linux 5 Update 4. Both the public and private networks are bonded. In this case Oracle installed without any problems at the first attempt.

In Oracle and above this issue has been resolved by Oracle. The installation process now attempts to detect whether multicasting is enabled; if available then multicasting is used; if not available then the installer reverts to the non-multicasting algorithm used in Oracle