Multipath errors for ASM devices in /var/log/messages

We are currently running failover tests on a three node Linux x86-84 cluster running Oracle 11.2.0.2 with the 11.2.0.2.2 PSU. As part of this process we have been examining how each failure is recorded in the log files. One of the log files we have been monitoring is /var/log/messages.

When a node is shutdown messages similar to the follow appear in /var/log/messages:

Jun 14 19:19:38 server20 multipathd: uevent trigger error
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg3: 
remove path (uevent)
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg3: 
spurious uevent, path not in pathvec

When the node is restarted further messages are written to /var/log/messages:

Jun 14 19:19:38 server20 multipathd: uevent trigger error
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg0: 
  add path (uevent)
Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg0: 
  failed to store path info

Similar messages appear for a number of devices in our case including:

We found the ofsctl device very easily:
[root@server20]# ls -l /dev/ofsctl
brw-rw-r-- 1 root oinstall 251, 0 Jun 14 19:19 /dev/ofsctl

The remaining devices were harder to locate. They did not appear to be located in /dev/asm.

[root@server20]# ls -l /dev/asm
total 0

However, on further inspection we discovered they had a "dot" prefix:

[root@server20]# ls -la /dev/asm
total 0
drwxrwx---  2 root oinstall     280 Jun 14 19:19 .
drwxr-xr-x 17 root root        5680 Jun 14 19:19 ..
brwxrwx---  1 root oinstall 252,  0 Jun 14 19:19 .asm_ctl_spec
brwxrwx---  1 root oinstall 252, 10 Jun 14 19:19 .asm_ctl_vbg0
brwxrwx---  1 root oinstall 252, 11 Jun 14 19:19 .asm_ctl_vbg1
brwxrwx---  1 root oinstall 252, 12 Jun 14 19:19 .asm_ctl_vbg2
brwxrwx---  1 root oinstall 252, 13 Jun 14 19:19 .asm_ctl_vbg3
brwxrwx---  1 root oinstall 252, 14 Jun 14 19:19 .asm_ctl_vbg4
brwxrwx---  1 root oinstall 252, 15 Jun 14 19:19 .asm_ctl_vbg5
brwxrwx---  1 root oinstall 252, 16 Jun 14 19:19 .asm_ctl_vbg6
brwxrwx---  1 root oinstall 252, 17 Jun 14 19:19 .asm_ctl_vbg7
brwxrwx---  1 root oinstall 252, 18 Jun 14 19:19 .asm_ctl_vbg8
brwxrwx---  1 root oinstall 252,  1 Jun 14 19:19 .asm_ctl_vdbg
brwxrwx---  1 root oinstall 252,  2 Jun 14 19:19 .asm_ctl_vmb
brwxrwx---  1 root oinstall 252,  2 Jun 10 15:06 .asm_ctl_vmb

It appears these devices were added in Oracle 11.2.0.2 and the messages in /var/log/messages are fairly benign although still annoying.

MOS Note 1268895.1 "Multipathd: Asm!.Asm_ctl_spec: Failed To Store Path Info' found In /var/log/messages" describes a similar error and recommends blacklisting the new devices in /etc/multipath.conf. For example:

blacklist {
  devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
  devnode "^(hd|xvd|vd)[a-z]*"
  devnode "ofsctl"
  devnode "^asm/*" 
}

Adding the new blacklist entries (shown in bold) to /etc/multipath.conf eliminated the warnings in /var/log/messages in our environment.