We are currently running failover tests on a three node Linux x86-84 cluster running Oracle 11.2.0.2 with the 11.2.0.2.2 PSU. As part of this process we have been examining how each failure is recorded in the log files. One of the log files we have been monitoring is /var/log/messages.
When a node is shutdown messages similar to the follow appear in /var/log/messages:
Jun 14 19:19:38 server20 multipathd: uevent trigger error Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg3: remove path (uevent) Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg3: spurious uevent, path not in pathvec
When the node is restarted further messages are written to /var/log/messages:
Jun 14 19:19:38 server20 multipathd: uevent trigger error Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg0: add path (uevent) Jun 14 19:19:38 server20 multipathd: asm!.asm_ctl_vbg0: failed to store path info
Similar messages appear for a number of devices in our case including:
[root@server20]# ls -l /dev/ofsctl brw-rw-r-- 1 root oinstall 251, 0 Jun 14 19:19 /dev/ofsctl
The remaining devices were harder to locate. They did not appear to be located in /dev/asm.
[root@server20]# ls -l /dev/asm total 0
However, on further inspection we discovered they had a "dot" prefix:
[root@server20]# ls -la /dev/asm total 0 drwxrwx--- 2 root oinstall 280 Jun 14 19:19 . drwxr-xr-x 17 root root 5680 Jun 14 19:19 .. brwxrwx--- 1 root oinstall 252, 0 Jun 14 19:19 .asm_ctl_spec brwxrwx--- 1 root oinstall 252, 10 Jun 14 19:19 .asm_ctl_vbg0 brwxrwx--- 1 root oinstall 252, 11 Jun 14 19:19 .asm_ctl_vbg1 brwxrwx--- 1 root oinstall 252, 12 Jun 14 19:19 .asm_ctl_vbg2 brwxrwx--- 1 root oinstall 252, 13 Jun 14 19:19 .asm_ctl_vbg3 brwxrwx--- 1 root oinstall 252, 14 Jun 14 19:19 .asm_ctl_vbg4 brwxrwx--- 1 root oinstall 252, 15 Jun 14 19:19 .asm_ctl_vbg5 brwxrwx--- 1 root oinstall 252, 16 Jun 14 19:19 .asm_ctl_vbg6 brwxrwx--- 1 root oinstall 252, 17 Jun 14 19:19 .asm_ctl_vbg7 brwxrwx--- 1 root oinstall 252, 18 Jun 14 19:19 .asm_ctl_vbg8 brwxrwx--- 1 root oinstall 252, 1 Jun 14 19:19 .asm_ctl_vdbg brwxrwx--- 1 root oinstall 252, 2 Jun 14 19:19 .asm_ctl_vmb brwxrwx--- 1 root oinstall 252, 2 Jun 10 15:06 .asm_ctl_vmb
It appears these devices were added in Oracle 11.2.0.2 and the messages in /var/log/messages are fairly benign although still annoying.
MOS Note 1268895.1 "Multipathd: Asm!.Asm_ctl_spec: Failed To Store Path Info' found In /var/log/messages" describes a similar error and recommends blacklisting the new devices in /etc/multipath.conf. For example:
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^(hd|xvd|vd)[a-z]*"
devnode "ofsctl"
devnode "^asm/*"
}
Adding the new blacklist entries (shown in bold) to /etc/multipath.conf eliminated the warnings in /var/log/messages in our environment.