Details
Multiple redundant soft links in /etc/mapper pointing to the same device may cause application discovery to fail on ECX. This is related to a quirk with how some distributions set up mapper devices and the ‘dmsetup’ command being confused by these extra links.
The issue was discovered on SUSE 12, but the condition may generically affect other versions of Linux. SUSE 12 may be configured with or without an /etc/multipath.conf file, so device and mapper names referenced might be different. The host SUSE 12 system discovered used the following /etc/multipath.conf file:
defaults {
user_friendly_names yes
find_multipaths yes
}
Error analysis
You may see errors in the ECX job log similar to below:
Failed to determine covering disks for volume /dev/mapper/mpatha-part1
The device above relates to a multipath enabled mount point that ECX discovered application resources to protect (such as database and log files). Your system’s mapper and devices will be different. Following here is an example you can use to verify the issue.
Discern the /dev entry for which the device in question relates to:
# ls -l /dev/mapper/mpatha-part1
total 0
lrwxrwxrwx 1 root root 7 Aug 12 11:50 mpatha_part1 -> ../dm-6
Try to manually run dmsetup against the referenced device:
# dmsetup info -c --noheadings -o name,uuid /dev/dm-6
Device does not exist.
Command failed.
Failure of the dmsetup command directly relates to ECX agent’s inability to discern the devices needed to protect an application. If you list out all of the contents of /dev/mapper, you will find more than 1 soft link pointing to the same device as seen below:
# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 236 Aug 12 11:49 control
lrwxrwxrwx 1 root root 7 Aug 12 11:50 mpatha -> ../dm-5
lrwxrwxrwx 1 root root 7 Aug 12 11:50 mpatha_part1 -> ../dm-6
lrwxrwxrwx 1 root root 7 Aug 12 11:50 mpatha-part1 -> ../dm-6
You will note above that the base whole device (mpatha) is represented with a single link, but the devices representing partitions have multiple entries pointing to the same /dev/device. In the case above the mpatha device has only 1 partition. Although this specific example was taken from a SUSE 12 system the issue is not specific to SUSE 12. If you were to create any arbitrary soft link in /dev/mapper pointing to a common device (such as RandomLink -> ../dm-6), the same issue would occur, even on different Linux distributions using multipath.
This occurs with SUSE 12 due to a udev rule designed to create extra soft links, likely for some backwards compatibility reason. Udev will find names in /dev/mapper and create additional *_partN named devices to maintain this compatibility. The overall issue here is not specifically with udev, but with Linux dmsetup and related utilities being unable to deal with the extra redundant soft links in /dev/mapper. Udev on SUSE 12 is creating this condition by default whereas SUSE 15 and other Linux distribution do not configure udev to do this.
Correction
This issue can occur on any multipath enabled Linux system where there exists redundant soft links in /dev/mapper which point to common devices. In the simplest case, these redundant soft links should be deleted. Take note that you may need to adjust /etc/fstab (as indicated below) if the links you are removing were used for file system mounts.
For SUSE 12 you cannot simply delete the redundant soft links as udev will recreate them. In this case, you need to configure udev to disable the behavior. Look for the following file:
/usr/lib/udev/rules.d/67-kpartx-compat.rules
Comment out the following 2 lines:
ENV{DM_MPATH}=="?*", ENV{DM_PART}=="?*", \
SYMLINK+="mapper/$env{DM_MPATH}_part$env{DM_PART}"
This will prevent udev from creating the redundant soft links with the underscore character (like mpatha_part1). If you were using the underscore names in /etc/fstab to mount file systems, you will need to change /etc/fstab to reflect the corrected new device path name.
Once making the udev changes and adjusting /etc/fstab as needed, it is advised to reboot the system to observe that the redundant links are no longer created in /dev/mapper and that all necessary file systems are mounted automatically on reboot. If you cannot afford a reboot, consult the system manual pages on ‘systemd’ and ‘udev’ for additional instructions on how to safely restart the necessary services to avoid disruption.