Summary
HP monitoring software may cause false drive errors and set empty volsers to full.
Symptoms
File level or NDMP jobs mount multiple tapes and encounter errors that make them determine that the tapes are full. At the end of the sequence, DMM fails to find available tapes, which causes the job to fail.
Example log snippet follows:
<IP> tptmm Fri Oct 02 15:42:08 2015 SNBTMM5462O Attempting mount for node(Device server name) drive(Drive name) volser(name)
<IP> sstptmm Fri Oct 02 15:43:19 2015 SNBTMM5461O Node(DS name) drive(TD name) volser(name) partition(1) mount rc=(0)
<IP> ssndmpc Fri Oct 02 15:51:37 2015 SNBNCX6025J NDMP server entry: Type(NDMP_LOG_NORMAL), ID(6220), Text(Failed writing to Tape Device [\\.\Tape0]:[GetLastError(2):The system cannot find the file specified. ]. Attempting to change media... )
<IP >ssjobhnd Fri Oct 02 15:54:06 2015 SNBJH_3259J ===== Done tape [volser name] of job <job ID>: 584265 blocks [19145195520 bytes] rc=4 (tape is full) =====
And finally:
<IP> ssjobhnd Fri Oct 02 22:50:05 2015 SNBJH_3038E *** ssdatmgr returned error 2354 (reqid=27, function ID=145) ***
<IP> ssevthnd Fri Oct 02 22:50:05 2015 SNBEHT4235E 6132: *** Failed to find a free tape in tape pool <media pool name>: return code = 2354 ***
<IP> ssevthnd Fri Oct 02 22:50:05 2015 SNBEHT4236E 6132: *** Media bits for getfreetape query: EMPTY NEW APPENDABLE ***
<IP> ssevthnd Fri Oct 02 22:50:05 2015 SNBEHT3057E 6132: *** Last message from DB: Could not reserve any media at device (TD name) using the specified pattern. ***
Resolution
Historically it was known that HP Storage Agents services were probing the SCSI bus and causing intermittent drive failures. This was countered by disabling the service(s) which corrected the problem and allowed the backups to complete successfully.
A new generation/version of HP monitoring software has been released which causes the same symptoms to appear. In some cases, disabling the HP WMI Storage Providers service helps to resolve the problems. In other cases, only removal of the HP Monitoring Software components cleared the issues.
Below is a screenshot of all the components that must be removed for future reference: