NDMP backup backup fails with error: NDMP error(23, NDMP_CONNECT_ERR)

Summary

When attempting an NDMP backup from a NetApp clustered SVM, error 23 (NDMP_CONNECT_ERR) is received from the NetApp filer.

Symptoms

NDMP connections consist of a control and data paths; with this error, the control paths can be established, but the data path cannot. The Control paths are established between all NDMP components and uses port 10000 for the communication, while the data path uses a dynamic port and is established as a connection from the primary to the secondary storage. 

For the description below a backup is done with a clustered NetApp SVM as primary storage (with Proxy client configured on the DPX master server) to the secondary storage (which is a Linux host with the DPX client installed) where a Device server is configured to a DiskDirectory. In this case control paths are from  NDMP proxy node to the Primary storage, from NDMP Proxy node to device server, and between Primary storage node and Device server using  port 10000 for communications. The NDMP Data path should be established from the primary storage (NetApp) to the Secondary Storage listener IP and port, which is dynamically allocated per connection on DPX nibbler.

Errors in Joblog

192.168.23.39 ssjobhnd Thu Dec 13 14:23:51 2018 SNBJH_3036E *** Drive fail reported: drive Data1 on node lnx_svr1 in job 1544707422 ***
192.168.23.39 ssevthnd Thu Dec 13 14:23:51 2018 SNBEHB3021E *** Fail searching socket 5 of module ssdmbs at node  [dev ] ***
192.168.23.39 ssjobhnd Thu Dec 13 14:23:51 2018 SNBJHB3460J --- Putting definition on retry queue: node netApp1 disk /Backup_svm/vol_data of job 1544707422. ---
192.168.23.39 ssjobhnd Thu Dec 13 14:23:51 2018 SNBJH_3045E *** Task 1 of job 1544707422 for node netApp1 disk /Backup_svm/vol_data failed ***
192.168.23.39 ssjobhnd Thu Dec 13 14:23:51 2018 SNBJH_3012E *** final_stat: invalid rqstlink (reqid=4, class=NONE) function ID 896 ***

Error in Data Mover client log xxxxxxx.ncl

Thu Dec 13 14:23:51 2018 SNBNCX1505D Message() sent to module(ndmpd) at host(192.168.23.139) on connection 20
Thu Dec 13 14:23:51 2018 SNBNCX1506D Message(NDMP_DATA_CONNECT) received from module(ndmpd) at host(x.x.x.x) on connection 20
Thu Dec 13 14:23:51 2018 SNBNCX6014E ERROR: Task 11, type(BACKUP), state(DATA_1); Error(501, NC_TASK_ERROR_SERVER_RESPONSE), NDMP error(23, NDMP_CONNECT_ERR); Connection 0, Message token 0
 

Resolution

Possible Causes:

1) Secondary storage server has more than one NIC where one interface address is not reachable by the NDMP LIF of the SVM from NetApp filer (Primary Storage). Therefore no route to data can be established to the device server.