Summary
Once you have discovered and corrected a LUN-Busy condition on your Network Appliance Filer, you may still be in a position where you cannot delete your previous backups due to local data retention policy, and you may also be running out of volume space due to your inability to delete snapshots.
Resolution
Clearing Away LUN-Busy Snapshots
Prerequisites
This is a follow-up to Cannot Delete Filer Snapshot: Understanding the LUN-Busy Snapshot Condition. If you are not familiar with diagnosing the source of 'LUN-Busy' that can lead to Backup Express being unable to delete these snapshots, please read the referenced KB before continuing with this article. It will be important for you to know how this condition starts, and how to identify which snapshots are involved in the LUN-Busy condition.
This article is meant to be a guide for knowledgeable NetApp Filer administrators.
Introduction
This article describes a few methods you can use to help correct the 'LUN-Busy' conditions and free up space on your Filer.
As described in Cannot Delete Filer Snapshot: Understanding the LUN-Busy Snapshot Condition, an iSCSI map that has persisted on a filer for a long period of time can cause a large number of your snapshots to be stuck in an unremovable state. The only way to clear up this condition is to destroy the LUN that was created, and then delete all snapshots that are involved in the LUN-Busy condition.
This can pose a problem for administrators. Snapshots represent point-in-time backups of your volume; your site may not permit you to remove these snapshots for a long period of time. Snapshots in LUN-Busy can take up significant resources that are needed for current and future backups.
Please refer to Cannot Delete Filer Snapshot: Understanding the LUN-Busy Snapshot Condition regarding use of 'lun snap usage' command to investigate which snapshots are involved in the LUN-Busy condition, and how to deal with cleaning them up. Deleting snapshots that are stuck in LUN-Busy can be complicated when multiple iSCSI maps were initiated and/or overlap over a period of time. Cannot Delete Filer Snapshot: Understanding the LUN-Busy Snapshot Condition also outlines how to find and remove all snapshots keeping a volume busy, even if multiple iSCSI maps are involved. It also details how to look for LUNs that are defined on your filer, and how to remove dependencies on the special Snapvault base relationship snapshot if they exist.
Snapshot Cleanup Suggestions
Here are a few ways you can clean or move data from your Filer to help save space:
1) Delete your snapshots - If you can afford to do so, simply delete those snapshots that are involved with the LUN-Busy condition. If only a small number of snapshots are involved, then the overall loss of data might be acceptable. For more detailed procedures on deleting LUN-Busy snapshots, please read Cannot Delete Filer Snapshot: Understanding the LUN-Busy Snapshot Condition.
2) NDMP 'dump' backup - If you have only a few snapshots that are stuck, you can perform an ad-hoc NDMP dump backup of the affected resources to tape, and then delete them from the Filer. To do this, go to Configure > Enterprise, choose your Filer, and verify that dump is the selected backup type. The following figure is an example of what the bottom portion of the Right hand pane should contain after clicking on your NetApp Filer node:
Once you have verified that your Filer is set for dump backups, go to Backup > NDMP, open up your Filer, and drill down to your affected volume. Open up the .snapshot directory, and then choose to back up a snapshot that is affected by the LUN-Busy condition. Only 1 snapshot may be backed up at a time in this way. The following example shows a Filer volume where 1 snap within the .snapshot directory is chosen to be backed up:
Define which tape/media resources to send this backup to, save the job, and run it as you would any other NDMP tape backup job. Backup any other affected snapshots in this same way. When you are finished backing up your affected snapshots, simply delete them all from the filer. Remember to change the backup type for this filer back to its original value if you changed it.
This solution should be considered for small numbers of snapshots only. Each snapshot backed up represents a point-in-time copy of all the QTree data on the volume. The benefit to backing up individual snapshots is that these can be individually restored and manually IA mapped for data access. The downside for 'dump' restored data is that the restored data cannot be easily accessed from the Backup Express GUI for OSSV restore or IA map; however the data can be manually iSCSI mapped from the Filer. Please see How to Protect SnapVault Data on Tertiary Storage and Recover the Data for details on how to restore these backups and manually IA map them for restore.
3) NDMP 'smtape' backup - If you have many snapshots that are stuck, you can perform an ad-hoc NDMP smtape backup of the entire volume to tape, and then delete all affected snapshots from the Filer. To do this, go to Configure > Enterprise, choose your Filer, and verify that smtape is the selected backup type. The following figure is an example of what the bottom portion of the Right hand pane should contain after clicking on your NetApp Filer node:
Once you have verified that your Filer is set for smtape backups, go to Backup > NDMP, open up your Filer, and drill down to your affected volume -- smtape backup is only valid at the volume level.
Once you have chosen the volume, define which tape/media resources to send this backup to, save the job, and run it as you would any other NDMP tape backup job. When you are finished backing up the whole volume, you can delete all the affected snapshots from your filer. Remember to change the backup type for this filer back to its original value if you changed it.
This solution should be considered if a large number of snapshots need to be removed. The benefit to backing up the whole volume via smtape is that restored data can be easily used for ExpressDR, and the 'alternate secondary' feature of Snapvault restore can be used if the affected jobs have not been condensed from the catalog. Manual IA map of snapshot data can always be accessed. The downside of smtape restore is that the whole volume must be restored at once and this could take up significant time and resources (but the data can be placed on another aggregate or alternate Filer if necessary). How to Protect SnapVault Data on Tertiary Storage and Recover the Data details how to manually IA map restored snapshots.
4) SnapMirror - If your NetApp Filer is licensed for SnapMirror, then you may want to SnapMirror the affected volume to another aggregate or Filer for temporary archiving. Once the SnapMirror is complete, you can break the SnapMirror relationship at the ONTAP command line via snapmirror break <vol> and then remove the affected snapshots from the original volume. Once the archive is no longer needed, simply destroy the SnapMirrored copy. For Snapvault jobs that have not condensed out of your catalog, you can use the 'alternate secondary' feature to restore files and IA map. If your jobs have condensed out of the catalog, then you can manually create iSCSI mapping from these snapshots and recover files.
5) Create a new volume - If you have the space available in another aggregate or local Filer, then it might be easier to simply create a new volume somewhere else. Once your local data retention policy allows, you can go back and remove the old volume. If the volume was used for NFS or CIFS sharing, you will need to transfer over any important data, stop access to the old volume, and set up access to your new volume. If the volume is used for OSSV backups, a new base backup will be necessary in the new volume. This can be done by deactivating the old job, and creating a new job that sends backups to the new volume. If you need to retain the original job name, you can change the destination, however you will need to clear out the client-side Change Journals before they will successfully backup to the new (empty) volume. Please call Backup Express technical support if you need help clearing out a client-side change journals.
6) Copy data with 'vol copy' - NetApp filers also support copying data between aggregates and between filers through use of the 'vol copy start' command. Please consult your ONTAP documentation regarding options and prerequisites needed in order to use the vol copy command. Before using 'vol copy', you need to create the destination volume and set it to the 'restricted' mode. By default, vol copy start<src-vol> <dest-vol> only copies data at the root level (QTrees, directories, etc.) and snapshots are not copied. To copy the entire volume to another aggregate (including snapshots), use the vol copy start-S<src-vol> <dest-vol>. For OSSV backup volumes, it is suggested that you copy all of your data to a volume with a new name for archival purpose, and then follow Step 5 (Create a new volume) above. If you copy your data to a different volume name, then you will be able use the 'alternate secondary' option to access OSSV restore and IA map features from the DPX GUI. If your data is copied on the original Filer and renamed the original volume name, you will not be able to use OSSV restore or IA map features from the GUI; you will only be able to manually iSCSI map your data unless you temporarily re-name the volume to something different. If an OSSV backup job attempts to write an incremental backup into the copied volume it will default to running a new Base backup. Regardless of the volume name, ExpressDR will be able to access and restore systems from data moved with vol copy start-S.
7) File backup (NDMP backup not available) - If you do not have an NDMP license for your Filer, it is possible to backup data by mounting your volume via NFS or CIFS (requires separate ONTAP licensing), copy the affected snapshots to a DPX client/device machine, back up the files to tape and then remove the affected snapshots. The snapshot data can be restored and copied back to an alternate Filer folder via NFS/CIFS. That data recovered will only be usable via manual iSCSI LUN creation after the data has been restored to a volume and the volume has had a snapshot taken.
8) iSCSI map - If you have a small number of snapshots to archive, you can iSCSI map all the affected snapshots to a DPX Windows client machine and run a standard file-level backup as described in How to Protect SnapVault Data on Tertiary Storage and Recover the Data. This allows for individual file backup from those snapshots, however you need to restore such files back to a Windows accessible client machine. After your backup is successful, detach and destroy your iSCSI LUNs, and then remove the necessary snapshots.