Summary
The number of concurrent SnapVault and SnapMirror transfers to a filer is limited. One should avoid exceeding this limit during Open System SnapVault (OSSV) and SnapVault (SV) jobs. This article explains an elegant way to stay within this limit imposed by the filer.
Resolution
Executing the OSSV Jobs through DPX
It is safe to assume that every Windows server running the Catalogic Software OSSV agent opens roughly 5 simultaneous transfers to a filer during an OSSV update. Filers are limited as far as the maximum number of simultaneous transfers are concerned. As an example, a FAS3020 with the NearStore Personality installed can have a maximum of 32 simultaneous transfers. This equates to a maximum of 6 servers backing up at once (32 streams / 5 streams per server).
In this example, to ensure that no more than 6 servers will backup to the filer at once, one will have to schedule the backup jobs so to make sure their runtimes don't overlap. If a job unexpectedly runs longer than expected, this longer running job can cause the maximum number of transfers to be exceeded if scheduled jobs start while this job is still running.
To avoid a situation like this one can start the jobs sequentially using the DPX command line utility BEX (see the Catalogic DPX Reference Guide, Interfaces section for more information on BEX). Using one instance of bex, only one job will run at any given point in time, and as soon as this job finishes, the second (and so on) job will start. One can then run multiple instances of bex to make sure that the maximum number of transfer limit is not exceeded. In our example where the filer can accept a maximum number of 32 streams, one will run no more than 6 instances of bex at any given point in time.
As an example, let us assume we are writing data to a FAS3020 with the NearStore Personality installed. As stated above, this filer can accept a maximum of 32 simultaneous SV/SM streams. Let us further assume that we have a total of 50 servers that we would like to backup to this filer. The first step would be to create the flexible volumes on the filer to house the data or more information on volume creation). To keep it simple, create a volume for every server to be backed up. Let the volume name equal that of the servers. So one would create a total of 50 flexible volumes on the filer as per the Knowledge Base article referenced above. Now create a SnapVault backup job for each of these servers. Let the jobname also be the same as the server name. If the servers are named Server01, Server02, etc. one would have the OSSV backup job Server01 backup all (or a subset) of the data on Server01 to a volume on the filer called /vol/Server01. (One can also add as part of the volume name a site descriptor, or job descriptor to distinguish the backup volumes from the other volumes on the filer. An example would be to name the volumes /vol/SV_Server01 etc.)
At the end of this step, one will have a total of 50 OSSV backup jobs. Make sure not to define these jobs with any schedules associated with them. It is also good practice to run the full backup before proceeding. That will reduce the amount of data that will be transferred during the next step.
Now create 6 files (32 streams / 5 streams per server) each containing a list of the DPX Job Names created in the step above. For sake of example, we will assume that bex will run on a Windows node (which may or may not be the Master Server - please see the bex documentation for more details). Name these files Sched01.txt, Sched02.txt etc and place them (as an example) in the folder C:\Schedules on this server. An example of the contents of these files is shown below:
Sched01.txt will contain:
Server01
Server02
Server03
Server04
Server05
Server06
Server07
Server08
Sched02.txt will contain:
Server09
Server10
Server11
Server12
Server13
Server14
Server15
Server16
and so on. Sched06.txt will have as final entry the DPX job Server50 in it, which backs up the server with hostname Server50 to a volume on the filer named /vol/Server50 in our example.
The next step is now to create the script that will execute these jobs. Place these scripts in the C:\Program Files\Backup Express\sched\scripts folder on this server. Create one of these scripts for each of the schedule files defined above. So in our example, we will create 6 of these scripts (call them Backup01.bat, Backup02.bat and so on). They will contain the following entries:
Backup01.bat
@echo off
set PATH=%PATH%;"C:\Program Files\Backup Express\sched\scripts"
for /f %%u in (C:\Schedules\Sched01.txt) do call SV_start.bat %%u
Backup02.bat
@echo off
set PATH=%PATH%;"C:\Program Files\Backup Express\sched\scripts"
for /f %%u in (C:\Schedules\Sched02.txt) do call SV_start.bat %%u
and so on. So each of these scripts will call another script (defined below) named SV_start.bat that will pass to it as argument one by one the contents of the schedule files Sched01.txt, Sched02.txt.
The scripts SV_start.bat that should also reside in the C:\Program Files\Backup Express\sched\scripts folder on this server contain the following entries:
SV_start.bat
set SSPRODIR=C:\Program Files\Backup Express
"%SSPRODIR%\bin\bex" -i"%SSPRODIR%\sched\scripts\bex.ini" -c "backup run -n %1 -t backup_snapvault_incr"
This script resides on the file C:\Program Files\Backup Express\sched\scripts\bex.ini that contains the DPX login information:
bex.ini:
server <MS_Server>
user <BEXUsername>
password <BEXpasswd>
with <MS_Server> the hostname of the master server, <BEXUsername> the DPX username with rights to execute the defined DPX OSSV jobs (normally sysadmin), and <BEXpasswd> the corresponding DPX user's password.
So by executing the scripts Backup01.bat, the OSSV backup jobs listed in Sched01.txt will be executed one after another. So running all six scripts (Backup01.bat - Backup06.bat) concurrently, one will ensure that no more than 6 jobs will ever run at the same time. And since each job opens on average 5 SV/SM streams to the filer, one will always have less that 32 streams writing to the filer at any point in time. The logic also extends to other filers models with different stream requirements. For example in the case of a filer where the maximum number of streams equal 16, one will have no more than 3 (16/5) backup scripts (Sched01.txt, Sched02.txt, Sched03.txt). And each of these will contain one by one the defined OSSV backup jobs.
One can now schedule these scrtips to run by using any external scheduler. As an example, one can use the Windows scheduler at to schedule these scripts. An example output from the at command on this server may look like this:
Status ID Day Time Command Line
-----------------------------------------------------------------------------
1 Each M T W Th F S Su 07:00 PM cmd /C "C:\Program Files\Backup Express\
sched\scripts\Backup01.bat"
2 Each M T W Th F S Su 07:00 PM cmd /C "C:\Program Files\Backup Express\
sched\scripts\Backup02.bat"
3 Each M T W Th F S Su 07:00 PM cmd /C "C:\Program Files\Backup Express\
sched\scripts\Backup03.bat"
4 Each M T W Th F S Su 07:00 PM cmd /C "C:\Program Files\Backup Express\
sched\scripts\Backup04.bat"
5 Each M T W Th F S Su 07:00 PM cmd /C "C:\Program Files\Backup Express\
sched\scripts\Backup05.bat"
6 Each M T W Th F S Su 07:00 PM cmd /C "C:\Program Files\Backup Express\
sched\scripts\Backup06.bat"
These schedules will execute the jobs (shown under the Command Line column) everyday starting at 07:00pm.
Conclusion
As soon as any of these jobs are started using bex, the job will appear in the Job Monitor window (and will also be visible in the Backup Express Reporter if applicable). Normal job operations (cancel, monitor etc.) can be performed on the job. One will, however not see these jobs as being scheduled through the DPX scheduler (i.e. they will not show up as Waiting in the Job Monitor window).