Networker

[Networker] Staging: Looks like the wrong save sets have been staged

2013-05-06 11:30:29
Subject: [Networker] Staging: Looks like the wrong save sets have been staged
From: tammclaughlin <networker-forum AT BACKUPCENTRAL DOT COM>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Mon, 6 May 2013 08:30:20 -0700
I have an issue with a staging policy where the most recent save sets have been 
staged rather than the oldest.
Let me explain this:


Some background:

Networker 7.6.09
backup to adv_files and then clone to tape
staging from adv_files to tape.

staging policy:  start: 95%, stop: 91%, oldest, max days: 7, recover: 5 days: 
check fs: 120minutes

We have a faulty tape drive so currently running on 1 drive until the 
replacement arrives.
As the weekend backups were full backups, on Friday, I forced staging by 
changing the thresholds to ensure I had as much free space as I
could get for the weekend. This was to allow the backup jobs to be cloned to 
tape with minimal contention if staging set in.

So today I saw that the backups had hung as networker was waiting on a stage 
tape that was 100% full.
When I investigated I found that nsrstage was not running and the filesystems 
were within the threshold limits. 
What was happening was that a job was trying to clone a save set just created 
but the save set was now on a stage tape.
It could not load the stage tape because there was only 1 drive which had the 
"destination" tape for the clone.

So why did the stage tape have most recent save sets?
I looked at the volumes were on the filesystem which seems to give some clues.

filesystem:  /diskbackup1

Total:   16TB,  1.8TB free (currently)

volume: size on disk        monthly backup size
notes:   13T                     6.8T                 
unix:      17G                    300G
linux:      79M                   400G


The most recent save sets staged were from the volumes unix and linux and very 
few from the volume notes. In fact almost all
of the unix and linux save sets have been staged.
Now some of the largest save sets from notes are 500GB so it's possible that 
all of the linux save sets can be staged in the 
time it takes to stage just one notes save set.

I expected the staging police to compile a list of the oldest stage sets across 
all devices and then move to tape which would mean that the most recent
would still be on disk. So it seems that staging is selecting save sets in a 
different manner.
Could it be that it treats each  volume separately?
Could it be looking for the oldest save sets in each volume and starts to stage 
them. While still writing the larger notes save sets, it goes back to look at 
other volumes and takes from other volumes as the notes is still busy with such 
a large save set?

Another possibility is that the next filesystem check starts while it is still 
staging and cannot read from the notes volume as it is still being used and 
takes from the unix/linux volume instead and just keep taking until it meets 
the required threshold?

Thanks.

+----------------------------------------------------------------------
|This was sent by tam.mclaughlin AT gmail DOT com via Backup Central.
|Forward SPAM to abuse AT backupcentral DOT com.
+----------------------------------------------------------------------