[Veritas-bu] Backing up to disk (and then tape ??)
2003-09-15 11:59:04
Here's my script for doing this. It's pretty long but it has error
handling, extensive logging, etc. It can use a configuration file that'll
lookup policy/sched pairs and cross reference a tape pool from that (since a
disk STU doesn't use pools). The conf file looks simply like:
# Class/policy Sched Volume_Pool
Oracle_Arcs Archive Ora_Arc
Archives * Arc
Pkgs 1year Pkg
(white space separated, comments allowed) You can use a "*" as a schedule
name to do just a policy with all its schedules.
It'll group images together into sets (GLOB variable) to keep the individual
bpduplicate jobs from running too long. It'll also do multiple sets
simultaneously (SIMUL variabe) if you so desire.
It use bpimagelist to query the disk STU's imagelist, groups images by
destination tape pool, and then uses bpduplicate to create a second copy of
sets of those images. If the second copy of an images is verified after the
duplication process, then it uses bpexpdate to expire the disk copy. The
promotion of the tape copy to the primary restore image is automatic when
the disk image is expired.
Look it over, cherry-pick or use it as you will.
-M
#!/bin/ksh
PROGNAME=`basename $0`
#Default volume pool if config file doesn't have a match
DEFAULTPOOL="HSM"
#Number of simultaneous bpduplications to run (per pool)
SIMUL=2
#Filesystem for disk STU
DISK="/images"
#Destination Storage unit
STU="L3500_4"
#Number of images per duplication group per pool
GLOB=5
#email address for reports
MAILADDR=YOU AT YOURDOMAIN DOT COM
#Configuration file - lookup pools for class/sched pairs
CONFFILE=/usr/openv/local/$PROGNAME.conf
PATH=$PATH:/usr/openv/netbackup/bin/admincmd
LOGDIR=/usr/openv/netbackup/logs
LOGFILE=$LOGDIR/$PROGNAME.log.1.$$
DETAILS=$LOGDIR/$PROGNAME.log.2.$$
KEEPLOG=$LOGDIR/$PROGNAME.log.`date +%a`
LOCKFILE=/tmp/$PROGNAME.LCK
BIDFILE=/tmp/$PROGNAME.bidfile.$$
POOLFILE=/tmp/$PROGNAME.poolfile.$$
SPLITFILE=/tmp/$PROGNAME.set.
###################
## Function Defs ##
###################
verifytape () {
# Function: Supply bpimageid, returns 0=not-on-tape, else n=copy on tape
image=$1
pattern=`echo "$DISK" | sed 's,\/,\\\/,g'`
copy=`bpimagelist -backupid $image|\
awk 'BEGIN {copy=0}
{if ($1=="FRAG" && $9!~/^'$pattern'/) {copy=$2}}
END {print copy}'`
echo $copy
return $copy
}
diskcopy () {
# Function: Supply bpimageid, returns 0=not-on-disk, else n=copy on disk
image=$1
pattern=`echo "$DISK" | sed 's,\/,\\\/,g'`
copy=`bpimagelist -backupid $image|\
awk 'BEGIN {copy=0}
{if ($1=="FRAG" && $9~/^'$pattern'/) {copy=$2}}
END {print copy}'`
echo $copy
return $copy
}
dupit () {
#Function to duplicate & verify a list of images given a bidfile
dupfile=$1
echo "## Start: `basename $dupfile`"
# The copy below is to correct for the bpduplicate "bug" where
# the bidfile is deleted after use by bpduplicate.
cp ${dupfile} ${dupfile}.2
sed 's/^/## /' $dupfile
echo "## Calling bpduplicate for `basename $dupfile`."
bpduplicate -L $DETAILS -dp $POOL -dstunit $STU -Bidfile $dupfile
2>>$DETAILS
RC=$?
if [ $RC -ne 0 ]
then
MRC=1
echo "## WRN: bpduplicate error: $RC."
fi
mv ${dupfile}.2 ${dupfile}
echo "## Expire: `basename $dupfile`"
for bpid in `cat $dupfile`
do
#Update the value
retval=`verifytape $bpid`
if [ $retval -eq 0 ]
then
#Duplication failed - no tape version
echo "## WRN: Duplicate of $bpid not found on tape." | tee -a $DETAILS
MRC=1
else
copynum=`diskcopy $bpid`
if [ $copynum -ne 0 ]
then
echo "## Expiring copy number $copynum of $bpid." >>$DETAILS
bpexpdate -d 0 -backupid $bpid -force -copy $copynum 2>>$DETAILS
if [ $? -ne 0 ]
then
echo "## ERR: bpexpdate had non-zero return code for \"$bpid\"." |
tee -a $DETAILS
MRC=1
fi
else
echo "## ERR: lookup of disk copy number failed for $bpid." | tee -a
$DETAILS
MRC=1
fi
fi
done
[ -f $dupfile ] && rm -f $dupfile
echo "## Complete: `basename $dupfile`"
return $MRC
}
cleanup () {
#Function to clean up after trap or end of script
echo "## Cleanup"
[ -f $LOCKFILE ] && rm $LOCKFILE
echo "## $PROGNAME Complete: `date`"
#Delete old cumulative log file
find $LOGDIR -name "$PROGNAME.log.[SMTWF]??" -mtime +5 -exec rm {} \;
#Append to current cumulative log file
[ -f $LOGFILE ] && cat $LOGFILE >>$KEEPLOG
[ -f $DETAILS ] && cat $DETAILS >>$KEEPLOG
if [ $MRC -ne 0 ]
then
subject="$PROGNAME: Errors"
else
subject="$PROGNAME: Output"
fi
cat $LOGFILE $DETAILS | mailx -s "$subject" $MAILADDR
# Logfile Cleanup
[ -f $LOGFILE ] && rm $LOGFILE
[ -f $DETAILS ] && rm $DETAILS
[ -f $BIDFILE ] && rm $BIDFILE
[ -f $POOLFILE ] && rm $POOLFILE
[ -f ${SPLITFILE}?? ] && rm ${SPLITFILE}??
}
##########
## MAIN ##
##########
if [ -n "$1" ]
then
#Something on command line - give usage statement
echo "Usage: $PROGNAME"
echo "\nThis script will duplicate images from a disk storage area to a "
echo "set of tapes. Usage should be regularily via cron as root."
else
MRC=0
#Init log files
exec 1>$LOGFILE 2>&1
echo "###### $PROGNAME Start: `date`\n" >$LOGFILE
echo "\n###### $PROGNAME Detail Log: `date`\n" >$DETAILS
if [ -f $LOCKFILE ]
then
echo "## LOCKFILE found - exiting : `date`" | tee -a $DETAILS
mailx -s "$PROGNAME: Lockfile Warning" $MAILADDR <$LOGFILE
MRC=1
else
date > $LOCKFILE
trap "MRC=1;echo 'TRAPPED INTERRUPT-please wait for cleanup.';echo
'Background tasks may complete without futher logging' ; cleanup ; exit
$MRC" 1 2 15
#program misbehaves if no conf file so make one if missing
[ ! -f $CONFFILE ] && touch $CONFFILE
echo "## Getting imagelist for media \"$DISK\"."
images=`bpimmedia -mediaid $DISK | awk '$1=="IMAGE" {print $4}'`
#For each backupid on the disk storage unit (NB Database lookup...)
for bpid in $images
do
echo "## Beginning processing of bpid: $bpid" >>$DETAILS
retval=`verifytape $bpid`
if [ $retval -eq 0 ]
then
#Image not on tape yet
#Lookup Volume pool info for class & schedule
info=`bpimagelist -backupid $bpid | awk '$1=="IMAGE" {print
$7":"$11}'`
class=`echo $info | cut -f1 -d:`
sched=`echo $info | cut -f2 -d:`
#Lookup pool for given policy & sched or policy & wildcard for sched
POOL=`awk '{if ($1=="'$class'" && $2=="'$sched'") {print $3} else
{if ($1=="'$class'" && $2=="*" ) {print $3}}}' $CONFFILE|tail -1`
if [ -z "$POOL" ]
then
#Failed lookup of pool from config file
echo "## WRN: could not lookup a pool for this image, using
\"$DEFAULTPOOL\"." | tee -a $DETAILS
POOL=$DEFAULTPOOL
MRC=1
fi
echo "## INF: Assigning $bpid to pool \"$POOL\"." >>$DETAILS
echo "$POOL $bpid" >>$POOLFILE
else
# Image is already on tape - must be a stray image from previous
# duplication attempt that didn't complete expire process - delete
it.
#
# Remove this section if you don't want duplicated images deleted
#
copynum=`diskcopy $bpid`
if [ $copynum -ne 0 ]
then
echo "## INF: Expiring stranded copy of $bpid." |tee -a $DETAILS
bpexpdate -d 0 -backupid $bpid -force -copy $copynum 2>>$DETAILS
if [ $? -ne 0 ]
then
echo "## ERR: bpexpdate had non-zero return code for \"$bpid\"."
| tee -a $DETAILS
MRC=1
fi
else
echo "## ERR: lookup of disk copy number failed for $bpid." | tee
-a $DETAILS
MRC=1
fi
fi
done
# All backupids assigned to volume pools - start the duplications
for POOL in `awk '{print $1}' $POOLFILE | sort -u`
do
lkup=`vmpool -listall -b | awk 'NR>2 && $1=="'$POOL'" {print $1}'`
if [ -z "$lkup" ]
then
# Pool name doesn't exist so I will create it.
# This is potentially bad but I want to avoid disk overflow.
vmpool -add $POOL "Added by $PROGNAME" ANYHOST -1 -2
echo "## WRN: Pool \"$POOL\" did not exist. Created automatically."
MRC=1
fi
#Sort out only those images for this pool
awk '$1=="'$POOL'" {print $2}' $POOLFILE | sort >$BIDFILE
echo "\n\n### Duplicating pool \"$POOL\" with `wc -l $BIDFILE|awk
'{print $1}'` images: `date`."
echo "## Starting pool \"$POOL\"." >>$DETAILS
#split the big image file into manageable portions
split -l $GLOB $BIDFILE $SPLITFILE
count=0
for smallbid in `ls $SPLITFILE??`
do
echo "## INF: Pool \"$POOL\", set $count, fileset `basename
$smallbid`." | tee -a $DETAILS
dupit $smallbid &
sleep 5
while [ `jobs | wc -l` -ge $SIMUL ]
do
sleep 30
done
count=`expr $count + 1`
done
#Wait for remaing jobs to complete
while [ `jobs | wc -l` -gt 0 ]
do
sleep 30
done
echo "## INF: Done with pool \"$POOL\": `date`."
#Go back for next pool name
done
cleanup
fi
fi
exit $MRC
-----Original Message-----
From: Richard Hellier [mailto:rlh AT lsil DOT com]
Sent: Monday, September 15, 2003 8:35 AM
To: (veritas-bu AT mailman.eng.auburn DOT edu)
Subject: [Veritas-bu] Backing up to disk (and then tape ??)
Folks,
In various recent threads, some people have mentioned using
"backup to disk" (and then (e.g.) dup'ing to tape) as a way of
accelerating backups, getting reduced backup windows etc.
If you are doing this, could I ask you, please, to let me know
some details of what sort of equipment you're using, any gotchas with
this
approach?
I'm asking because our current 1.8 Tbyte data volume looks set to
grow to 5-6 Tbyte and I don't think our current approach (ATL P3000 +
5 DLT drives) will scale to that sort of data volume.
|
<Prev in Thread] |
Current Thread |
[Next in Thread>
|
- [Veritas-bu] Backing up to disk (and then tape ??), Paul Boatman
- [Veritas-bu] Backing up to disk (and then tape ??), Steve Jenkins
- [Veritas-bu] Backing up to disk (and then tape ??), Hart, Charles
- [Veritas-bu] Backing up to disk (and then tape ??), Doug Braden
- [Veritas-bu] Backing up to disk (and then tape ??),
Donaldson, Mark <=
- [Veritas-bu] Backing up to disk (and then tape ??), Donaldson, Mark
- [Veritas-bu] Backing up to disk (and then tape ??), Steve Jenkins
- [Veritas-bu] Backing up to disk (and then tape ??), markjessup AT northwesternmutual DOT com
- [Veritas-bu] Backing up to disk (and then tape ??), Greenberg, Katherine A
|
|
|