Valid points.
A multistream backup can have the streams start out of order (by number), in
sequence, or nearly in parallel. All possiblities have to be accounted for.
The bpend equivalent code has to be run by the *last* stream to complete.
Streams that don't complete normally may leave this script in a bad cleanup
situation. This wasn't usually the case, though. Checking the logfile from
this process is a good daily task, though.
As far as restarted streams go, I handled this simply by only allowing one
attempt at backup. It was too difficult to handle it otherwise.
The STREAM_COUNT variable came in v3.4.
Here's my bpstart script (it's pretty heavily commented so read carefully).
It's an "evolved" script so it could probably be written better but we've
moved to an RMAN backup and I have little need to maintain this script. The
bpend is similar and follows the first script.
-Mark
$ cat bpstart_notify.Oracle
#!/bin/ksh
PATH=$PATH:/usr/sbin:/usr/local/bin:/usr/openv/netbackup/bin
PATH=$PATH:/usr/openv/netbackup/bin/admincmd:/usr/openv/volmgr/bin:/usr/open
v/local
export PATH
PROGNAME="`basename $0`"
LOGNAME="nb.$2.$3.$4"
LOGFILE=/usr/openv/netbackup/logs/$LOGNAME.log
ENDFILE=/tmp/$LOGNAME.END
PROCSTART=/tmp/$PROGNAME.S
PROCDONE=/tmp/$PROGNAME.D
PROCFIRST=/tmp/$PROGNAME.F
exec 1>>$LOGFILE 2>&1
## Function definitions
bigprocess () {
#
# The entire main module is generic to any multistream process
##
## Call DB preparation here.
##
}
cleanup () {
#Trap function
echo "Stream $STREAM_NUMBER: Trapped Interrupt `date`"
[ -f $PROCFIRST ] && rm $PROCFIRST
echo "Stream $STREAM_NUMBER: TERMINATED" >>$PROCDONE
if [ `wc -l $PROCDONE | awk '{print $1}'` -ge $STREAM_COUNT ]
then
[ -f $PROCSTART ] && rm $PROCSTART
[ -f $PROCDONE ] && rm $PROCDONE
echo "Stream $STREAM_NUMBER: Tracking files Removed."
fi
}
## Main
trap "cleanup;exit 1" 1 2 3 4 5 6 7 8 10 11 12 13 14 15
nbsched=$3
echo "Stream $STREAM_NUMBER: $PROGNAME Beginning: `date`"
if [ $STREAM_NUMBER -eq 0 ]
then
#Single-streamed backup
echo "Single-streamed Backup: $BACKUPID"
bigprocess
else
#
#If PROCSTART is present with no PROCDONE, then first stream is preparing
# and other streams have yet to start
#
#If PROCDONE is present with no PROCSTART then it is an error
#
#If PROCDONE and PROCSTART are both there, then backups are still
preparing
#
#If PROCDONE and PROCSTART are both missing then preparations are complete
# or have not begun
#
#If PROCDONE has less lines than number of streams, then each streams
# start script has not completed
#
echo "Stream $STREAM_NUMBER: Start `date`"
echo "Stream $STREAM_NUMBER environment variables:"
env | sed "s/^/Stream $STREAM_NUMBER: /"
if [ -f $PROCFIRST -o -f $PROCSTART ]
then
#This stream is definately not first
echo "Stream $STREAM_NUMBER: not first stream."
[ -f $PROCFIRST ] && echo "Stream $STREAM_NUMBER: File $PROCFIRST
found."
[ -f $PROCSTART ] && echo "Stream $STREAM_NUMBER: File $PROCSTART
found."
echo "Stream $STREAM_NUMBER: Init Sleep"
sleep 30
else
#This stream may be first - it may also race with another.
echo "$STREAM_NUMBER " >>$PROCFIRST
echo "Stream $STREAM_NUMBER: Possible first start"
if [ `awk 'NR==1 {print $1}' $PROCFIRST` -ne $STREAM_NUMBER ]
then
#This stream didn't win the race.
echo "Stream $STREAM_NUMBER: Init Sleep - RACE RESOLUTION!"
sleep 30
fi
fi
if [ ! -f $PROCSTART ]
then
echo "Stream $STREAM_NUMBER: Beginning first stream preparations
`date`."
#First Stream preparations
[ -f $PROCDONE -a ! -f $PROCSTART ] && rm $PROCDONE && \
echo "Stream $STREAM_NUMBER: Cleared stray $PROCDONE file."
echo "Stream $STREAM_NUMBER: `date` Begin." > $PROCSTART
echo "Stream $STREAM_NUMBER: Created file $PROCSTART."
[ -f $PROCFIRST ] && rm $PROCFIRST && \
echo "Stream $STREAM_NUMBER: $PROCFIRST removed."
#First stream to start - responsible for preparing backup
[ -f $PROCDONE ] && rm $PROCDONE && \
echo "Stream $STREAM_NUMBER: Removed old $PROCDONE file."
echo "Stream $STREAM_NUMBER: Beginning first stream preparations
`date`."
bigprocess
echo "Stream $STREAM_NUMBER: proceeding $BACKUPID `date`"
echo "Stream $STREAM_NUMBER: `date` Done " > $PROCDONE
else
# Any other stream other than first should come here
echo "Stream $STREAM_NUMBER: `date` Begin." >>$PROCSTART
count=0
while [ ! -f $PROCDONE -a $count -lt 50 ]
do
# Loop until first stream completes or too many loops
echo "Stream $STREAM_NUMBER: Wait $count `date`"
sleep 50
# slight randomizing sleep to try to prevent any race conditions on
exit.
sleep `expr $RANDOM / 3276 + 1`
count=`expr $count + 1`
done
echo "Stream $STREAM_NUMBER: `date` Done." >>$PROCDONE
if [ `wc -l $PROCDONE | awk '{print $1}'` -ge $STREAM_COUNT ]
then
echo "Stream $STREAM_NUMBER: Reset Files `date`"
rm $PROCSTART && echo "Stream $STREAM_NUMBER: Removed $PROCSTART"
rm $PROCDONE && echo "Stream $STREAM_NUMBER: Removed $PROCDONE"
fi
echo "Stream $STREAM_NUMBER: proceeding $BACKUPID `date`"
fi
fi
echo "Stream $STREAM_NUMBER: $PROGNAME Done: `date`"
exit
############## END bpstart ##############
$ cat bpend_notify.Oracle_mmprod1
#!/bin/ksh
#[ $2 = "Oracle_mmprod1" ] && exit 0
NB_CLASS=$2
LOGNAME="nb.$2.$3.$4"
LOGFILE=/usr/openv/netbackup/logs/$LOGNAME.log
ENDFILE=/tmp/$LOGNAME.END
PATH=$PATH:/usr/sbin:/usr/local/bin:/usr/openv/netbackup/bin:/usr/openv/netb
ackup/bin/admincmd
PATH=$PATH:/usr/openv/volmgr/bin:/usr/openv/local
exec 1>>$LOGFILE 2>&1
endprocess () {
echo "End Process begin `date`"
##
## Insert DB end process here
##
echo "End Process finish `date`"
RCString="`cat $ENDFILE| tr '\n' ':'`"
/usr/bin/mailx -s "$NB_CLASS Backup Results: ${RCString}"
ops-backup AT mailhub.exactis DOT com <$LOGFILE
[ -f $ENDFILE ] && rm $ENDFILE
[ -f $LOGFILE ] && mv $LOGFILE $LOGFILE.`date +%a`
}
## Main
nbsched=$3
if [ $STREAM_NUMBER -eq 0 ]
then
#Single-streamed
echo "$5" >>$ENDFILE
endprocess
else
#Multi-streamed
#Echo streams RC to ENDFILE
echo "$5" >>$ENDFILE
numdone=`wc -l $ENDFILE | awk '{print $1}'`
if [ $numdone -ge $STREAM_COUNT ]
then
echo "Stream $STREAM_NUMBER: Stream $numdone of $STREAM_COUNT with
RC=$5. - Exiting `date`"
echo "Stream $STREAM_NUMBER: Beginning End process."
endprocess
echo "Stream $STREAM_NUMBER: Finishing End Process."
else
echo "Stream $STREAM_NUMBER: Stream $numdone of $STREAM_COUNT with
RC=$5. - Exiting `date`"
fi
fi
exit
-----Original Message-----
From: scott.kendall AT abbott DOT com [mailto:scott.kendall AT abbott DOT com]
Sent: Thursday, September 18, 2003 2:24 PM
To: ida3248b AT post.cybercity DOT dk
Cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] multi bpstart/bpend scripts test (NBU3.4) example
for those that take this and run with it, a few things to think about...
1.) you're usually running the scripts to do something, like take down a
database. you have to make sure the database is down before ANY of the
streams start their backups to get a useful backup. you have to have
subsequent streams know that the first stream is still trying to take down
the database and wait for it to finish before they start, otherwise,
depending on where you put the logic to take down the database, you'll
either try to stop the database multiple times or you'll start subsequent
streams before the database is completely down and not get a good backup.
2.) what happens if a stream dies and becomes requeued? you might want to
have the bpend script only rm the temp file if the backup completed ok.
otherwise, bpend deletes the temp file and then when the last active job
completes, it sees no more temp files and starts the database back up. then
when the requeud job goes active it will shutdown the database, backup the
data, and start the database... but it won't be consistent with the other
streams.
3.) and for that matter, what happens if all the jobs are queued and they
only go active one at a time, or some finish before others go active
(because of drives not available, mpx settings, jobs per client setting)?
the database could go up and down several times before all streams are
finished. the script will probably need to use the variable for total
streams (STREAM_COUNT) somehow (maybe to write another temp file during the
bpend and make sure you have as many of these as total streams before
bringing up the database)... only problem is I think this variable was added
in 4.5.
4.) what happens if multi-stream backups from different Policies run at the
same time and they both need to call scripts with this type of logic? the
temp files are going to cause issues between the two Policies backups.
you're either going to have to put the temp files in unique directories or
name the files in a unique way (maybe including the Policy name or
something).
5.) what happens if the system crashes or there are NetBackup problems
during this backup. if you rely on something like an rm to clean up things
during the bpend script only, you could get in a situation where the next
backup runs but because the temp files are still there from the previous
backup's bpstarts (that were never cleaned up), you would not shut down the
database and would get a backup thinking it was ok.
- Scott
ida3248b AT post.cybercity DOT dk
Sent by: veritas-bu-admin AT mailman.eng.auburn DOT edu
09/18/2003 01:04 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
cc:
Subject: [Veritas-bu] multi bpstart/bpend scripts test
(NBU3.4) example
Hello All
As there have been a lot of interest in these scripts, I decided to mail
them to the list.
These scripts are NOT tested with any kind of application/database, our
customer decided to pay for a Netbackup-Extension after all.
But I think they should could get people started.
Regards
Michael
bpstart_notify
#
# Script for of test multi stream bpnotify scripts
#
# Michael Graff Andersen
#
LOCKYES=`ls /tmp/.lock.*`
if [ -z "$LOCKYES" ]; then
echo "BEGIN BACKUP" >> /tmp/bpstart.log
fi
touch /tmp/.lock.$STREAM_NUMBER
echo $STREAM_NUMBER >> /tmp/bpstart.log
bpend_notify
#!/bin/ksh
#
# Script for of test multi stream bpnotify scripts
#
# Michael Graff Andersen
#
echo $STREAM_NUMBER >> /tmp/bpend.log
rm /tmp/.lock.$STREAM_NUMBER
LOCKYES=`ls /tmp/.lock.*`
if [ -z "$LOCKYES" ]; then
echo "END BACKUP" >> /tmp/bpend.log
fi
--
Cybercity Webhosting (http://www.cybercity.dk)
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
