Veritas-bu

[Veritas-bu] netbackup slow for large number of small files.

2002-05-02 15:32:49
Subject: [Veritas-bu] netbackup slow for large number of small files.
From: YLi AT ea DOT com (Li, Jackie (Yanhui))
Date: Thu, 2 May 2002 12:32:49 -0700
> David
> 
> <><><><><><><><><><><><><><><><><><><><>
> David A. Chapa
> NetBackup Consultant
> DataStaff, Inc.
> http://www.consulting.datastaff.com
> 847 413 1144
> ---------------------------------------
> NBU-LSERV AT datastaff DOT com - Adv. Scripting
> http://www.xbpadm-commands.com
> 
> -----Original Message-----
> From: veritas-bu-admin AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]On Behalf Of Daniel
> Teklu
> Sent: Friday, April 19, 2002 5:46 PM
> To: veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] EXIT 24
> 
> 
> On only a couple of client servers, I get EXIT 24 with this on my status log
> 
> 
> 23:36:30 INF - Waiting for mount of media id M00020 on server mndt-sess1.
> 23:37:27 INF - Waiting for positioning of media id M00020 on server
> mndt-sess1.
> 23:37:35 ERR - Cannot write to STDOUT. Errno = 32: Broken pipe
> 23:38:45 INF - Backup by root on client mnapp3: socket write failed.
> 
> 
> Any ideas on this? My master server and clients are all Solaris?
> 
> Thanks
> 
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> 
> --__--__--
> 
> Message: 2
> Date: Tue, 23 Apr 2002 00:34:09 +0200
> To: "Donaldson, Mark" <Mark.Donaldson AT experianems DOT com>
> From: "Peter L. Buschman" <plb AT iotk DOT com>
> Subject: Re: [Veritas-bu] Homemade HSM
> Cc: Veritas-bu AT mailman.eng.auburn DOT edu
> 
> --=====================_89748882==_.ALT
> Content-Type: text/plain; charset="us-ascii"; format=flowed
> 
> 
> Mark:
> 
> I've used this approach many times.  The only caveat is that you need to 
> build enough intelligence into
> your script *not* to expire images that did not successfully duplicate.
> 
> I have some non-critical backups of a slow machine that I send to disk 
> because it isn't fast enough
> to stream the drive.  This is my quick-and-dirty destage script, but it 
> should give you a basic idea
> of how such a script should work (note that this script is not intelligent 
> per my above definition... you'll
> need to modify it to suit.)
> 
> I run this from cron once a day.  It also means my drive is only needed 
> once a day even though backups
> are once an hour... hence less wear and tear. :)
> 
> --PLB
> 
> #!/bin/ksh
> PATH=/usr/bin\
> :/sbin\
> :/usr/sbin\
> :/usr/ccs/bin\
> :/usr/ucb\
> :/usr/local/bin\
> :/opt/sfw/bin\
> :/usr/openwin/bin\
> :/usr/dt/bin\
> :/opt/openv/netbackup/bin\
> :/opt/openv/netbackup/bin/goodies\
> :/opt/openv/netbackup/bin/admincmd\
> :/opt/openv/netbackup/vault/scripts\
> :/opt/openv/volmgr/bin\
> :/opt/openv/volmgr/bin/goodies\
> :.
> 
> SPOOLDIR=/iotk/spool
> BIDFILE=/tmp/destage.$$
> EXPIREFILE=/tmp/expire.$$
> DSTUNIT=L280
> DSTPOOL=NetBackup
> OUTPUTFILE=/tmp/bpduplicate.$$
> ls $SPOOLDIR | sed 's/_C1.*//' | sort | uniq | tee $BIDFILE $EXPIREFILE
> bpduplicate -dstunit $DSTUNIT -v -Bidfile $BIDFILE -L $OUTPUTFILE -dp $DSTPOOL
> for image in $(cat $EXPIREFILE)
> do
>    echo "Expiring $image"
>    bpexpdate -backupid $image -d 0 -copy 1 -force
> done
> rm $EXPIREFILE
> 
> At 03:12 PM 04/22/2002 -0600, you wrote:
> 
> >We had a library failure over the weekend and it underscored the degree to 
> >which our library is a single point of failure for production 
> >processes.  (Oracle redo logs are archived to tape via netbackup: when the 
> >library fails, the archive fails, the DB filesystems fill up and the DB 
> >crashes.  Nice, huh?)
> >
> >Anyway, to compensate, I want to build a disk-based storage unit to be 
> >used for critical files.  It'll only be a couple hundred of gig for our 
> >most important stuff (like archived redo logs).
> >
> >I have this idea that I can backup to the disk, then periodically do a 
> >bpduplicate to tape, rename the duplicate copy to be the primary, then > 
> >delete the disk image.  When the library is out of service, I'll let the 
> >images accumulate on disk until it's available again.
> >
> >Has anybody tried this?  Any caveats or experiences you'd like to share?
> >
> >-Mark
> >
> >-----------------------------------------------------------------------
> >    Mark Donaldson - Sr. Systems Engineer
> >    Experian EMS - Denver Colorado
> >-----------------------------------------------------------------------
> >    Linux?  Wasn't he the kid with the blanket?
> >-----------------------------------------------------------------------
> 
> --=====================_89748882==_.ALT
> Content-Type: text/html; charset="us-ascii"
> 
> <html>
> <br>
> Mark:<br>
> <br>
> I've used this approach many times.&nbsp; The only caveat is that you
> need to build enough intelligence into<br>
> your script *not* to expire images that did not successfully
> duplicate.<br>
> <br>
> I have some non-critical backups of a slow machine that I send to disk
> because it isn't fast enough<br>
> to stream the drive.&nbsp; This is my quick-and-dirty destage script, but
> it should give you a basic idea<br>
> of how such a script should work (note that this script is not
> intelligent per my above definition... you'll<br>
> need to modify it to suit.)<br>
> <br>
> I run this from cron once a day.&nbsp; It also means my drive is only
> needed once a day even though backups<br>
> are once an hour... hence less wear and tear. :)<br>
> <br>
> --PLB<br>
> <br>
> #!/bin/ksh<br>
> PATH=/usr/bin\<br>
> :/sbin\<br>
> :/usr/sbin\<br>
> :/usr/ccs/bin\<br>
> :/usr/ucb\<br>
> :/usr/local/bin\<br>
> :/opt/sfw/bin\<br>
> :/usr/openwin/bin\<br>
> :/usr/dt/bin\<br>
> :/opt/openv/netbackup/bin\<br>
> :/opt/openv/netbackup/bin/goodies\<br>
> :/opt/openv/netbackup/bin/admincmd\<br>
> :/opt/openv/netbackup/vault/scripts\<br>
> :/opt/openv/volmgr/bin\<br>
> :/opt/openv/volmgr/bin/goodies\<br>
> :.<br>
> <br>
> SPOOLDIR=/iotk/spool<br>
> BIDFILE=/tmp/destage.$$<br>
> EXPIREFILE=/tmp/expire.$$<br>
> DSTUNIT=L280<br>
> DSTPOOL=NetBackup<br>
> OUTPUTFILE=/tmp/bpduplicate.$$<br>
> ls $SPOOLDIR | sed 's/_C1.*//' | sort | uniq | tee $BIDFILE
> $EXPIREFILE<br>
> bpduplicate -dstunit $DSTUNIT -v -Bidfile $BIDFILE -L $OUTPUTFILE -dp
> $DSTPOOL<br>
> for image in $(cat $EXPIREFILE)<br>
> do<br>
> &nbsp; echo &quot;Expiring $image&quot;<br>
> &nbsp; bpexpdate -backupid $image -d 0 -copy 1 -force<br>
> done<br>
> rm $EXPIREFILE<br>
> <br>
> At 03:12 PM 04/22/2002 -0600, you wrote:<br>
> <br>
> <blockquote type=cite class=cite cite><font size=2>We had a library
> failure over the weekend and it underscored the degree to which our
> library is a single point of failure for production processes.&nbsp;
> (Oracle redo logs are archived to tape via netbackup: when the library
> fails, the archive fails, the DB filesystems fill up and the DB
> crashes.&nbsp; Nice, huh?)<br>
> </font><br>
> <font size=2>Anyway, to compensate, I want to build a disk-based storage
> unit to be used for critical files.&nbsp; It'll only be a couple hundred
> of gig for our most important stuff (like archived redo logs).&nbsp;
> <br>
> </font><br>
> <font size=2>I have this idea that I can backup to the disk, then
> periodically do a bpduplicate to tape, rename the duplicate copy to be
> the primary, then delete the disk image.&nbsp; When the library is out of
> service, I'll let the images accumulate on disk until it's available
> again.<br>
> </font><br>
> <font size=2>Has anybody tried this?&nbsp; Any caveats or experiences
> you'd like to share?</font> <br>
> <br>
> <font size=2>-Mark</font> <br>
> <br>
> <font 
> size=2>-----------------------------------------------------------------------</font>
> <br>
> <font size=2>&nbsp;&nbsp; Mark Donaldson - Sr. Systems Engineer</font> <br>
> <font size=2>&nbsp;&nbsp; Experian EMS - Denver Colorado</font> <br>
> <font 
> size=2>-----------------------------------------------------------------------&nbsp;
>  </font><br>> 
> <font size=2>&nbsp;&nbsp; Linux?&nbsp; Wasn't he the kid with the 
> blanket?</font> <br>
> <font 
> size=2>-----------------------------------------------------------------------</font>
>  <br>
> </blockquote></html>
> 
> --=====================_89748882==_.ALT--
> 
> 
> --__--__--
> 
> Message: 3
> Date: Mon, 22 Apr 2002 21:47:03 -0400 (EDT)
> From: Michael Suen <suen AT cosmos.buffalo DOT edu>
> To: Veritas-bu AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] NBU Oracle Agent / RMAN
> 
> 
> Hello EveryOne
> 
>       I have a Question about the Retention and stuff.. here is what I
>       understand (I think)
> 
>       1) The Retention Setting in the "Default-Policy" is the
>               REAL/Actual retention of the NBU Images
>       2) The Retention Setting in the Schedules (Automatic,
>               Differential, Cumulative) is just used to find out "When" 
> should the
>               backup being run.
>       3) Expiring images under NBU does NOT expire the reference under
>               RMAN
> 
>       O.K.. with that...
> 
>       Since I have to use RMAN to do the expire, Can I set the Retention
>       under "Default-Policy" to be INFINITY
> 
>       AND the Retention Setting under the Schedules (Auto,Diff) ALSO to 
>       be INFINITY ???
> 
>       Then I just use RMAN to do ALL of the expire??
> 
>       BTW, I have NBU 3.4.2 on Solaris
> 
> 
>       Is that make sense or I am missing somewhere along the line??
> 
> 
>       Thank you very much for your help...
> 
> 
>       thanks
> 
>                       Mike
> 
>       
> 
> 
> --__--__--
> 
> Message: 4
> Date: Tue, 23 Apr 2002 07:27:41 +0200
> From: "Bertie Booysens" <Bertie.Booysens AT mgxgroup DOT com>
> To: <veritas-bu AT mailman.eng.auburn DOT edu>
> Subject: [Veritas-bu] Error 52 with SSO
> 
> This is a multi-part message in MIME format.
> 
> ------_=_NextPart_001_01C1EA87.96B9C022
> Content-Type: multipart/alternative;
>       boundary="----_=_NextPart_002_01C1EA87.96B9C022"
> 
> 
> ------_=_NextPart_002_01C1EA87.96B9C022
> Content-Type: text/plain;
>       charset="iso-8859-1"
> Content-Transfer-Encoding: quoted-printable
> 
> I have discovered a strange scenario in my SSO environment and will =
> appreciate your input. This is currently happening at three sites =
> running netbackup in an SSO environment. I have logged a call with =
> Veritas but had no positive feedback as yet. Netbackup running on a SUN =
> >     AND the Retention Setting under the Schedules (Auto,Diff) ALSO to 
> >     be INFINITY ???
> > 
> >     Then I just use RMAN to do ALL of the expire??
> > 
> >     BTW, I have NBU 3.4.2 on Solaris
> > 
> > 
> >     Is that make sense or I am missing somewhere along the line??
> > 
> > 
> >     Thank you very much for your help...
> > 
> > 
> >     thanks
> > 
> >                     Mike
> > 
> >     
> > 
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> 
> 
> 
> --__--__--
> 
> Message: 6
> Date: Mon, 22 Apr 2002 23:28:15 -0700
> From: jago <javier.graf AT usa DOT net>
> To: <dignacio AT sanstarinc DOT com>,
>         "Veritas NBU" <veritas-bu AT mailman.eng.auburn DOT edu>
> Subject: Re: [[Veritas-bu] Restoring Expired Images]
> 
> Hi,
> 
> If the original media tape is available, you have a chance.
> 
> First you must import the media:
> bpimport: -create_db_info -id <media_id> [-server <name>] [-L <output_file>
> can import the images you want with the second step:
> bpimport: [-l] [-p] [-pb] [-PD] [-PM] [-v] [-local] [-client <name>]
>     [-M master_server]
>     [-Bidfile <file_name>]
>     [-backup_copy <backup_copy_value>]
>     [-st <sched_type>] [-sl <sched_label>] [-L <output_file> [-en]]
>     [-policy <name>] [-s <startdate>] [-e <enddate>]
>     [-pt <policy_type>] [-hoursago <hours>] [-cn <copy number>]
>     [-backupid <backup_id>] [-id <media_id>]
> 
> Javier Gonzalez
> T4B Group AG
> www.t4b.ch
> 
> Then you 
> 
> "Don U. Ignacio" <dignacio AT sanstarinc DOT com> wrote:
> > Hi All,
> > 
> > Would it be possible to restore data from expired
> > images within NBU? If possible, what's the procedure?> 
> > 
> > 
> > Thanks,
> > 
> > Don U. Ignacio
> > Systems Engineer
> > SANstar, Inc.
> > Fax: (781) 383-8903
> > Mobile: (781) 640-5028
> > Email: dignacio AT sanstarinc DOT com
> > 
> > "Specialists in Storage Area Networks" 
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> 
> 
> 
> --__--__--
> 
> Message: 7
> Date: Tue, 23 Apr 2002 02:31:28 -0400 (EDT)
> From: Michael Suen <suen AT cosmos.buffalo DOT edu>
> To: jago <javier.graf AT usa DOT net>
> cc: Veritas-bu AT mailman.eng.auburn DOT edu
> Subject: Re: [[Veritas-bu] NBU Oracle Agent / RMAN]
> 
> Hello Jago
> 
>       Thanks for your reply...
>       hmhmmmm
>       
>       You are saying that the "Def-policy" is like the "User Backup" in
>       the standard class?
>       
>       -----------------------------------------------------------
>       I am reading the SAG for Oracle Agent...
>       (I printed the March 2000, 100-001474) Version
>       On page 54.. 
>       "All Oracle database operations are performed through NetBackup
>       for Oracle on Unix using a BACKUP POLICY schedule. This includes those
>       backups started automatically."
> 
>       Also..on page 56
>       Retention:
>       The retention period for an Auto Full Backup... controls how long
>       Netbackup keeps records of when a schedule backups have occurred. Note
>       that this is different than with a Backup Policy schedule.
>       
> 
>       thx
> 
>       Mike
> 
> 
> 
> On Mon, 22 Apr 2002, jago wrote:
> 
> > Hi there,
> > 
> > Michel I think you are confuse about the retention level definitions, take a
> > look at the SAG.  On your Policy Manager you can can define either "user,
> > full, diff, incr, or Default Policy" schedules depending what type of Backup
> > are you planing to do. The retention level apply to each of these schedules,
> > in your case if you use Def-Pol (only available for certain class types) for
> > your RMAN backups (it's a kind of user backup) started from the client, the
> > retension apply to all your images done with this schedule, but RMAN also
> > keeps this info in his own catalog db (recovery catalog), and that has 
> > nothing
> > to do with the NBU catalog.
> > 
> > SY
> > 
> > Javier Gonzalez
> > T4B Group AG
> > www.t4b.ch
> > 
> >  Michael Suen <suen AT cosmos.buffalo DOT edu> wrote:
> > > 
> > > Hello EveryOne
> > > 
> > >   I have a Question about the Retention and stuff.. here is what I
> > >   understand (I think)
> > > 
> > >   1) The Retention Setting in the "Default-Policy" is the
> > >           REAL/Actual retention of the NBU Images
> > >   2) The Retention Setting in the Schedules (Automatic,
> > >           Differential, Cumulative) is just used to find out "When" 
> > > should the
> > >           backup being run.
> > >   3) Expiring images under NBU does NOT expire the reference under
> > >           RMAN
> > > 
> > >   O.K.. with that...
> > > 
> > >   Since I have to use RMAN to do the expire, Can I set the Retention
> > >   under "Default-Policy" to be INFINITY
> > > 
> > >   AND the Retention Setting under the Schedules (Auto,Diff) ALSO to 
> > >   be INFINITY ???
> > > 
> > >   Then I just use RMAN to do ALL of the expire??
> > > 
> > >   BTW, I have NBU 3.4.2 on Solaris
> > > 
> > > 
> > >   Is that make sense or I am missing somewhere along the line??
> > > 
> > > 
> > >   Thank you very much for your help...
> > > 
> > > 
> > >   thanks
> > > 
> > >                   Mike
> > > 
        Netbackup 3.4.2

        We have some clients has a large number of small files (300K), the size 
of each file is
        about 500 bytes, netbakcup is very slow to backup these small 
files(more that 5 hours),
        On legato netbackup, it is less than 2 housrs.

        Does anybody has similar situation and how to solve it?


        Thanks
>  
>  > 

<Prev in Thread] Current Thread [Next in Thread>
  • [Veritas-bu] netbackup slow for large number of small files., Li, Jackie (Yanhui) <=