Veritas-bu

[Veritas-bu] Netbackup Database size and speed

2000-10-17 18:57:18
Subject: [Veritas-bu] Netbackup Database size and speed
From: Jonathan Geibel Jonathan.Geibel AT disney DOT com
Date: Tue, 17 Oct 2000 15:57:18 -0700
It's currently sitting on a striped VxFS filesystem...  I'm not sure we
can really tune the actual filesystem much more.  

On Tue, 17 Oct 2000, Rob Worman wrote:

> All too often I see NBU databases that aren't tuned for performance - you'd 
> be surprised what striping your NBU DB filesystem (and the use of VXFS 
> instead of UFS, in the case of Unix master servers) can do for your overall 
> backup performance...
> 
> rob
> 
> 
> At 3:34 PM -0700 10/17/00, Jonathan Geibel wrote:
> >our current machine is a dual-processor Sun E4000 with 1GB of RAM..
> >/usr/openv is sitting on a SCSI-attached Sun RAID box (an older Maxstrat)
> >
> >we are about to upgrade all of this as soon as some new hardware comes
> >in so we'll see if that makes any difference.
> >
> >My guess is that this flatfile database just does not scale very well and
> >better hardware won't make a difference at this point.
> >
> >the problem with restores is not with how fast the actual data transfer
> >is, but just with netbackup munging through it's database trying to find
> >out which images to restore from.
> >
> >also, using bplist or any of the GUI utilities that access the database
> >are excessively slow at this point and are almost useless.
> >
> >we oftentimes find ourselves grepping through the db/images directories to
> >try and find data instead of using the netbackup utilities..
> >
> >the worst is trying to find a single filename that could exist in any of
> >our larger backups..  trying to search through everything can take many
> >hours to complete.  being able to dump this information into a real
> >database where we could do fast searches for single filenames should fix
> >this problem..
> >
> >On Tue, 17 Oct 2000, Joshua Fielden wrote:
> >
> >> 3.4 removes this artificial limitation (db == tape size), as do various 
> >> posted tricks (search veritas' web site for it). We are an EMC shop, so we 
> >> have the database backed up to disk units on machines other than the 
> >> master server. This equals databases on multiple EMC disk arrays, plus the 
> >> soon-to-be-implemented vaulting. We feel rather safe with our databases.
> >> 
> >> I'm concerned about the time it takes to read the indexes, and the size. 
> >> We're already at ~20G in just a couple of months in the images directory. 
> >> What sort of machine/load/disk do you use that it takes that long to 
> >> search, Jonathan? We have a Marathon (4-proc 420R equivalent, 2G ram) as 
> >> our master server with 3 dmp connections to an EMC 3870 with 16G cache, 
> >> and we see some pretty zippy performance in restores. I realize this is 
> >> overkill for most shops, so I'm wondering what configurations cause 
> >> performance cramp? We may be forced to downsize, if the load on the 
> >> machine stays as consistantly low as it is.
> >> 
> >> As to flat files vs databases, our legato indexes were ~70G on one of our 
> >> data zones, which did  90mm small files to tape on 'full night'. On a 
> >> 8x400 E4500 with 4.5G RAM, on EMC storage, it would take sometimes 90m to 
> >> 'changetime' in the recover command on an index of around 18G. Once you 
> >> get to a certain size, response time sucks no matter what you do, AFAIK. :)
> >> 
> >> JF
> >> 
> >> On Tue, Oct 17, 2000 at 01:22:59PM -0600, McMurphy, Tim filled up my inbox 
> >> with:
> >> > 
> >> > If anyone has any insight please post to the group! I am soon to 
> >> > re-design
> >> > and implement our netbackup structure and one of the problems is, you
> >> > guessed it. Our db is only 22 gig (small compared to some) but big 
> >> > enough to
> >> > cause me concern for the future.
> >> > 
> >> > I was thinking along the lines of having one database per year (1999, 
> >> > 2000,
> >> > 2001 etc) and trying to keep the size in line that way. Alternatively
> >> > separate a separate db for different platforms (not my fav by a long
> >> > stretch).
> >> > 
> >> > A question for you Jonathan is what do you backup your db to? I was under
> >> > the impression that the netbackup db couldn't span multiple tapes. If I 
> >> > am
> >> > wrong on that then good but if it is true then you must be near the 
> >> > limit of
> >> > any tape technology that I am aware of.
> >> > 
> >> > -----Original Message-----
> >> > From: Jonathan Geibel [mailto:Jonathan.Geibel AT disney DOT com]
> >> > Sent: Tuesday, October 17, 2000 12:41 PM
> >> > To: veritas-bu AT mailman.eng.auburn DOT edu
> >> > Subject: [Veritas-bu] Netbackup Database size and speed
> >> > 
> >> > 
> >> > Hello,
> >> > 
> >> > Question for you all:
> >> > 
> >> > Has anyone had any troubles with the netbackup database size getting out
> >> > of hand and creating excessively bad performance issues?  Has anyone come
> >> > up with any intuitive solutions for how to solve this problem?
> >> > 
> >> > With 13TBs of data being backed up every week, our netbackup db/images
> >> > directory has grown to obscene sizes (60GB of flat-file data compressed).
> >> > 
> >> > Because netbackup doesn't use a real database to hold this information
> >> > (simple flatfiles..) the scalability of this design is rather weak..
> >> > 
> >> > these huge flatfile databases are causing us major performance problems
> >> > when trying to do simple restores..  the simple act of looking for a
> >> > single file can take several hours to complete as it chugs through all of
> >> > it's backups files.
> >> > 
> >> > we've indexed everything, but that doesn't seem to help..
> >> > 
> >> > I've heard there are tools to dump this information into an actual SQL
> >> > database which could be an extremely useful tool for hunting down the
> >> > backups for specific files..
> >> > 
> >> > has anyone done this?  we were thinking of possible dumping the data once
> >> > a day into a database for doing quick searches on..
> >> > 
> >> > I'd be curious to hear if anyone else has run into these performance
> >> > problems..
> >> > 
> >> > Jon
> >> > 
> >> > 
> >> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >> > Walt Disney Feature Animation      The Secret Lab
> >> > Senior Systems Engineer              818.526.3051
> >> > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >> > 
> >> > 
> >> > 
> >> > 
> >> > 
> >> > _______________________________________________
> >> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >> > _______________________________________________
> >> > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >> 
> >> -- 
> >> "Any man page that includes the words "USE AT YOUR OWN RISK. BEWARE OF 
> >> DOG. 
> >> SLIPPERY WHEN WET" means trouble" - Michael Lucas
> >> Joshua Fielden, Senior Systems Administrator and Backups Team Lead
> >> eXcite@Home, Inc. jfielden AT excitecorp DOT com 650-556-3316
> >> _______________________________________________
> >> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >> 
> >
> >_______________________________________________
> >Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> 
> 




<Prev in Thread] Current Thread [Next in Thread>