I'll third that but there is something to watch out for when using
Flashbackup. The inode scan, as Paul mentions below, can also be a
bottleneck.
Flashbackup will use a temporary directory to store this inode information
and that directory itself can become a bottleneck (or more precisely the
filesystem it resides on). /usr/openv/netbackup/BPFSMAP_TMPDIR is the
directory in question -- what I did is actually make this a sym link to a
seperate filesystem (/flashbkp). In my case this separate filesystem is 5
striped EMC disks. Before I did this I wasn't having great performance (12+
hours to backup 4 50GB filesystems) but once I did backup times were
reduced to under 4 hours.
Hopefully you have access to some extra disk you can do this with... play
around with the number of disks in your stripe set to get the optimal
performance.
-Dave
-----Original Message-----
From: paulg AT CDCNA DOT COM [mailto:paulg AT CDCNA DOT COM]
Sent: Wednesday, April 23, 2003 08:36
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Backup Millions of small files with Solaris
client & server
I would definitely second that. I have a couple of filesystems with 12
million
tiny files (who needs databases, righ? :-( ) and the backup time on it is
comparable to a regular filesystem. With FlashBackup you loose a little on
the
initial inode scan, but the rest of it is pure raw dump performance.
paulg
On Apr 23, 7:48am, Rob Worman wrote:
> Subject: Re: [Veritas-bu] Backup Millions of small files with Solaris clie
> JV-
>
> You are talking about the classic "lots of small files" scenario.
> 4M files in a single directory, ouch!
>
> The best and single answer to your problem is "back up the raw
> partition, thereby avoiding the OS inode overhead".
>
> While it's possible to do raw partition backups using vanilla
> NetBackup, this choice has several big caveats:
>
> -backup size is always the size of the partition, even if it's empty
> -can only restore the WHOLE partition, not an individual file or dir.
> -can only do full backups
> -the filesystem should be quiescent, if not unmounted
>
> But you could at least get a feel for the performance improvements
> you'd see when backing up raw - a substantial one, I'd bet - by
> making a test class and seeing how long it takes to back up your data
> as /devices/pci@blahblahblah. The NBU SysAdmin guide has more info
> on this, look up "raw partition backups" in the index.
>
> The answer most people in your situation have chosen is the NetBackup
> FlashBackup option. It's purpose is to provide the speed of raw
> partition backups but without all of the caveats listed above. I
> recommend you check it out, it can make the difference between a
> 24-hour full and a 4-hour full. :-)
>
> HTH
> rob
>
> At 9:15 PM -0700 4/22/03, JV wrote:
> >Dear Veritas Netbackup pros:
> >
> >I have a gigabit ethernet client with special backup needs, I'm
> >asking if you can give any pointers if you've been down this road:
> >
> >Netbackup client: Sun e250, Solaris 2.6 with a single gigabit
> >ethernet adapter on a private switched storage network running
> >SUNWged 2.0 driver, 2x400 Mhz cpus, 2 GB RAM. Storage is two A1000
> >arrays, they have a hardware RAID 5 controller with ufs filesystems,
> >ten mount points of 35 GB each, with 4 million files each (total
> >storage 350 GB, 40 million files, on 20 x 18 GB SCSI disks, 2
> >controllers on the A1000s). No client-side compression, HW
> >compression on the LTO drives. I can only do FULL backups once per
> >week and sneak in an incremental M-F at offpeak times. Netbackup
> >master is E450 running Solaris 8, Netbackup 4.5_M3, on 5 LTO tape
> >drives on Brocade 1Gbps switch.
> >
> >As you can imagine, the tape drives are starving for data with a
> >pathetic throughput rate. I have set the policy to 2 active jobs per
> >policy, two storage units (LTO tape drives). The problem is READING
> >the A1000 arrays, the Solaris host mildly pages (2000 pg/sec) when
> >running this backup according to vmstat.
> >
> >Playing games with the filecard mask in an include list [ie
> >/data01/*] doesn't look hopeful, there aren't any subdirectories -
> >all 4M files are 1 level deep in the mount point [ie
> >/vault01/file_aaaaaaaaaaaaa*]. An "ls -l a*" won't return in a reason
> >span of time.
> >
> >
> >What can I do to the configuration of Netbackup to improve the
> >throughput?!?!
> >I have seen several threads in the archives here that reference Win2K
> >setups, what are the unix ppl out there doing?
> >
> >Thanks
> >JV
> >
> >__________________________________________________
> >Do you Yahoo!?
> >The New Yahoo! Search - Faster. Easier. Bingo
> >http://search.yahoo.com
> >_______________________________________________
> >Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> >http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>-- End of excerpt from Rob Worman
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
