We do our NetApp backups (old GF940 metro cluster) with a combination of
snapshots (which are available at both sites) and NDMP here. Our NDMP obviously
holds all the long retention data.
The data size isn't huge compared to some (6.5TB, ~40M files), but we've had
to resort to multiple policies and multi-streaming to back it up in a
reasonable amount of time (less than 60 hours for a full on the wknd). The way
it works is:
PolicyA (vol1): explicitly lists 37 different paths for the backup selection
list, we've empirically determined these are the "smaller" folders
PolicyB (vol1-long): explicitly lists 16 paths for the backup selection list,
we've determined these are the "large" folders
PolicyC (vol1-Catch-missed-directory): we've mounted the root of vol1 on a
Linux host, and we back it up via NFS, excluding the 37+16 paths defined
above. If this policy's full backup every gets too large (over 10GB or so),
we review the contents and add new paths to PolicyA or PolicyB as necessary.
This is necessary because you can't specify wildcards on NDMP backups
(discussed in this forum several times I believe).
We repeat the above process for vol2.
The above backup data sits on a deduped VTL for 2 months, and then the data
that is held longer than that is duplicated to tape for long-term storage and
expired off the VTL. As we run 5 streams concurrently, the throughput is
decent, but we peg out the CPU on the NetApp frequently during the backup
window, which is a concern.
HTH...
Jon
-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Jeff
Cleverley
Sent: June 14, 2010 5:57 PM
To: Rusty.Major AT sungard DOT com
Cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] NDMP and Millions of Files
Rusty,
If you have a way to use Snapvault to backup to another location, I
would use it. We have a number of file systems like what you have. I
tried NDMP over TCP and NFS backups using dedicated snapshots mounted
on a client. Both used a dedicated 10G network. We basically overran
our 6030 filer. We could have jumped through a lot of hoops and split
backups over multiple weekends, etc, but we decided it wasn't worth
it. I haven't tried Flash Backup for a while but it didn't really buy
us much on what we tried to do with it. It may work better now.
We backup everything (~200 TB) to NearStores in another building. We
use SnapVault instead of SnapMirror. We can still revert our
destination volumes to primary r/w file systems if we need to. We
don't have the requirement to send tapes off site. If you do, you
could still make the tapes from your secondary filer.
Using NFS to tape will give you check points and you can run multiple
streams to each tape drive. If you are not hitting the filer
throughput limits this may work for you.
Jeff
On Mon, Jun 14, 2010 at 3:24 PM, <Rusty.Major AT sungard DOT com> wrote:
>
> We have a NetApp filer that has a few TB of data made up largely of millions
> of small files (about 30 million or so) and we are using several NDMP
> policies to back up this data. The two main problems are length of time it
> takes to backup (we usually have 2-3 backups running all day every day) and
> when there is a maintenance or other event in the NBU domain, we have to
> kill the job, resulting in having to start all over (no checkpoints).
>
> For those of you who have faced a similar situation, how are you backing up
> this data?
>
> Current thoughts are moving away from NDMP and going with just snapshots and
> then getting the snap offsite either by backing it up or replicating it.
> We've also thought about backing it up via NFS, but that will probably be
> slower, though we would get checkpoints.
>
> I appreciate any other suggestions anyone has.
>
> Rusty Major, MCSE, BCFP, VCS ▪ Sr. Storage Engineer ▪ SunGard Availability
> Services ▪ 757 N. Eldridge Suite 200, Houston TX 77079 ▪ 281-584-4693
> Keeping People and Information Connected® ▪ http://availability.sungard.com/
> P Think before you print
> CONFIDENTIALITY: This e-mail (including any attachments) may contain
> confidential, proprietary and privileged information, and unauthorized
> disclosure or use is prohibited. If you received this e-mail in error,
> please notify the sender and delete this e-mail from your system.
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
--
Jeff Cleverley
Unix Systems Administrator
4380 Ziegler Road
Fort Collins, Colorado 80525
970-288-4611
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
====================================================================================
La version française suit le texte anglais.
------------------------------------------------------------------------------------
This email may contain privileged and/or confidential information, and the Bank
of
Canada does not waive any related rights. Any distribution, use, or copying of
this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately
from
your system and notify the sender promptly by email that you have done so.
------------------------------------------------------------------------------------
Le présent courriel peut contenir de l'information privilégiée ou
confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute
diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous
recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans
délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de
votre
ordinateur toute copie du courriel reçu.
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
