Veritas-bu

[Veritas-bu] Backing up a large amount of small files

2002-11-02 15:23:59
Subject: [Veritas-bu] Backing up a large amount of small files
From: joe AT joe DOT net (Johnny Oestergaard)
Date: Sat, 02 Nov 2002 21:23:59 +0100
I could live with only a daily backup for DR on the volume with all our 
scanned documents (lots and lots of small files) and then do a normal full 
backup every weekend and new files every day.

I haven't found out how to do the "track by track" backup, but I know that 
flash backup is not supported at the moment on StorageTek SVA9500 and also 
not on Windows.

If I get a real nasty problem I could also just restore the volume to a 
different LUN and then move the file I need back to the production LUN. The 
SVA9500 (and V960 and V2X) is a great platform for doing this kind of stuff.

Since we only need a volume of less the 100 GB for our image files a total 
restore wouldn't take that long. Compression on that kind of volume with a 
lot of free space should be good so restoring 30-60 MB/s should be 
something we should see (I would hope)

Do you remember where in the policy you tell that you want a raw backup and 
not a normal file backup?
We run 4.5

/johnny

At 13:48 02-11-2002 -0600, scott.kendall AT abbott DOT com wrote:

>netbackup by itself can do an image, or block by block, backup.  you just 
>have to use the right thing in the file list. can't remember off the top 
>of my head what it is but it is documented in one of the netbackup 
>docs.  the problem with this is it's really only for full DR.  you can't 
>restore a single file, only the entire volume.
>
>to take a backup block by block, but be able to restore a single file, you 
>have to also look at and keep the file to block mapping so you can 
>reference it to pull a single file out of the image.  that's exactly what 
>flashbackup from veritas does.  unfortunately, it's only available on 
>certain platforms and windows isn't one of them yet... but it's supposed 
>to be in the works!!!  I think a lot of us will be looking at this when it 
>finally comes out.
>
>
>- Scott
>
>
>
>Johnny Oestergaard <joe AT joe DOT net>
>Sent by: veritas-bu-admin AT mailman.eng.auburn DOT edu
>
>11/02/2002 04:43 AM
>
>         To:        "David A. Chapa" <david AT datastaff DOT com>, 
> Bruno.Bossier AT comparex DOT be
>         cc:        veritas-bu AT mailman.eng.auburn DOT edu
>         Subject:        Re: [Veritas-bu] Backing up a large amount of 
> small files
>
>
>We have the same type of problem and except that we have turned OTM off and
>have less files the only way I see that we can solve the problem is by
>dividing the directories into different streems and streem the backup to
>disk (so that it doesn't hold a tapedrive for days)
>Later we will move the backup from disk to tape but that should happen at a
>speed that we can handle.
>
>It's great to have so fast tapedrives as we have today (if NT/Win2K could
>pump the data at that speed)
>
>Our files are on a StorageTek SVA9500 and it's still so slow that we cry.
>It's not the disks that are the problem but the time it takes for the
>operatingsystem to open and close the files.
>It would also be a problem on any other operatingsystem (but could be it
>would be a little smaller)
>
>I have been thinking if NBU could take a physical backup of the drive
>(track by track) This should be fast but would also backup empty tracks but
>at 30MB/s or more who cares.
>
>/johnny
>
>At 12:03 30-10-2002 -0500, David A. Chapa wrote:
> >Well, I did see something like this at a client and I always found the best
> >solution to something like this is to just format the dang thing :-)
> >
> >But Seriously now...
> >
> >What RAID level, 5?
> >
> >RAID5 is great for writes but not for reads, and with as many reads as you
> >need
> >done for this backup you would definitely hit timeouts.  This is exactly the
> >problem one of my current clients is facing.
> >
> >What about the last time SCANDISK was run?  We found a heavily fragmented
> >disk
> >caused these types of timeouts as well.
> >
> >Some of the other things we tried was to break up the server backup at the
> >file
> >list.  This was a bit more labor intensive, but it did work to some degree.
> >
> >HTH
> >
> >David
> >
> >Quoting Bruno.Bossier AT comparex DOT be:
> >
> > > We have a server with a directory which contains over 3 million files 
> for a
> > > total amount of nearly 150 Gb. We are trying to back this up in a
> > > reasonable amount of time (12 to 14 hours maximum). OTM is enabled. 
> We have
> > > not succeeded so far. A lot of time is lost at the beginning of the 
> backup
> > > when the backup is going through the directory structure to check all
> > > files. This takes several hours. The first we already did until now is to
> > > set the maximum cache size to 0 (unlimited). This stopped the errors 
> we saw
> > > in the Windows eventlog when OTM is started and then aborted after 40
> > > minutes, then restarted again ....
> > >
> > > Can someone give some suggestions to speed up this backup ?
> > >
> > > Thanks,
> > > Bruno
> > >
> > >
> > > _______________________________________________
> > > Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> > > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> > >
> >
> >
> >
> >_______________________________________________
> >Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
> >http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>_______________________________________________
>Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>