[Veritas-bu] Backing up a large amount of small files
2002-11-02 14:48:26
This is a multipart message in MIME format.
--=_alternative 006D026986256C65_=
Content-Type: text/plain; charset="us-ascii"
netbackup by itself can do an image, or block by block, backup. you just
have to use the right thing in the file list. can't remember off the top
of my head what it is but it is documented in one of the netbackup docs.
the problem with this is it's really only for full DR. you can't restore
a single file, only the entire volume.
to take a backup block by block, but be able to restore a single file, you
have to also look at and keep the file to block mapping so you can
reference it to pull a single file out of the image. that's exactly what
flashbackup from veritas does. unfortunately, it's only available on
certain platforms and windows isn't one of them yet... but it's supposed
to be in the works!!! I think a lot of us will be looking at this when it
finally comes out.
- Scott
Johnny Oestergaard <joe AT joe DOT net>
Sent by: veritas-bu-admin AT mailman.eng.auburn DOT edu
11/02/2002 04:43 AM
To: "David A. Chapa" <david AT datastaff DOT com>, Bruno.Bossier AT
comparex DOT be
cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Backing up a large amount of small
files
We have the same type of problem and except that we have turned OTM off
and
have less files the only way I see that we can solve the problem is by
dividing the directories into different streems and streem the backup to
disk (so that it doesn't hold a tapedrive for days)
Later we will move the backup from disk to tape but that should happen at
a
speed that we can handle.
It's great to have so fast tapedrives as we have today (if NT/Win2K could
pump the data at that speed)
Our files are on a StorageTek SVA9500 and it's still so slow that we cry.
It's not the disks that are the problem but the time it takes for the
operatingsystem to open and close the files.
It would also be a problem on any other operatingsystem (but could be it
would be a little smaller)
I have been thinking if NBU could take a physical backup of the drive
(track by track) This should be fast but would also backup empty tracks
but
at 30MB/s or more who cares.
/johnny
At 12:03 30-10-2002 -0500, David A. Chapa wrote:
>Well, I did see something like this at a client and I always found the
best
>solution to something like this is to just format the dang thing :-)
>
>But Seriously now...
>
>What RAID level, 5?
>
>RAID5 is great for writes but not for reads, and with as many reads as
you
>need
>done for this backup you would definitely hit timeouts. This is exactly
the
>problem one of my current clients is facing.
>
>What about the last time SCANDISK was run? We found a heavily fragmented
>disk
>caused these types of timeouts as well.
>
>Some of the other things we tried was to break up the server backup at
the
>file
>list. This was a bit more labor intensive, but it did work to some
degree.
>
>HTH
>
>David
>
>Quoting Bruno.Bossier AT comparex DOT be:
>
> > We have a server with a directory which contains over 3 million files
for a
> > total amount of nearly 150 Gb. We are trying to back this up in a
> > reasonable amount of time (12 to 14 hours maximum). OTM is enabled. We
have
> > not succeeded so far. A lot of time is lost at the beginning of the
backup
> > when the backup is going through the directory structure to check all
> > files. This takes several hours. The first we already did until now is
to
> > set the maximum cache size to 0 (unlimited). This stopped the errors
we saw
> > in the Windows eventlog when OTM is started and then aborted after 40
> > minutes, then restarted again ....
> >
> > Can someone give some suggestions to speed up this backup ?
> >
> > Thanks,
> > Bruno
> >
> >
> > _______________________________________________
> > Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >
>
>
>
>_______________________________________________
>Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
--=_alternative 006D026986256C65_=
Content-Type: text/html; charset="us-ascii"
<br><font size=2 face="Arial">netbackup by itself can do an image, or block by
block, backup. you just have to use the right thing in the file list.
can't remember off the top of my head what it is but it is documented in one of
the netbackup docs. the problem with this is it's really only for full
DR. you can't restore a single file, only the entire volume.</font>
<br>
<br><font size=2 face="Arial">to take a backup block by block, but be able to
restore a single file, you have to also look at and keep the file to block
mapping so you can reference it to pull a single file out of the image.
that's exactly what flashbackup from veritas does. unfortunately,
it's only available on certain platforms and windows isn't one of them yet...
but it's supposed to be in the works!!! I think a lot of us will be
looking at this when it finally comes out.</font>
<br>
<br>
<br><font size=2 face="Arial">- Scott</font>
<br>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td>
<td><font size=1 face="sans-serif"><b>Johnny Oestergaard <joe AT joe DOT
net></b></font>
<br><font size=1 face="sans-serif">Sent by: veritas-bu-admin AT
mailman.eng.auburn DOT edu</font>
<p><font size=1 face="sans-serif">11/02/2002 04:43 AM</font>
<br>
<td><font size=1 face="Arial"> </font>
<br><font size=1 face="sans-serif"> To:
"David A. Chapa" <david AT datastaff DOT
com>, Bruno.Bossier AT comparex DOT be</font>
<br><font size=1 face="sans-serif"> cc:
veritas-bu AT mailman.eng.auburn DOT edu</font>
<br><font size=1 face="sans-serif"> Subject:
Re: [Veritas-bu] Backing up a large amount of small
files</font></table>
<br>
<br>
<br><font size=2 face="Courier New">We have the same type of problem and except
that we have turned OTM off and <br>
have less files the only way I see that we can solve the problem is by <br>
dividing the directories into different streems and streem the backup to <br>
disk (so that it doesn't hold a tapedrive for days)<br>
Later we will move the backup from disk to tape but that should happen at a <br>
speed that we can handle.<br>
<br>
It's great to have so fast tapedrives as we have today (if NT/Win2K could <br>
pump the data at that speed)<br>
<br>
Our files are on a StorageTek SVA9500 and it's still so slow that we cry. <br>
It's not the disks that are the problem but the time it takes for the <br>
operatingsystem to open and close the files.<br>
It would also be a problem on any other operatingsystem (but could be it <br>
would be a little smaller)<br>
<br>
I have been thinking if NBU could take a physical backup of the drive <br>
(track by track) This should be fast but would also backup empty tracks but <br>
at 30MB/s or more who cares.<br>
<br>
/johnny<br>
<br>
At 12:03 30-10-2002 -0500, David A. Chapa wrote:<br>
>Well, I did see something like this at a client and I always found the
best<br>
>solution to something like this is to just format the dang thing :-)<br>
><br>
>But Seriously now...<br>
><br>
>What RAID level, 5?<br>
><br>
>RAID5 is great for writes but not for reads, and with as many reads as you
<br>
>need<br>
>done for this backup you would definitely hit timeouts. This is
exactly the<br>
>problem one of my current clients is facing.<br>
><br>
>What about the last time SCANDISK was run? We found a heavily
fragmented <br>
>disk<br>
>caused these types of timeouts as well.<br>
><br>
>Some of the other things we tried was to break up the server backup at the
<br>
>file<br>
>list. This was a bit more labor intensive, but it did work to some
degree.<br>
><br>
>HTH<br>
><br>
>David<br>
><br>
>Quoting Bruno.Bossier AT comparex DOT be:<br>
><br>
> > We have a server with a directory which contains over 3 million files
for a<br>
> > total amount of nearly 150 Gb. We are trying to back this up in a<br>
> > reasonable amount of time (12 to 14 hours maximum). OTM is enabled.
We have<br>
> > not succeeded so far. A lot of time is lost at the beginning of the
backup<br>
> > when the backup is going through the directory structure to check
all<br>
> > files. This takes several hours. The first we already did until now
is to<br>
> > set the maximum cache size to 0 (unlimited). This stopped the errors
we saw<br>
> > in the Windows eventlog when OTM is started and then aborted after
40<br>
> > minutes, then restarted again ....<br>
> ><br>
> > Can someone give some suggestions to speed up this backup ?<br>
> ><br>
> > Thanks,<br>
> > Bruno<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn
DOT edu<br>
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu<br>
> ><br>
><br>
><br>
><br>
>_______________________________________________<br>
>Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT
edu<br>
>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu<br>
<br>
_______________________________________________<br>
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu<br>
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu<br>
</font>
<br>
<br>
--=_alternative 006D026986256C65_=--
|
<Prev in Thread] |
Current Thread |
[Next in Thread>
|
- [Veritas-bu] Backing up a large amount of small files, Johnny Oestergaard
- [Veritas-bu] Backing up a large amount of small files, scott.kendall AT abbott DOT com
- [Veritas-bu] Backing up a large amount of small files, scott.kendall AT abbott DOT com
- [Veritas-bu] Backing up a large amount of small files, scott.kendall AT abbott DOT com
- [Veritas-bu] Backing up a large amount of small files,
scott.kendall AT abbott DOT com <=
- [Veritas-bu] Backing up a large amount of small files, Johnny Oestergaard
- [Veritas-bu] Backing up a large amount of small files, scott.kendall AT abbott DOT com
- [Veritas-bu] Backing up a large amount of small files, Bruno.Bossier AT comparex DOT be
- [Veritas-bu] Backing up a large amount of small files, Bruno.Bossier AT comparex DOT be
- [Veritas-bu] Backing up a large amount of small files, Vapiwala, Kevin V [IT]
- [Veritas-bu] Backing up a large amount of small files, Quarantine
- [Veritas-bu] Backing up a large amount of small files, Bruno.Bossier AT comparex DOT be
|
|
|