--------------000905020608060107050702
Content-Type: text/plain;
charset=us-ascii;
format=flowed
Content-Transfer-Encoding: 7bit
You guys are right. I suspected the catalog processing, but couln't be
sure. An email from Karen Shoenbauer of Veritas clued me in to checking
the bptm logs. There I saw that after the backup completed for the
first volume, NBU kicks off ndmp_bpfsmap_create, which runs for 17
hours! Only after this completes does the backup of the next volume begin.
Apparantly 4.5 eliminates this catalog process (as Steve mentions), so
it should speed up our total backup time significantly!
As far as the dump filesystem walking goes, this has been improved with
subsequent releases of Netapp software, and is now much quicker than it was.
Moshe
Steve Kappel wrote:
>There are two things going on here. One is the catalog post-
>processing that NetBackup NDMP must do to convert the inode-based
>information provided by the NetApp into the native path-based
>catalog. This overhead depends on the number of objects in
>the backup. Starting with NetBackup 4.5 there is no longer any
>catalog post-processing so this overhead is eliminated.
>
>Secondly, dump walks the filesystem before it starts sending
>backup data. A quick google search comes up with this
>short overview of dump:
>http://www.usenix.org/publications/library/proceedings/osdi99/full_papers/hutchi
>nson/hutchinson_html/node6.html
>
>
>Jeff Kennedy wrote:
>
>>I believe the problem is *before* the dump even begins. I have seen
>>this problem on volumes that have a tremendous number of small files or
>>are terribly fragmented. But I've never seen the problem it just start
>>out of the blue, it's usually a gradual process that gets worse with
>>time.
>>
>>~JK
>>
>>John D Stephens wrote:
>>
>>>Moshe -
>>>I have seen huge delays between volumes as well. What is going
>>>on is this. After NBU starts the NDMP backup, it steps out
>>>of the way and idle until the filer signals NBU that the dump
>>>has completed. Then the filer sends all the meta data to NBU
>>>and NBU then catalogs that data. Depending on the size of your
>>>backup and the amount of files, it takes a little while for NBU
>>>to chug through the catalogs. Hopefully, 4.5 is faster at this.
>>>Check your iostat -x and look for the activity on your catalogs
>>>after the filer finishes sending the NDMP job, but NBU still
>>>shows it to be active. The only way around this is to use
>>>multiple streams. Be carefull, multiple streaming does use the
>>>filer's CPU more than normal.
>>>
>>>Hope this helps.
>>>
>>>John
>>>
>>>Moshe Linzer wrote:
>>>
>>>>Lately we have started seeing delays between volumes when backing up our
>>>>Netapp filer. I have a single AIT-2 tape attached to my F760, and I run
>>>>a full backup of the filer, which contains 3 volumes. The performance
>>>>on the first volume is as expected, but then there is a delay of
>>>>sometimes up to 12 hours before the second volume begins to be backed
>>>>up! The same thing happens before the third volume. volume sizes are
>>>>between 375-480GB. I think this began after we applied some patches
>>>>lately, but the patch descriptions don't seem to have any bearing on
>>>>this problem. Had anyone seen this type of behaviour?
>>>>
>>>>Thanks,
>>>>
>>>>Moshe
>>>>
>>>>_______________________________________________
>>>>Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
>>>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>>>
>>>--
>>> +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
>>> + John D Stephens ITS Design Systems +
>>>+ Texas Instruments 12500 TI BLVD, Dallas +
>>> + jstephens AT ti DOT com 214-480-6229 +
>>> +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+
>>>_______________________________________________
>>>Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
>>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>>
>>--
>>=====================
>>Jeff Kennedy
>>Unix Administrator
>>AMCC
>>jlkennedy AT amcc DOT com
>>_______________________________________________
>>Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
>>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>>
>
>
>__________________________________________________________________________
>Steve Kappel steve.kappel AT veritas DOT com
>VERITAS Software (Engineering)
>
--------------000905020608060107050702
Content-Type: text/html;
charset=us-ascii
Content-Transfer-Encoding: 7bit
<html>
<head>
</head>
<body>
You guys are right. I suspected the catalog processing, but couln't be
sure.
An email from Karen Shoenbauer of Veritas clued me in to checking the bptm
logs. There I saw that after the backup completed for the first volume,
NBU kicks off ndmp_bpfsmap_create, which runs for 17 hours! Only
after
this completes does the backup of the next volume begin.<br>
Apparantly 4.5 eliminates this catalog process (as Steve mentions), so it
should speed up our total backup time significantly!<br>
<br>
As far as the dump filesystem walking goes, this has been improved with
subsequent
releases of Netapp software, and is now much quicker than it was.<br>
<br>
Moshe<br>
<br>
Steve Kappel wrote:<br>
<blockquote type="cite" cite="mid:200209091524.g89FOOE02706 AT
brome.min.veritas DOT com">
<pre wrap="">There are two things going on here. One is the catalog
post-<br>processing that NetBackup NDMP must do to convert the
inode-based<br>information provided by the NetApp into the native
path-based<br>catalog. This overhead depends on the number of objects
in<br>the backup. Starting with NetBackup 4.5 there is no longer
any<br>catalog post-processing so this overhead is eliminated.<br><br>Secondly,
dump walks the filesystem before it starts sending<br>backup data. A quick
google search comes up with this<br>short overview of dump:<br><a
class="moz-txt-link-freetext"
href="http://www.usenix.org/publications/library/proceedings/osdi99/full_papers/hutchi";>http://www.usenix.org/publications/library/proceedings/osdi99/full_papers/hutchi</a><br>nson/hutchinson_html/node6.html<br><br><br>Jeff
Kennedy wrote:<br></pre>
<blockquote type="cite">
<pre wrap="">I believe the problem is *before* the dump even begins. I
have seen<br>this problem on volumes that have a tremendous number of small
files or<br>are terribly fragmented. But I've never seen the problem it just
start<br>out of the blue, it's usually a gradual process that gets worse
with<br>time.<br><br>~JK<br><br>John D Stephens wrote:<br></pre>
<blockquote type="cite">
<pre wrap="">Moshe -<br>I have seen huge delays between volumes as well.
What is going<br>on is this. After NBU starts the NDMP backup, it steps
out<br>of the way and idle until the filer signals NBU that the dump<br>has
completed. Then the filer sends all the meta data to NBU<br>and NBU then
catalogs that data. Depending on the size of your<br>backup and the amount of
files, it takes a little while for NBU<br>to chug through the catalogs.
Hopefully, 4.5 is faster at this.<br>Check your iostat -x and look for the
activity on your catalogs<br>after the filer finishes sending the NDMP job, but
NBU still<br>shows it to be active. The only way around this is to
use<br>multiple streams. Be carefull, multiple streaming does use
the<br>filer's CPU more than normal.<br><br>Hope this
helps.<br><br>John<br><br>Moshe Linzer wrote:<br></pre>
<blockquote type="cite">
<pre wrap="">Lately we have started seeing delays between volumes when
backing up our<br>Netapp filer. I have a single AIT-2 tape attached to my
F760, and I run<br>a full backup of the filer, which contains 3 volumes. The
performance<br>on the first volume is as expected, but then there is a delay
of<br>sometimes up to 12 hours before the second volume begins to be
backed<br>up! The same thing happens before the third volume. volume sizes
are<br>between 375-480GB. I think this began after we applied some
patches<br>lately, but the patch descriptions don't seem to have any bearing
on<br>this problem. Had anyone seen this type of
behaviour?<br><br>Thanks,<br><br>Moshe<br><br>_______________________________________________<br>Veritas-bu
maillist - <a class="moz-txt-link-abbreviated" href="mailto:Veritas-bu AT
mailman.eng.auburn DOT edu">Veritas-bu AT mailman.eng.auburn DOT edu</a><br><a
class="moz-txt-link-freetext"
href="http://mailman.eng.auburn.edu/mailman/listinfo/veritas-
bu">http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu</a><br></pre>
</blockquote>
<pre wrap="">--<br> +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+<br> +
John D Stephens ITS Design Systems +<br>+ Texas Instruments 12500 TI
BLVD, Dallas +<br> + <a class="moz-txt-link-abbreviated"
href="mailto:jstephens AT ti DOT com">jstephens AT ti DOT com</a>
214-480-6229 +<br>
+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+<br>_______________________________________________<br>Veritas-bu
maillist - <a class="moz-txt-link-abbreviated" href="mailto:Veritas-bu AT
mailman.eng.auburn DOT edu">Veritas-bu AT mailman.eng.auburn DOT edu</a><br><a
class="moz-txt-link-freetext"
href="http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu";>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu</a><br></pre>
</blockquote>
<pre wrap="">-- <br>=====================<br>Jeff Kennedy<br>Unix
Administrator<br>AMCC<br><a class="moz-txt-link-abbreviated"
href="mailto:jlkennedy AT amcc DOT com">jlkennedy AT amcc DOT
com</a><br>_______________________________________________<br>Veritas-bu
maillist - <a class="moz-txt-link-abbreviated" href="mailto:Veritas-bu AT
mailman.eng.auburn DOT edu">Veritas-bu AT mailman.eng.auburn DOT edu</a><br><a
class="moz-txt-link-freetext"
href="http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu";>http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu</a><br></pre>
</blockquote>
<pre
wrap=""><!----><br><br>__________________________________________________________________________<br>Steve
Kappel <a class="moz-txt-link-abbreviated"
href="mailto:steve.kappel AT veritas DOT com">steve.kappel AT veritas DOT
com</a><br>VERITAS Software (Engineering)<br><br></pre>
</blockquote>
<br>
</body>
</html>
--------------000905020608060107050702--
|
