This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.
------_=_NextPart_001_01C46566.D1F69808
Content-Type: text/plain;
charset="iso-8859-1"
Have you talked to your local SE about using Flashbackup?
-----Original Message-----
From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]
Sent: Thursday, July 08, 2004 12:26 PM
To: Cornely, David; Netbackup User_Group
Subject: RE: [Veritas-bu] Millions of files in one directory
I've got a similar problem as an open case with veritas backline support.
its on a solaris client, so slightly different but nbu 5.0 all the same,
about 2.5m files in the backup, which times out eventually after crawling
along for awhile.
On solaris its easier to see what its doing, and I think I've demonstrated
to veritas support that there is a bug because the process spends a lot of
time doing nothing between the close of one file and the open of the next
(like nearly 1 second between files). exclude lists make things worse ( ie
the patten match against each and every file name in your 2m files), but
still I've shown that tar (on which bpbkar is based ) can run the backup
400% faster than netbackup client.
so, if the engineer comes back with a useful update on whether its possible
to fix, I will let the group know, otherwise I suggest a pre-backup process
to tar the 2m files into 1 archive and then back that up, and exclude the
directory with the 2m files in. you can get tar for windows at
http://gnuwin32.sourceforge.net/packages/tar.htm
<http://gnuwin32.sourceforge.net/packages/tar.htm> ...or use something else
windows based.
regards
mark
-----Original Message-----
From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]On Behalf Of Cornely,
David
Sent: 08 July 2004 18:27
To: Netbackup User_Group
Subject: RE: [Veritas-bu] Millions of files in one directory
We've encountered similar problems here but not to that degree. 1-2 million
files is nuts.
The best solution I've seen is splitting up the files into manageable chunks
and making each one a separate stream, but that can have it's own issues if
it's a dynamic file list. Sounds to me like the application design team for
this server didn't consider a few things. Windows is notorious for its
dismal performance with a large number of small files, 3-4 times worse than
Unix.
-----Original Message-----
From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of WILLIAMS,
KRISTOPHER L
Sent: Thursday, July 08, 2004 9:56
To: Netbackup User_Group
Subject: [Veritas-bu] Millions of files in one directory
Hello,
We have a client that is failing because there are too many small files in
one directory. There are apprx. 1 - 2 million files in two different
directories that are causing the failures. The system is a Win2k server
using NetBackup 5.0, master and media servers are also Win2K with NBU 5.0.
Has anyone had this problem and found a good resolution?
Thanks,
Kris
------_=_NextPart_001_01C46566.D1F69808
Content-Type: text/html;
charset="iso-8859-1"
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>RE: [Veritas-bu] Cleaining Tapes LTO's-2</TITLE>
<META content="MSHTML 6.00.2800.1106" name=GENERATOR>
<STYLE>@font-face {
font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: blue; TEXT-DECORATION: underline
}
P.MsoAutoSig {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoAutoSig {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoAutoSig {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY:
"Times New Roman"
}
SPAN.emailstyle18 {
COLOR: navy; FONT-FAMILY: Arial
}
SPAN.EmailStyle19 {
COLOR: navy; FONT-FAMILY: Arial
}
DIV.Section1 {
page: Section1
}
</STYLE>
</HEAD>
<BODY lang=EN-US vLink=blue link=blue>
<DIV><SPAN class=071504303-09072004><FONT face=Arial color=#0000ff size=2>Have
you talked to your local SE about using Flashbackup?</FONT></SPAN></DIV>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B>
veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]<BR><B>Sent:</B>
Thursday,
July 08, 2004 12:26 PM<BR><B>To:</B> Cornely, David; Netbackup
User_Group<BR><B>Subject:</B> RE: [Veritas-bu] Millions of files in one
directory<BR><BR></FONT></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2>I've
got a similar problem as an open case with veritas backline
support.</FONT></SPAN></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff size=2>its
on a solaris client, so slightly different but nbu 5.0 all the same, about
2.5m files in the backup, which times out eventually after crawling along for
awhile.</FONT></SPAN></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff size=2>On
solaris its easier to see what its doing, and I think I've demonstrated to
veritas support that there is a bug because the process spends a lot of time
doing nothing between the close of one file and the open of the next (like
nearly 1 second between files). exclude lists make things worse ( ie the
patten match against each and every file name in your 2m files), but
still I've shown that tar (on which bpbkar is based ) can run the backup 400%
faster than netbackup client.</FONT></SPAN></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff size=2>so,
if the engineer comes back with a useful update on whether its possible to
fix, I will let the group know, otherwise I suggest a pre-backup process to
tar the 2m files into 1 archive and then back that up, and exclude the
directory with the 2m files in. you can get tar for windows at <A
href="http://gnuwin32.sourceforge.net/packages/tar.htm";>http://gnuwin32.sourceforge.net/packages/tar.htm</A> ...or
use something else windows based.</FONT></SPAN></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2>regards</FONT></SPAN></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2>mark</FONT></SPAN></DIV>
<DIV><SPAN class=906072018-08072004><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<BLOCKQUOTE dir=ltr style="MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader dir=ltr align=left><FONT face=Tahoma
size=2>-----Original Message-----<BR><B>From:</B>
veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]<B>On Behalf Of
</B>Cornely,
David<BR><B>Sent:</B> 08 July 2004 18:27<BR><B>To:</B> Netbackup
User_Group<BR><B>Subject:</B> RE: [Veritas-bu] Millions of files in one
directory<BR><BR></FONT></DIV>
<DIV class=Section1>
<P class=MsoNormal><FONT face="Times New Roman" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy">We've encountered similar problems
here
but not to that degree. 1-2 million files is nuts.</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy">The best solution I've seen is
splitting up the files into manageable chunks and making each one a
separate
stream, but that can have it's own issues if it's a dynamic file
list.
Sounds to me like the application design team for this server didn't
consider a few things. Windows is notorious for its dismal
performance
with a large number of small files, 3-4 times worse than
Unix.</SPAN></FONT></P>
<P class=MsoNormal><FONT face="Times New Roman" color=navy size=3><SPAN
style="FONT-SIZE: 12pt; COLOR: navy"></SPAN></FONT> </P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY:
Arial"></SPAN></FONT> </P>
<P class=MsoNormal style="MARGIN-LEFT: 0.5in"><FONT face=Tahoma
size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">-----Original
Message-----<BR><B><SPAN style="FONT-WEIGHT: bold">From:</SPAN></B>
veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] <B><SPAN
style="FONT-WEIGHT: bold">On Behalf Of </SPAN></B>WILLIAMS, KRISTOPHER
L<BR><B><SPAN style="FONT-WEIGHT: bold">Sent:</SPAN></B>
</SPAN></FONT><FONT
face=Tahoma size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">Thursday, July 08,
2004</SPAN></FONT><FONT face=Tahoma size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> </SPAN></FONT><FONT
face=Tahoma size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">9:56</SPAN></FONT><FONT
face=Tahoma size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><BR><B><SPAN
style="FONT-WEIGHT: bold">To:</SPAN></B> Netbackup User_Group<BR><B><SPAN
style="FONT-WEIGHT: bold">Subject:</SPAN></B> [Veritas-bu] Millions of
files
in one directory</SPAN></FONT></P>
<P class=MsoNormal style="MARGIN-LEFT: 0.5in"><FONT face="Times New Roman"
size=3><SPAN style="FONT-SIZE: 12pt"></SPAN></FONT> </P>
<P style="MARGIN-LEFT: 1in"><FONT face="Times New Roman" color=navy
size=3><SPAN style="FONT-SIZE: 12pt; COLOR: navy">Hello,</SPAN></FONT></P>
<P style="MARGIN-LEFT: 1in"><FONT face="Times New Roman" color=navy
size=3><SPAN style="FONT-SIZE: 12pt; COLOR: navy"></SPAN></FONT> </P>
<P style="MARGIN-LEFT: 1in"><FONT face="Times New Roman" color=navy
size=3><SPAN style="FONT-SIZE: 12pt; COLOR: navy">We have a client that is
failing because there are too many small files in one directory. There are
apprx. 1 - 2 million files in two different directories that are causing
the
failures. The system is a Win2k server using NetBackup 5.0, master and
media
servers are also Win2K with NBU 5.0. Has anyone had this problem and found
a
good resolution?</SPAN></FONT> </P>
<P style="MARGIN-LEFT: 0.5in"><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY:
Arial"></SPAN></FONT> </P>
<P style="MARGIN-LEFT: 0.5in"><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY:
Arial"></SPAN></FONT> </P>
<P style="MARGIN-LEFT: 0.5in"><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY:
Arial">Thanks,</SPAN></FONT></P>
<P style="MARGIN-LEFT: 0.5in"><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY:
Arial"></SPAN></FONT> </P>
<P style="MARGIN-LEFT: 0.5in"><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY:
Arial">Kris</SPAN></FONT></P></DIV></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>
------_=_NextPart_001_01C46566.D1F69808--
|
