Veritas-bu

[Veritas-bu] Millions of files in one directory

2004-07-08 14:26:11
Subject: [Veritas-bu] Millions of files in one directory
From: mark AT steelfamily DOT org (mark)
Date: Thu, 8 Jul 2004 19:26:11 +0100
This is a multi-part message in MIME format.

------=_NextPart_000_0010_01C46521.6D39FEE0
Content-Type: text/plain;
        charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

RE: [Veritas-bu] Cleaining Tapes LTO's-2I've got a similar problem as an
open case with veritas backline support.

its on a solaris client, so slightly different but nbu 5.0 all the same,
about 2.5m files in the backup, which times out eventually after crawling
along for awhile.

On solaris its easier to see what its doing, and I think I've demonstrated
to veritas support that there is a bug because the process spends a lot of
time doing nothing between the close of one file and the open of the next
(like nearly 1 second between files). exclude lists make things worse ( ie
the patten match against each and every  file name in your 2m files), but
still I've shown that tar (on which bpbkar is based ) can run the backup
400% faster than netbackup client.

so, if the engineer comes back with a useful update on whether its possible
to fix, I will let the group know, otherwise I suggest a pre-backup process
to tar the 2m files into 1 archive and then back that up, and exclude the
directory with the 2m files in. you can get tar for windows at
http://gnuwin32.sourceforge.net/packages/tar.htm ...or use something else
windows based.

regards
mark

  -----Original Message-----
  From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]On Behalf Of Cornely, 
David
  Sent: 08 July 2004 18:27
  To: Netbackup User_Group
  Subject: RE: [Veritas-bu] Millions of files in one directory


  We've encountered similar problems here but not to that degree.  1-2
million files is nuts.

  The best solution I've seen is splitting up the files into manageable
chunks and making each one a separate stream, but that can have it's own
issues if it's a dynamic file list.  Sounds to me like the application
design team for this server didn't consider a few things.  Windows is
notorious for its dismal performance with a large number of small files, 3-4
times worse than Unix.





  -----Original Message-----
  From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of WILLIAMS,
KRISTOPHER L
  Sent: Thursday, July 08, 2004 9:56
  To: Netbackup User_Group
  Subject: [Veritas-bu] Millions of files in one directory



  Hello,



  We have a client that is failing because there are too many small files in
one directory. There are apprx. 1 - 2 million files in two different
directories that are causing the failures. The system is a Win2k server
using NetBackup 5.0, master and media servers are also Win2K with NBU 5.0.
Has anyone had this problem and found a good resolution?





  Thanks,



  Kris

------=_NextPart_000_0010_01C46521.6D39FEE0
Content-Type: text/html;
        charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>RE: [Veritas-bu] Cleaining Tapes LTO's-2</TITLE>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2800.1400" name=3DGENERATOR>
<STYLE>@font-face {
        font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
        FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
        FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
        FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
A:link {
        COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
        COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
        COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
        COLOR: blue; TEXT-DECORATION: underline
}
P.MsoAutoSig {
        FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoAutoSig {
        FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoAutoSig {
        FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
P {
        FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: =
"Times New Roman"
}
SPAN.emailstyle18 {
        COLOR: navy; FONT-FAMILY: Arial
}
SPAN.EmailStyle19 {
        COLOR: navy; FONT-FAMILY: Arial
}
DIV.Section1 {
        page: Section1
}
</STYLE>
</HEAD>
<BODY lang=3DEN-US vLink=3Dblue link=3Dblue>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =
size=3D2>I've=20
got a similar problem as an open case with veritas backline=20
support.</FONT></SPAN></DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =
size=3D2>its on=20
a solaris client, so slightly different but nbu 5.0 all the same, about =
2.5m=20
files in the backup, which times out eventually after crawling along for =

awhile.</FONT></SPAN></DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =
size=3D2>On=20
solaris its easier to see what its doing, and I think I've demonstrated =
to=20
veritas support that there is a bug because the process spends a lot of =
time=20
doing nothing between the close of one file and the open of the next =
(like=20
nearly 1 second between files). exclude lists make things worse ( ie the =
patten=20
match against each and every&nbsp; file name in your 2m files), but =
still I've=20
shown that tar (on which bpbkar is based ) can run the backup 400% =
faster than=20
netbackup client.</FONT></SPAN></DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =
size=3D2>so, if=20
the engineer comes back with a useful update on whether its possible to =
fix, I=20
will let the group know, otherwise I suggest a pre-backup process to tar =
the 2m=20
files into 1 archive and then back that up, and exclude the directory =
with the=20
2m files in. you can get tar for windows at <A=20
href=3D"http://gnuwin32.sourceforge.net/packages/tar.htm";>http://gnuwin32=
.sourceforge.net/packages/tar.htm</A>&nbsp;...or=20
use something else windows based.</FONT></SPAN></DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2></FONT></SPAN>&nbsp;</DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2>regards</FONT></SPAN></DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2>mark</FONT></SPAN></DIV>
<DIV><SPAN class=3D906072018-08072004><FONT face=3DArial color=3D#0000ff =

size=3D2></FONT></SPAN>&nbsp;</DIV>
<BLOCKQUOTE dir=3Dltr style=3D"MARGIN-RIGHT: 0px">
  <DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft><FONT =
face=3DTahoma=20
  size=3D2>-----Original Message-----<BR><B>From:</B>=20
  veritas-bu-admin AT mailman.eng.auburn DOT edu=20
  [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]<B>On Behalf Of =
</B>Cornely,=20
  David<BR><B>Sent:</B> 08 July 2004 18:27<BR><B>To:</B> Netbackup=20
  User_Group<BR><B>Subject:</B> RE: [Veritas-bu] Millions of files in =
one=20
  directory<BR><BR></FONT></DIV>
  <DIV class=3DSection1>
  <P class=3DMsoNormal><FONT face=3D"Times New Roman" color=3Dnavy =
size=3D3><SPAN=20
  style=3D"FONT-SIZE: 12pt; COLOR: navy">We've encountered similar =
problems here=20
  but not to that degree.&nbsp; 1-2 million files is =
nuts.</SPAN></FONT></P>
  <P class=3DMsoNormal><FONT face=3D"Times New Roman" color=3Dnavy =
size=3D3><SPAN=20
  style=3D"FONT-SIZE: 12pt; COLOR: navy">The best solution I've seen is =
splitting=20
  up the files into manageable chunks and making each one a separate =
stream, but=20
  that can have it's own issues if it's a dynamic file list.&nbsp; =
Sounds to me=20
  like the application design team for this server didn't consider a few =

  things.&nbsp; Windows is notorious for its dismal performance with a =
large=20
  number of small files, 3-4 times worse than Unix.</SPAN></FONT></P>
  <P class=3DMsoNormal><FONT face=3D"Times New Roman" color=3Dnavy =
size=3D3><SPAN=20
  style=3D"FONT-SIZE: 12pt; COLOR: navy"></SPAN></FONT>&nbsp;</P>
  <P class=3DMsoNormal><FONT face=3DArial color=3Dnavy size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial"></SPAN></FONT>&nbsp;</P>
  <P class=3DMsoNormal style=3D"MARGIN-LEFT: 0.5in"><FONT face=3DTahoma =
size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">-----Original=20
  Message-----<BR><B><SPAN style=3D"FONT-WEIGHT: bold">From:</SPAN></B>=20
  veritas-bu-admin AT mailman.eng.auburn DOT edu=20
  [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] <B><SPAN=20
  style=3D"FONT-WEIGHT: bold">On Behalf Of </SPAN></B>WILLIAMS, =
KRISTOPHER=20
  L<BR><B><SPAN style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> =
</SPAN></FONT><FONT=20
  face=3DTahoma size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">Thursday, July 08,=20
  2004</SPAN></FONT><FONT face=3DTahoma size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> </SPAN></FONT><FONT =
face=3DTahoma=20
  size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">9:56</SPAN></FONT><FONT =

  face=3DTahoma size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"><BR><B><SPAN=20
  style=3D"FONT-WEIGHT: bold">To:</SPAN></B> Netbackup =
User_Group<BR><B><SPAN=20
  style=3D"FONT-WEIGHT: bold">Subject:</SPAN></B> [Veritas-bu] Millions =
of files=20
  in one directory</SPAN></FONT></P>
  <P class=3DMsoNormal style=3D"MARGIN-LEFT: 0.5in"><FONT face=3D"Times =
New Roman"=20
  size=3D3><SPAN style=3D"FONT-SIZE: 12pt"></SPAN></FONT>&nbsp;</P>
  <P style=3D"MARGIN-LEFT: 1in"><FONT face=3D"Times New Roman" =
color=3Dnavy=20
  size=3D3><SPAN style=3D"FONT-SIZE: 12pt; COLOR: =
navy">Hello,</SPAN></FONT></P>
  <P style=3D"MARGIN-LEFT: 1in"><FONT face=3D"Times New Roman" =
color=3Dnavy=20
  size=3D3><SPAN style=3D"FONT-SIZE: 12pt; COLOR: =
navy"></SPAN></FONT>&nbsp;</P>
  <P style=3D"MARGIN-LEFT: 1in"><FONT face=3D"Times New Roman" =
color=3Dnavy=20
  size=3D3><SPAN style=3D"FONT-SIZE: 12pt; COLOR: navy">We have a client =
that is=20
  failing because there are too many small files in one directory. There =
are=20
  apprx. 1 - 2 million files in two different directories that are =
causing the=20
  failures. The system is a Win2k server using NetBackup 5.0, master and =
media=20
  servers are also Win2K with NBU 5.0. Has anyone had this problem and =
found a=20
  good resolution?</SPAN></FONT> </P>
  <P style=3D"MARGIN-LEFT: 0.5in"><FONT face=3DArial color=3Dnavy =
size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial"></SPAN></FONT>&nbsp;</P>
  <P style=3D"MARGIN-LEFT: 0.5in"><FONT face=3DArial color=3Dnavy =
size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial"></SPAN></FONT>&nbsp;</P>
  <P style=3D"MARGIN-LEFT: 0.5in"><FONT face=3DArial color=3Dnavy =
size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial">Thanks,</SPAN></FONT></P>
  <P style=3D"MARGIN-LEFT: 0.5in"><FONT face=3DArial color=3Dnavy =
size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial"></SPAN></FONT>&nbsp;</P>
  <P style=3D"MARGIN-LEFT: 0.5in"><FONT face=3DArial color=3Dnavy =
size=3D2><SPAN=20
  style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: =
Arial">Kris</SPAN></FONT></P></DIV></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_0010_01C46521.6D39FEE0--