Hi Calvin
What performance do you see when you do a full backup of this file system?
A while back, I did MANY tests the EMC IP4700 File Server using a directory
tree containing, at its peek, over 20,000,000 files. I put 1,000 4 byte
files per single directory, then create a directory of 10 of these
directories (yielding 10,000 files); then created 10 of those directories,
and so on.
In testing this, I found performance, both for full and incremental backups,
that was much better than what you're seeing. So, one of the factors here
could well be the disk subsystem that the file system is housed in; another
factor could be the file system itself. Most file systems do not excel at
reads and writes of many small files.
If you use Zip or another tool like it, would you plan to use it to do
incremental backups? If not, you'd be comparing apples to oranges. Again,
I'm interested in how the full backups perform. Is always doing a full
backup an option?
I also like the idea suggested in another response of investigating the use
of the Windows 2000 Change Journal; don't know anything about it myself.
Good luck!
tl
-----Original Message-----
From: Calvin Thomas [mailto:calvin.thomas AT nacalogistics DOT com]
Sent: Wednesday, October 16, 2002 3:28 AM
To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
Subject: [Networker] UPDATE:Re: [Networker] What is the best way to
backup 100,000 files in a single subdirectory on a W2K Client
My problem isn't with the save streams that start... They are running at
1/2 the speed of my DLT4000 anyway.
As far as defragmenting the file system, this is a fax server that replaces
these 100,000 file in 5 days. It is a good thought but I don't think it
would help much because the real problem is with the save streams that don't
start.
I have an update on this part of the problem:
I started a save session of the subdirectory from the command line on the
client, and opened task manager to watch save.exe perform. It has been 10
hours since I started the save session, and it is still cranking away as I
write this. So far, in 10 hours it has accessed the disk drive (I/O Reads)
46,486 times, using 50% processor time, and shows no sign of slowing up. I
figure that the client is accessing the disk one time for each file it will
be backing up. The speed seems to be about 1 file access per second, so that
means 100,000 files = 100,000 seconds, or 27 hours.
Kind of a long time to start a backup session........
I guess this means that save.exe can't do the job no matter how I configure
it. I will have to look for a third party solution like the zip or rar
utilities to zip all these files up to a single file that save.exe can
handle in a reasonable amount of time.
If anyone see anything I missed, please let me know.
Thanks
Calvin E. Thomas
----- Original Message -----
From: <lemons_terry AT emc DOT com>
To: <calvin.thomas AT nacalogistics DOT com>
Cc: <NETWORKER AT LISTMAIL.TEMPLE DOT EDU>
Sent: Tuesday, October 15, 2002 8:43 PM
Subject: RE: [Networker] What is the best way to backup 100,000 files in a
single subdirectory on a W2K Client
> Hi Calvin
>
> My gut feel is that 12 simultaneous streams might be too many. Unless you
> have a very high speed file system and disk subsystem, I'd cut that number
> at least in half.
>
> By the by, has the file system holding these 100,000 files been
defragmented
> recently? Fragmentation certainly won't help the backup performance.
>
> Thanks
> tl
>
> -----Original Message-----
> From: Calvin Thomas [mailto:calvin.thomas AT nacalogistics DOT com]
> Sent: Tuesday, October 15, 2002 2:44 PM
> To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
> Subject: Re: [Networker] What is the best way to backup 100,000 files in
> a single subdirectory on a W2K Client
>
>
> Most of the streams start (after a variable amount of time). The
*.302,
> *.303,*.304 Etc up to *.320 all start. I have seen 12 streams running
from
> this client at one time, but the *.301 doesn't start since it has 1/2 the
> files in it. It appears the client doesn't have enough time to select all
> 60K files to backup before the 90 minute timeout. I can't raise the
timeout
> value because it extends out of our backup window.
>
> I know it isn't the tape drive, because I am backing up to a hard
drive.
> This eliminates such problems as shoe-shining, and contention between
> clients.
>
> What I seem to need is a way of archiveing the files on the client to
a
> single file, that I can then backup in place of this subdirectory. I
tried
> winzip to create a zip archive, but it is limited to 65K files.
> Any other ideas?
> Calvin Thomas
>
>
>
> ----- Original Message -----
> From: "Patrick Sibilio" <patrick.sibilio AT COMMERZBANK DOT COM>
> To: <NETWORKER AT LISTMAIL.TEMPLE DOT EDU>
> Sent: Tuesday, October 15, 2002 1:01 AM
> Subject: Re: [Networker] What is the best way to backup 100,000 files in a
> single subdirectory on a W2K Client
>
>
> > Hi,
> >
> > how many streams are acitve during the backup? How many clients are in
the
> > group?
> >
> > Try backing up the client with a new group and a different start time.
> >
> > regards,
> > Patrick Sibilio
> >
> > On Mon, 14 Oct 2002 18:10:00 -0700, Calvin Thomas
> > <calvin.thomas AT NACALOGISTICS DOT COM> wrote:
> >
> > >I have tried seperate save streams by specifying
> > >
> > >C:\Directory\*.301
> > >C:\Directory\*.302
> > >C:\Directory\*.303
> > >Etc.
> > >The problem is the biggest save stream (c:\directory\*.301) never
starts
> > >even though I have set the time out to 90Minutes. I can't set the time
> out
> > >any longer due to other considerations.
> > >When I back up using ALL, the writes happen at 32-64KBPS, and the
backup
> > >takes 20 hours.
> > >I tried to create a zip file using WinZip, but WinZip is limited to 65K
> > >files.
> > >
> > >What have your experiences been in similar sittuations?
> > >
> > >Thanks
> > >Calvin Thomas
> > >UNIX system administrator
> > >NACA Logistics
> > >
> > >--
> > >Note: To sign off this list, send a "signoff" command via email
> > >to listserv AT listmail.temple DOT edu or visit the list's Web site at
> > >http://listmail.temple.edu/archives/networker.html where you can
> > >also view and post messages to the list.
> > >=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
> >
> > --
> > Note: To sign off this list, send a "signoff" command via email
> > to listserv AT listmail.temple DOT edu or visit the list's Web site at
> > http://listmail.temple.edu/archives/networker.html where you can
> > also view and post messages to the list.
> > =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
> >
>
> --
> Note: To sign off this list, send a "signoff" command via email
> to listserv AT listmail.temple DOT edu or visit the list's Web site at
> http://listmail.temple.edu/archives/networker.html where you can
> also view and post messages to the list.
> =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
>
--
Note: To sign off this list, send a "signoff" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
--
Note: To sign off this list, send a "signoff" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
|
