Bill,
Thanks for your input.
You're right. This is probably the epitomy of a nightmare restore. Not all
of the 16Gb was made up of 2k files.
That was more of a generalisation perhaps even an exaggeration rather than
an exact measurement.
The bottom line though is that the majority of that 16Gb was made up of
files that were anything from 1-4kb.
The SMS server stored a log file for each piece of software installed on
each lterm within the company. This helps the SMS guys
trouleshoot software distribution. Each lterm had it's own directory and
within that directory the log files.
The stats which you outline suggest that the bottleneck may have existed on
the client O/S and Filesystem.
Fortunately this server has now been re-engineered. They are now spreading
the log files across multiple physical and logical drives , instead of it
all being on one huge logical drive... so this should help some.
I will try to pursue your suggestion though regarding a simple zip/unzip of
a few hundred thousand files. It may yield some clues as to what exactly I
can expect out of the client O/S and filesystem. NTFS certainly has a
reputation for being a slow filesystem (despite what Microsoft might have
you believe).
I'd like to know if anyone has tested a rival backup product such as Legato
alongside ADSM and seeing what happens when small files are encountered.
We do plan to do this at some point in the future using Compaq's EBS, but
we're waiting on Legato to get certified. At the moment only Veritas
(Seagate Backup Exec) and Arcserve are certified and I've had my share of
bad experiences with them already.
Our Intel Server team is currently testing Compaq's RA8000 Fibre Channel
Array. We do plan to do some testing of restores to this array to see if we
can increase throughput.
I'll bear in mind your suggestion with the zip/unzip and let you know what I
find out.
Thanks,
Nathan
> -----Original Message-----
> From: Bill Colwell [SMTP:bcolwell AT DRAPER DOT COM]
> Sent: Friday, July 23, 1999 12:40 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: Performance
>
> Nathan,
>
> I did some calculations on your numbers. 16GB of 2K files
> is 8,000,000 files. Is this right? Do you have that many on this server?
> (ouch!)
>
> Restoring them in 27 hours works out to 82 files per second.
> Put another way, you have to create a new file in the client filesystem
> every 12 milliseconds. That's pretty fast. If you wanted to do the
> restore in a third of the time that would be a new file every 3
> milliseconds.
> Can your filesystem do this for sustained periods?
>
> If you want to persue this, I suggest zipping up 100,000 of these files
> and then unzipping them to a new spot. This should be a clean test
> of filesystem performance. Your results would be very useful to us and
> IBM.
>
> --
> --------------------------
> Bill Colwell
> C. S. Draper Lab
> Cambridge, Ma.
> bcolwell AT draper DOT com
> --------------------------
>
>
> In <777DE9DB19FAD111897E00204840162801849682 AT exchange20.usaa001 DOT com>,
> on
> 07/23/99
> at 12:16 PM, Nathan King <nathan.king AT USAA DOT COM> said:
>
> >Richard,
>
> >I agree whole heartedly with your comments. However I think ADSM
> >Developement is well aware of the performance degradation caused by small
> >files.
> >As noted in the ADSM NT Server readme.
>
> > Next to system hardware, the number and size of client files has a
> > big impact on ADSM performance. The performance when tranferring
> > small files is lower than when transferring large files.
>
> > For example, using a 133MHZ pentium, FAT file system, local ADSM
> > client with no compression, and named pipes to a disk storage pool
> > throughput has been measured at:
>
> > File Size
> > Throughput 1KB 10KB 100KB 10MB
> > KB/Sec 57 445 1280 1625
> > GB/hr 0.20 1.53 4.39 5.58
>
> >Ok so this is a pentium 133 but with it being a local client you can rule
> >out the network as an issue.
> >Named pipes is the fastest way to backup locally on NT.
>
> >So even if I took the hardware and made it five times faster and
> supposing
> >that by some magic that 5times faster hardware=5times faster adsm
> >performance, my 16gb restore of small 1-2kb files would still have taken
> >16hours!!
>
> >Unacceptable!
>
> >Nathan
>
>
> >> To: ADSM-L AT VM.MARIST DOT EDU
> >> Subject: Re: Performance
> >>
> >> ...
> >> >However when it comes to restoring servers which are 90% small files
> then
> >> we
> >> >may aswell be on a 4Mb Token Ring. We once
> >> >had the grand oppurtunity to restore the drive of an SMS Server which
> was
> >> >made up of about 16Gb worth of 2Kb log files.
> >> >This took 27hrs.
> >> ...
> >> > when it comes to small files Adsm just plain sucks!
> >> ...
> >> >Any thoughts?
> >>
> >> Nathan - You clearly had a very frustrating, unsatisfactory restoral
> >> experience. But realize that it's not helping the many
> customers
> >> on
> >> the mailing list to just receive a splat without any details as to how
> the
> >> restoral was performed...like network configuration, network load at
> the
> >> time,
> >> server system load, what else the ADSM server was doing, server
> >> configuration,
> >> server options, database cache hit ratio, client load, disk
> configuration,
> >> file system topology, ADSM client options, restoral command, results
> from
> >> testing various combinations in your configuration, etc.
> >>
> >> The great value in our mailing list is in discovering optimal
> techniques,
> >> and
> >> then sharing them, to the betterment of all. Realize that it's plain
> >> frustrating for the people on the list to see complaints rather than
> the
> >> results from measured analysis. And it does not point to solid
> evidence
> >> that
> >> IBM can react to in bettering the product - which will help everyone.
> >>
> >> Sure - there's a lot of frustration out in the customer base. But I as
> a
> >> customer feel that I have a responsibility both as a customer and a
> >> professional in the data processing field to do some research, to prod
> the
> >> system, change variables, and find out where the problems are. We can
> >> shower
> >> IBM with raw complaints and they can tell us about KLOC numbers, but
> >> working
> >> together with specifics will really solve problems. Yes, sometimes
> it's
> >> necessary to shake up vendors to get them to be responsive to the
> >> realities we
> >> face, but the rest of the time customers and vendor need to dig in and
> get
> >> to
> >> the bottom of what's really wrong. Hey, we're not here as a personal
> >> pursuit:
> >> we're in company and institutional positions as employees expected to
> >> effectively deal with problems for the better operation of the
> >> organization so
> >> that we can all make more money!
> >>
> >> Naturally, not all customer sites have the resources to pursue things
> to
> >> the
> >> depth that better-equipped sites can, and hence depend upon the results
> >> that
> >> others find. The mailing list is a dissemination and discussion point
> for
> >> what ails us and what we can do for each other as well as solicit the
> >> vendor,
> >> IBM, to improve conditions beyond the pointed problems we need to bring
> to
> >> the
> >> Support Center. IBM, in turn, needs to be responsive to the discontent
> >> evidenced in the customer base, which is the List's great value to
> them;
> >> and
> >> they need to feed back on optimal techniques, which they often do very
> >> well in
> >> the famous Redbooks [please encourage them]. From management feedback
> to
> >> the
> >> List, it is very evident that IBM is listening; but what they
> particularly
> >> want to hear is substantive feedback to allow them to respond as
> >> substantively. This is what we need to do.
> >>
> >> So channel that energy in ways that will help the general case.
> >>
> >> Richard Sims, BU
|
