ADSM-L

Re: Large number of objects

1999-06-09 10:08:17
Subject: Re: Large number of objects
From: DAVID HENDRIX <dmhendri AT FEDEX DOT COM>
Date: Wed, 9 Jun 1999 08:08:17 -0600
Steve,

We have a few systems which generate anywhere from 10,000-60,000 files per day.
We don't keep the data long term - maybe no more than a quarter.

A few things helped us:

1) Version 3 client and server.  Allows PIT protocol and also incremental of
directories rather than filesystems only.
2) App designers/development were asked to logically organize the files so that
we would not scan "unchanged" directories each day.  In other words, we backed
up the OS filesystems, etc separately from the data (using multiple server
stanzas and domain/inclexcl statements).  The app itself called dsmc and
performed an incremental on a specific (say a daily) directory.
3) Make sure you have enough memory and CPU to allow the client process to
finish.

A system is going into field test here in a few months that will generate
500,000 files per day or more, keeping them about a quarter.  We'll follow the
steps above and see if ADSM can handle it.  My major concern there is the
catalog space used.  Our SLA has the primary copy, plus on and offsite copies -
that's 1200 bytes per file.  Do the math and you can see the exposure and heavy
processing needed to perform any kind of restore.

At 200K per file, expect the aggregate throughput to be about 33% of what you
normally might see with large objects.  But that varies also and can only be
verified by testing.

David Hendrix
dmhendri AT fedex DOT com








Steve Watson <steve.watson AT XINETICA DOT COM> on 06/09/99 07:02:39 AM

Please respond to "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>








 To:      ADSM-L AT VM.MARIST DOT EDU

 cc:      (bcc: DAVID HENDRIX/216832/ITD/FEDEX)



 Subject: Large number of objects









I am working on a system which long term will store 10 million files on SSA RAID
yes that's million !

We obviously will not want to execute incremental style backups against these
files due to the delays in scanning for date/time changes during this type of
backup, but we will still have a fairly large number to backup on a daily basis
approx 25,000 avg file size 200k.

Does anyone have experience of these kinds of voumes, or any performance stats
on anything that can be scaled to give us some idea of what kind of performance
we can expect.

I suspect we will have to do some testing sooner than later.

Regards
Steve Watson.
Technical Consultant. Xinetica Ltd
<Prev in Thread] Current Thread [Next in Thread>