Re: 1.5 Million Files

1997-12-09 00:41:22
Subject: Re: 1.5 Million Files
From: "Thomas A. La Porte" <tlaporte AT ANIM.DREAMWORKS DOT COM>
Date: Mon, 8 Dec 1997 21:41:22 -0800
I believe that ADSM must traverse the inodes in order to
determine what has changed, and therefore what must be backed up.
We have similar situations with our SGI file servers, the largest
of which has over three million files. If we were to run against
a single backup process, the overhead in determining what to
backup would take on the order of five hours--before *any* data
was transferred.

I believe the person who suggested VIRTUALMountpoints as an
option was correct--that would reduce the amount of overhead for
any given filespace. It does, however, bring with it drawbacks of
its own, with respect to collocation and some other issues.

I'm not positive, but I think ADSM V3 alleviates this to a
certain extent, by issuing a get attribute call, instead of a
stat call. But it still needs to walk the file system in order to
determine what has to be backed up.

 -- Tom

"If I could dot the 'i' in a Michigan     Thomas A. La Porte
 game and the good lord came to take me   DBA, Feature Animation
 the next day ... at least I could        DreamWorks SKG
 die happy." - Beano Cook, ESPN           <tlaporte AT anim.dreamworks DOT com>

On Tue, 9 Dec 1997, David Hendrix wrote:

>Date:    Mon, 8 Dec 1997 09:00:11 -0700
>From:    David Hendrix <dmhendri AT FEDEX DOT COM>
>Looking for bright ideas:
>        C20 running AIX 4.1.5
>        67GB filesystem with 1 directory and 1.5 million files
> ADSM client
>After 40 hours, the idletimeout kicked in.  The client had not sent ANY
>data.  Thursday of last week, performed a trace and watched it going
>through the inodes (picks a file, then traverses it's inodes).  This is
>what is killing us.  To circumvent this from happening we tried:
>1. removing all filespaces from the server for a "clean" backup
>2. dsmc incr
>3. dsmc incr (slow incremental)
>4. dsmc selective
>It is set to shared static - would dynamic alleviate the problem?
>Has anyone been able to get the client to backup a file system without
>first traversing the inodes?
>V3 allows no query restore, does it have any new backup features which
>alleviates this problem?
>I understand the need to go through the inodes since if a file changes
>during backup it will know this (hence the thought that dynamic might
>help).  At least allow the option of NOT traversing the inodes prior to
>backup.  We have 3-4 Sun/AIX boxes with 1 Million+ files (the others
>actually have directory structures) and this is killing us.
>David Hendrix
>dmhendri AT fedex DOT com
<Prev in Thread] Current Thread [Next in Thread>