A copy for the list - seems like an issue that we'd all be interested in.  A
reply from a Symantec employee is below, Ed & my replies are below that.

-----Original Message-----
From: Steven Cashman 
To: Ed Wilts; Mark Donaldson
Subject: RE: [Veritas-bu] Exclude then search or vice-versa?


Ed and Mark

Question - Does NB, to anybody's knowledge, build a list of all changed
files then apply the exclusion rule or does it do the opposite.

Answer - From my Exp its neither, it never builds a complete list of all
changed files. But maybe little of both

NBU uses a MS call (on windows) to get a starting file / folder. This
structure is read and each file is compared against our exclude list. If
the file / folder is set to be backed up (archive bit set) and the file
is not in the exclude list it is backed up. We then move onto the next
file / folder and repeat the process.

So even when you exclude a large sub directory from backups NBU still
has to scan each file under that dir to see if it should back it up. I
would not consider this a bug because you can exclude /parent/sub1 yet
put an include in the same policy for /parent/sub1/exec/ So NBU has to
scan each file / folder and compare it against our include and exclude.
If NBU was changed to not scan a directory when it meet a parent exclude
this would not be possible.

The only thing I can think of is trying to move the folders with many
small files to another drive so they can be backed up that way.

Hope that helps


....to which Ed replied:
-----Original Message-----

Thanks for the explanation Steve.  Here's my problem though, and I'm
guessing that Mark's is similar.

RWare, a commercial product, creates hundreds of thousands of small
(mostly empty) files in /var/tmp on Solaris.  I need to backup /var
which should take well under an hour.  Because of all of the files in
/var/tmp, the backup takes well over 16 hours even though I'm excluding
/var/tmp.  Obviously I'm taking a hit on the client doing the
comparison and wasting resources.  How can I speed this backup up?

Thanks again,
        .../Ed

....and I countered....
-----Original Message-----

My product is Documentum but the concept is right on.  Our Docbase is huge,
growing fast, and fits the millions of small files profile.

I have no include list for this policy.  The contents of /parent are
variable but I always want to exclude backing up /parent/sub1.  Backing up
an open Docbase for Documentum is of no value - this size of docbase has to
be done cold.  Both Master & Client are Unix (The same server, in fact).

A workaround I can I can see is to specify, for my example, the /parent/sub3
directly in the filelist for the policy and not sub1 & sub2.  The problem
is, of course, that files located directly in /parent won't be backed up (A
problem for Ed's "/var" situation) and now I've got a real maintenance
problem whenever the contents of /parent change at all.

Another work-around would be to not backup the entire filesystem, opting for
a script generated list of items to be backed up, followed in the end by a
user-backup.  Fulls & incrementals become manually defined.  This would be a
major hassle, of course, and subverts the reason to buy any enterprise
backup product.

A partial "fix" to this would be to prevent the scan of excluded directories
if there's no applicable include file for the backup.  In my experience, the
use of an include file is pretty rare so this might be an effective
optimazation.  In the event an include file exists, then do the subdir scan
& compare as required.

-M

...and that's all there is so far...