Hi Wanda,

This is a tough one, probably no "one fell swoop" approach to this.

Some things to consider:

- What is the nature of these files that are created every day? Are they of
a "temporary" nature so they can be deleted after a period of time? If not,
once created, how often are they referenced again? Maybe they are a
candidate for archive (and delete after successful archive)?

* Is a standard TSM file backup sufficient to meet this applications needs
should the files need to be restored?

* For that matter, what are the restore requirements? Even if backup was
not an issue, can they all be restored in the event of a disk failure or
other event that requires a full restore?

* One approach could go something like this:
- Use two nodes: one node to back up the rest of the system, the second
node to back up this application directory.

- For the rest of the system, use standard incremental backup.Use
EXCLUDE.DIR to skip the application directory.

- For the application directory, exclude all directories and files except
for the application directory. Then use incremental by date (-incrbydate
option) to back up the application directory.

I'm sure you are well aware of the caveats associated with incremental by
date, but if these files are created once, then never updated again, it
might be a reasonable approach. Caveat: if the files are constantly created
even during the backup operation, then -incrbydate might not be viable
since files created while the backup is running might not get picked up by
that backup, and they won't get picked up by the next backup either.

* Another approach is a variation on the above:
- Use two nodes as above, and back up the rest of the system as above.

- Create a script that scans the application directory for files created
during the last 24-hour period. For each file that was created in the last
24 hours, write it out to a file that can be fed to a command line client
file list backup.

For example, let's say the application directory backup begins at 23:00. So
the script will search for files created between yesterday at 23:00 and
today at 23:00, generate a file list, then kick off the dsmc selective
-filelist=<listfile>.

* Image backup?

Best regards,

Andy Raibeck
IBM Software Group
Tivoli Storage Manager Client Product Development
Level 3 Team Lead
Internal Notes e-mail: Andrew Raibeck/Hartford/IBM@IBMUS
Internet e-mail: storman AT us.ibm DOT com

IBM Tivoli Storage Manager support web page:
http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager

"ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 2011-02-25
14:03:24:

> From: "Prather, Wanda" <wPrather AT ICFI DOT COM>
> To: ADSM-L AT vm.marist DOT edu
> Date: 2011-02-25 14:04
> Subject: Windows servers with a kazillion files and Win2K8...
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu>
>
> I have a site with an application that generates kazillions of tiny
> files that are stored forever.
> I've already yelled about it, but it's a purchased, customer-facing
> black-box app that they really can't change.
> (Naturally, when it was bought umpty years ago, nobody thought about
> the problem reaching this size or what the ramifications would be.)
> Every day the app creates more files.
>
> They have multiple Win2K3 servers that already have multiple luns
> containing over 35M files each, one is over 75M files.
>
> We are using journaling to back them up successfully (most days).
> But it's a struggle just to expand the file tree with Windows
> explorer, and there are exposures on the days when the journal gets
> overrun (takes 72 hours for TSM to scan the filesystem and
> revalidate the journal).
>
> Looking for anything that might help save our bacon.
>
> Has anybody had experience with this issue and Win2K8?
> Does Win2K8 do any better than Win2K3 at handling huge numbers of
> files in 1 NTFS directory?
> Upgrading the OS is something application-independent we might be able to
do.
>
> Thanks for any insight!
> W
>
>
> Wanda Prather  |  Senior Technical Specialist  | wprather AT icfi DOT com<
> mailto:wprather AT icfi DOT com>  |  www.jasi.com<www.jasi.com%20>
> ICF Jacob & Sundstrom  | 401 E. Pratt St, Suite 2214, Baltimore, MD
> 21202 | 410.539.1135