This was discussed a while back and you should be able to search it
in the archives. But, in a nut shell, your suggestion is what we did in our
case. We had our developers write scripts that tar up the many little files
in some logical fashion (by date, by directory, by part type, etc.) and then
have TSM back up those files and ignore all the little files. When a restore
is needed, they have a script that restores the tar and uncompresses it to
it's original location.

        The backups take much less time now and the restoration of 20,000
files now takes only 1 tar file and 3 minutes to restore, where as it used
to take hours to restore. Sure, the untar then takes a while to put the
files out there, but it is still much faster than a standard TSM restore.

        In our case, these hosts have millions of files that are written
once and never changed, and then deleted a few days later, so it works well
in our situation. If the existing files are constantly changing though, it
may be more complicated to implement a scheme like this. Versioning of the
files would be next to impossible with a scheme like this. Like, when a user
wants a file restored but has no idea what day it was created, backed up or
deleted,(that never happens, right? ;-), you want to have a way to know
which tar file that file is going to be found in. If you can figure out how
to tar up the files logically, it can work.

Ben Bullock
Unix system manager
Micron Technology Inc.


> -----Original Message-----
> From: Gerald Wichmann [mailto:gwichmann AT SANSIA DOT COM]
> Sent: Tuesday, July 31, 2001 10:38 AM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Backing up lots of small files
>
>
> Has anyone come up with any good strategies on backing up a
> file server with
> millions of small (1KB-10KB) files? Currently we have a
> system with 50GB of
> them that takes about 40 hours to backup. Many of the files change..
>
> I m wondering if anyone has anything in place that works well
> for this sort
> of situation..
>
> Something like this or perhaps something I haven t thought of:
>
> develop script that will tar/compress the files into a single
> file and back
> that single file up daily. Perhaps even search the file
> system by date and
> only backup files that have changed since the last backup
> (this seems like
> it wouldn t be any faster then simply backing them up though)
>