It isn't the backup that will kill you, it is the restore...
Trust me... if you have over 1.5 Million files in a mount point, expect
weeks or months to perform a full restore.

Remember, just because you can put 20 million files in a mount point doesn't
mean it is a good idea... discourage that at all costs.  Also warn the end
user that their data is, for all practical purposes, unrecoverable.  Let's
face it, a business couldn't endure a critical business function outage of 6
or 8 weeks.

 If they insist on that configuration (20M files per mount point) you'll
have to protect that by something such as Image backup.  If they can't
endure the outage to do that, you'll need to utilize SAN disk on the back
end to create a duplicate copy of the mount point and then back that up with
image backup.

Period...


Dwight

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Mehdi Salehi
Sent: Wednesday, April 29, 2009 11:28 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: [ADSM-L] best backup method for millions of small files?

Hi,
There is an NTFS filesystem in windows with 20 millions of small files less
than 20 KB each. The total size of the filesystem is less than 300GB.
I have not tried incremental backup, because that would be a nightmare.
Again, inc backup is not a good choice because the long restoration time is
not acceptable us.
I tested the snapshot image backup and satisfied with backup time. Now the
problem is how to perform the incremental for this image backup.
Unfortunately "incremental-by-date of last image" scans the whole files to
filter which files must be in the backup list and would take a tremendous
amount of time. This is not acceptable as well.
The ideal behavior that I expect from TSM is journal based incremental
backup for image which is apparently not supported by TSM 5.3.
more info:
-the change rate for this filesystem is about 20 thousands of files per day
-TSM version is 5.3

What do you recommend for this case? Please include TSM 6.1 featues if
exist.

Regards,
Mehdi Salehi