Bo

The problem with small files is that the TSM database entry may well be larger 
than the file you are storing. If your files are less than about 3000 bytes 
that will be the case.  

What is happening is that the file system is being used as a database.  A 
complex file path becomes the key and the file content is the data.
I realize this has probably been dumped on you without consultation, but a 
database is probably a better fit.  It could be something as simple as a 
key/value store (maybe one per day) or as complex as a document DB like 
Couchbase.

A previous customer of mine did something similar.  It was logs of ecommerce 
transactions that averaged about 1500 bytes each and had to be kept for 7 
years.  A million transactions a day and growing.  They killed a TSM 5.5 
database in 2 years, and when I left were well on the way to killing a TSM 6.3 
database as well.  Any requests to alter the application were met with active 
hostility.

Good Luck

Steve
Steven Harris
TSM Admin/Consultant 
Canberra Australia

   



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Rick Adamson
Sent: Tuesday, 21 March 2017 12:08 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] How to backup Billions of files ??

Bo
I suggest you provide a few more details about the data and you backup 
environment.
For example; what is this data, how frequently will it be accessed on average, 
what is its total space requirements, what is the source stored on?
Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed 
between data and target backup server?

-Rick Adamson
 Jacksonville,Fl.
 


-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Bo Nielsen
Sent: Monday, March 20, 2017 7:20 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: [ADSM-L] How to backup Billions of files ??

Hi TSM's

I have earlier asked for help with archiving of 80 Billion very small files, 
But now they want the files backed up. They expect an average change rate of 3 
percent/Month.

Anyone with experience of such an exercise, and will share it with me??

Regards

Bo


Bo Nielsen


IT Service



Technical University of Denmark

IT Service

Frederiksborgvej 399

Building 109

DK - 4000 Roskilde

Denmark

Mobil +45 2337 0271

boanie AT dtu DOT dk<mailto:boanie AT dtu DOT dk>

This message and any attachment is confidential and may be privileged or 
otherwise protected from disclosure. You should immediately delete the message 
if you are not the intended recipient. If you have received this email by 
mistake please delete it from your system; you should not copy the message or 
disclose its content to anyone. 

This electronic communication may contain general financial product advice but 
should not be relied upon or construed as a recommendation of any financial 
product. The information has been prepared without taking into account your 
objectives, financial situation or needs. You should consider the Product 
Disclosure Statement relating to the financial product and consult your 
financial adviser before making a decision about whether to acquire, hold or 
dispose of a financial product. 

For further details on the financial product please go to http://www.bt.com.au 

Past performance is not a reliable indicator of future performance.