I can't see any file-based backups working well in this case.  Depending on the 
filesystem used & size of volumes, I would lean toward a block-based backup 
product (e.g. Acronis).  Or multiple layers of replication & snapshots if 
feasible.  ISP might be a nice hammer, but this is not a nail.

Steve Schaub
Systems Engineer II, Backup/Recovery
Blue Cross Blue Shield of Tennessee

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Roger Deschner
Sent: Wednesday, March 22, 2017 7:24 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] How to backup Billions of files ??

How about periodic image backups, with daily journal-based backups to catch 
new/changed files?

OS-based filesystems are sometimes abused as databases, and have been for many 
years. (e.g. Z/VM VMSES) When that happens, such a filesystem needs to be 
backed up more like a database than a filesystem. The answer there is ISP image 
backups.

Roger Deschner      University of Illinois at Chicago     rogerd AT uic DOT edu
======I have not lost my mind -- it is backed up on tape somewhere.=====


On Mon, 20 Mar 2017, Harris, Steven wrote:

>Bo
>
>The problem with small files is that the TSM database entry may well be larger 
>than the file you are storing. If your files are less than about 3000 bytes 
>that will be the case.
>
>What is happening is that the file system is being used as a database.  A 
>complex file path becomes the key and the file content is the data.
>I realize this has probably been dumped on you without consultation, but a 
>database is probably a better fit.  It could be something as simple as a 
>key/value store (maybe one per day) or as complex as a document DB like 
>Couchbase.
>
>A previous customer of mine did something similar.  It was logs of ecommerce 
>transactions that averaged about 1500 bytes each and had to be kept for 7 
>years.  A million transactions a day and growing.  They killed a TSM 5.5 
>database in 2 years, and when I left were well on the way to killing a TSM 6.3 
>database as well.  Any requests to alter the application were met with active 
>hostility.
>
>Good Luck
>
>Steve
>Steven Harris
>TSM Admin/Consultant
>Canberra Australia
>
>
>
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
>Of Rick Adamson
>Sent: Tuesday, 21 March 2017 12:08 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: [ADSM-L] How to backup Billions of files ??
>
>Bo
>I suggest you provide a few more details about the data and you backup 
>environment.
>For example; what is this data, how frequently will it be accessed on average, 
>what is its total space requirements, what is the source stored on?
>Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed 
>between data and target backup server?
>
>-Rick Adamson
> Jacksonville,Fl.
>
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
>Of Bo Nielsen
>Sent: Monday, March 20, 2017 7:20 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: [ADSM-L] How to backup Billions of files ??
>
>Hi TSM's
>
>I have earlier asked for help with archiving of 80 Billion very small files, 
>But now they want the files backed up. They expect an average change rate of 3 
>percent/Month.
>
>Anyone with experience of such an exercise, and will share it with me??
>
>Regards
>
>Bo
>
>
>Bo Nielsen
>
>
>IT Service
>
>
>
>Technical University of Denmark
>
>IT Service
>
>Frederiksborgvej 399
>
>Building 109
>
>DK - 4000 Roskilde
>
>Denmark
>
>Mobil +45 2337 0271
>
>boanie AT dtu DOT dk<mailto:boanie AT dtu DOT dk>
>
>This message and any attachment is confidential and may be privileged or 
>otherwise protected from disclosure. You should immediately delete the message 
>if you are not the intended recipient. If you have received this email by 
>mistake please delete it from your system; you should not copy the message or 
>disclose its content to anyone.
>
>This electronic communication may contain general financial product advice but 
>should not be relied upon or construed as a recommendation of any financial 
>product. The information has been prepared without taking into account your 
>objectives, financial situation or needs. You should consider the Product 
>Disclosure Statement relating to the financial product and consult your 
>financial adviser before making a decision about whether to acquire, hold or 
>dispose of a financial product.
>
>For further details on the financial product please go to 
>http://www.bt.com.au
>
>Past performance is not a reliable indicator of future performance.
>

------------------------------------------------------------------------------
Please see the following link for the BlueCross BlueShield of Tennessee E-mail 
disclaimer:  http://www.bcbst.com/email_disclaimer.shtm