How about periodic image backups, with daily journal-based backups to
catch new/changed files?
OS-based filesystems are sometimes abused as databases, and have been
for many years. (e.g. Z/VM VMSES) When that happens, such a filesystem
needs to be backed up more like a database than a filesystem. The answer
there is ISP image backups.
Roger Deschner University of Illinois at Chicago rogerd AT uic DOT edu
======I have not lost my mind -- it is backed up on tape somewhere.=====
On Mon, 20 Mar 2017, Harris, Steven wrote:
>Bo
>
>The problem with small files is that the TSM database entry may well be larger
>than the file you are storing. If your files are less than about 3000 bytes
>that will be the case.
>
>What is happening is that the file system is being used as a database. A
>complex file path becomes the key and the file content is the data.
>I realize this has probably been dumped on you without consultation, but a
>database is probably a better fit. It could be something as simple as a
>key/value store (maybe one per day) or as complex as a document DB like
>Couchbase.
>
>A previous customer of mine did something similar. It was logs of ecommerce
>transactions that averaged about 1500 bytes each and had to be kept for 7
>years. A million transactions a day and growing. They killed a TSM 5.5
>database in 2 years, and when I left were well on the way to killing a TSM 6.3
>database as well. Any requests to alter the application were met with active
>hostility.
>
>Good Luck
>
>Steve
>Steven Harris
>TSM Admin/Consultant
>Canberra Australia
>
>
>
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
>Of Rick Adamson
>Sent: Tuesday, 21 March 2017 12:08 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: [ADSM-L] How to backup Billions of files ??
>
>Bo
>I suggest you provide a few more details about the data and you backup
>environment.
>For example; what is this data, how frequently will it be accessed on average,
>what is its total space requirements, what is the source stored on?
>Type of backup storage; tape, disk, cloud? (specifics) Bandwidth/network speed
>between data and target backup server?
>
>-Rick Adamson
> Jacksonville,Fl.
>
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
>Of Bo Nielsen
>Sent: Monday, March 20, 2017 7:20 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: [ADSM-L] How to backup Billions of files ??
>
>Hi TSM's
>
>I have earlier asked for help with archiving of 80 Billion very small files,
>But now they want the files backed up. They expect an average change rate of 3
>percent/Month.
>
>Anyone with experience of such an exercise, and will share it with me??
>
>Regards
>
>Bo
>
>
>Bo Nielsen
>
>
>IT Service
>
>
>
>Technical University of Denmark
>
>IT Service
>
>Frederiksborgvej 399
>
>Building 109
>
>DK - 4000 Roskilde
>
>Denmark
>
>Mobil +45 2337 0271
>
>boanie AT dtu DOT dk<mailto:boanie AT dtu DOT dk>
>
>This message and any attachment is confidential and may be privileged or
>otherwise protected from disclosure. You should immediately delete the message
>if you are not the intended recipient. If you have received this email by
>mistake please delete it from your system; you should not copy the message or
>disclose its content to anyone.
>
>This electronic communication may contain general financial product advice but
>should not be relied upon or construed as a recommendation of any financial
>product. The information has been prepared without taking into account your
>objectives, financial situation or needs. You should consider the Product
>Disclosure Statement relating to the financial product and consult your
>financial adviser before making a decision about whether to acquire, hold or
>dispose of a financial product.
>
>For further details on the financial product please go to http://www.bt.com.au
>
>Past performance is not a reliable indicator of future performance.
>
|
