• Please help support our sponsors by considering their products and services.
    Our sponsors enable us to serve you with this high-speed Internet connection and fast webservers you are currently using at ADSM.ORG.
    They support this free flow of information and knowledge exchange service at no cost to you.

    Please welcome our latest sponsor Tectrade . We can show our appreciation by learning more about Tectrade Solutions
  • Community Tip: Please Give Thanks to Those Sharing Their Knowledge.

    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.

  • Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)

    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Backup NFS filesystems with millions of files every day

#1
Hi Guys,

We have daily backups on nfs filesystems. Total amount of files is more than 50m. Unfortunately we cannot use journal, because it's not supported on nfs.
IBM Spectrum Protect server version is 8.1, BA client version is 8.1, OS is AIX 7.2 running on Power8, 4 virtual CPUs and 40GB of RAM. Server was installed from scratch (no version upgrades)
The problem/challenge that we are facing is how to complete this backup in daily cycle. Sometimes processing of the files is taking too much time, sometimes is faster. Performance of the server is stable, it seems that app doesn't use all available resources of the server.
I've attached output from the report of the last two days. yesterday the backup lasted 20 hours, and today 14 hours (similar amount of processed files, backed up and expired objects)
The other problem lately is expiration process (once per week) which is taking more than 14 hours.
DB size is around 300GB.

I'm wondering is this the maximum product can do, or something can be done to improve the performance.
Thanks in advance.
 

Attachments

RecoveryOne

ADSM.ORG Senior Member
#2
For faster backups, if your storage/network can handle it check out http://www.general-storage.com/PRODUCTS/dsmISI-MAGS/dsmisi-mags.html
I have used that product in the past with TSM and works great (Windows shares, never tried for NFS). Restores however, you should use standard client.

If the above isn't an option, you could look at setting up virtual mount points within the tsm client (Unix, Linux only no OS X or WIndows).
https://www.ibm.com/support/knowled...irtualmountpoint.html#r_opt_virtualmountpoint
I've had success with that for a few tough NFS mounts. But, no where near as many files as you have.

Why are you running expiration once per week? Generally one would want to run it once per day. Also, what other processes are running on the server when expire inventory runs? I found that a busy server can impact expire inventory time. Also, look at adjusting the resource utilization of the expire inv process. From the help:
REsource
Specifies the number of threads that can run in parallel. Specify a
value in the range 1 - 40. This parameter is optional. The default is
four.

Expiration runs as a single process, although the resources represent
parallel work by the server within the single expiration process.
Archive data for a node runs only on a single resource, but backup
data can be spread across resources on a file space level. For
example, if you specify NODE=X,Y,Z each with three file spaces and
RESOURCE=5, then expiration processing for the three X, Y, and Z
client nodes runs in parallel. At least one resource processes each
node, and at least one node uses multiple resources for processing
backup data across the multiple file spaces.
Hope it helps.
 

marclant

ADSM.ORG Moderator
#3
I've attached output from the report of the last two days. yesterday the backup lasted 20 hours, and today 14 hours (similar amount of processed files, backed up and expired objects)
If CPU and memory are not highly utilized on the client, try increasing the resourceutilization.
Is that one large filesystem? If so, try virutalmountpoint like RecoveryOne said.
If you have an anti-virus, make sure it doesn't scan files opened for backups.

Look at dsminstr.log to see where most of the time is spent. You will see several threads for a backup, some will do queries and processing, the others will be sending data to the server. I suspect "Process Dirs " will have a lot of time spent there (Actual seconds), this is where virtualmountpoint and higher resourceutil may help as it will increase the amount of work done in parallel.

The other problem lately is expiration process (once per week) which is taking more than 14 hours.
DB size is around 300GB.
Expiration is primarily DB intensive. The database should be on a few filesystems and using the fastest disks you can afford. And I agree with RecoveryOne, run that daily. Every day should look something like this:
 

mikeymac

ADSM.ORG Moderator
#5
Hi Guys,

We have daily backups on nfs filesystems. Total amount of files is more than 50m. Unfortunately we cannot use journal, because it's not supported on nfs.
IBM Spectrum Protect server version is 8.1, BA client version is 8.1, OS is AIX 7.2 running on Power8, 4 virtual CPUs and 40GB of RAM. Server was installed from scratch (no version upgrades)
The problem/challenge that we are facing is how to complete this backup in daily cycle. Sometimes processing of the files is taking too much time, sometimes is faster. Performance of the server is stable, it seems that app doesn't use all available resources of the server.
I've attached output from the report of the last two days. yesterday the backup lasted 20 hours, and today 14 hours (similar amount of processed files, backed up and expired objects)
The other problem lately is expiration process (once per week) which is taking more than 14 hours.
DB size is around 300GB.

I'm wondering is this the maximum product can do, or something can be done to improve the performance.
Thanks in advance.

You didn't mention where the NFS export is coming from. Is it coming from a NAS, or from another *NIX server?
 

Advertise at ADSM.ORG

If you are reading this, so are your potential customer. Advertise at ADSM.ORG right now.

UpCloud high performance VPS at $5/month

Get started with $25 in credits on Cloud Servers. You must use link below to receive the credit. Use the promo to get upto 5 month of FREE Linux VPS.

The Spectrum Protect TLA (Three-Letter Acronym): ISP or something else?

  • Every product needs a TLA, Let's call it ISP (IBM Spectrum Protect).

    Votes: 9 16.4%
  • Keep using TSM for Spectrum Protect.

    Votes: 32 58.2%
  • Let's be formal and just say Spectrum Protect

    Votes: 9 16.4%
  • Other (please comement)

    Votes: 5 9.1%

Forum statistics

Threads
31,150
Messages
132,626
Members
21,324
Latest member
h3llb0y
Top