Backup NFS filesystems with millions of files every day

Mikky83

Newcomer
Joined
Sep 13, 2018
Messages
2
Reaction score
0
Points
0
PREDATAR Control23

Hi Guys,

We have daily backups on nfs filesystems. Total amount of files is more than 50m. Unfortunately we cannot use journal, because it's not supported on nfs.
IBM Spectrum Protect server version is 8.1, BA client version is 8.1, OS is AIX 7.2 running on Power8, 4 virtual CPUs and 40GB of RAM. Server was installed from scratch (no version upgrades)
The problem/challenge that we are facing is how to complete this backup in daily cycle. Sometimes processing of the files is taking too much time, sometimes is faster. Performance of the server is stable, it seems that app doesn't use all available resources of the server.
I've attached output from the report of the last two days. yesterday the backup lasted 20 hours, and today 14 hours (similar amount of processed files, backed up and expired objects)
The other problem lately is expiration process (once per week) which is taking more than 14 hours.
DB size is around 300GB.

I'm wondering is this the maximum product can do, or something can be done to improve the performance.
Thanks in advance.
 

Attachments

  • tsm_report.txt
    2.5 KB · Views: 8
PREDATAR Control23

For faster backups, if your storage/network can handle it check out http://www.general-storage.com/PRODUCTS/dsmISI-MAGS/dsmisi-mags.html
I have used that product in the past with TSM and works great (Windows shares, never tried for NFS). Restores however, you should use standard client.

If the above isn't an option, you could look at setting up virtual mount points within the tsm client (Unix, Linux only no OS X or WIndows).
https://www.ibm.com/support/knowled...irtualmountpoint.html#r_opt_virtualmountpoint
I've had success with that for a few tough NFS mounts. But, no where near as many files as you have.

Why are you running expiration once per week? Generally one would want to run it once per day. Also, what other processes are running on the server when expire inventory runs? I found that a busy server can impact expire inventory time. Also, look at adjusting the resource utilization of the expire inv process. From the help:
REsource
Specifies the number of threads that can run in parallel. Specify a
value in the range 1 - 40. This parameter is optional. The default is
four.

Expiration runs as a single process, although the resources represent
parallel work by the server within the single expiration process.
Archive data for a node runs only on a single resource, but backup
data can be spread across resources on a file space level. For
example, if you specify NODE=X,Y,Z each with three file spaces and
RESOURCE=5, then expiration processing for the three X, Y, and Z
client nodes runs in parallel. At least one resource processes each
node, and at least one node uses multiple resources for processing
backup data across the multiple file spaces.

Hope it helps.
 
PREDATAR Control23

I've attached output from the report of the last two days. yesterday the backup lasted 20 hours, and today 14 hours (similar amount of processed files, backed up and expired objects)
If CPU and memory are not highly utilized on the client, try increasing the resourceutilization.
Is that one large filesystem? If so, try virutalmountpoint like RecoveryOne said.
If you have an anti-virus, make sure it doesn't scan files opened for backups.

Look at dsminstr.log to see where most of the time is spent. You will see several threads for a backup, some will do queries and processing, the others will be sending data to the server. I suspect "Process Dirs " will have a lot of time spent there (Actual seconds), this is where virtualmountpoint and higher resourceutil may help as it will increase the amount of work done in parallel.

The other problem lately is expiration process (once per week) which is taking more than 14 hours.
DB size is around 300GB.
Expiration is primarily DB intensive. The database should be on a few filesystems and using the fastest disks you can afford. And I agree with RecoveryOne, run that daily. Every day should look something like this:
daily_tape_sched_cycle.gif
 
PREDATAR Control23

We are using dsmISI-MAGS with TSM and it works great for SMB and NFS shares.
 
Top