Help with Ridiculously long backup times

[poolgod2004]

All



I posted a question in the scripting area about possibly utilizing the FILELIST option in conjunction with a script to find what files changed as I have backups that run sometimes in the range of 12-18Hrs in all mostly due to the sheer # of files being scanned in the filesystems which can be in excess of 10Million objects.



One of the persons in the post who replied said something about possibly running an individual TSM client for each and every filesystem where you run at least one tsm client per filesystem with ressourceutil set to 12.



Can anyone possibly explain this further to me and also if its valid and can be done can you provide me with a way to do it. My question to this would be how to map out a TSM client for each and every filesystem mounted, preferably an automated approach to it.



Thanks again for the assistance in advance.

 

[chad_small]

I would recommend making each large filesystem its own node to facilitate faster restore times. As for backups I would recommend you use the -incrbydate option. It can exponentially speed up backups since it only reviews the date and not the other numerous attributes of the file to validate it for backup. It has its issues but overall it can greatly help with backup speed. Just make sure you periodically do a normal incremental to pickup any changes outside the date parameter. Also 5.2.3 of AIX client is supposed to support Journal backups. If the client is NT you should be using the journalling option already.

 

[poolgod2004]

Thank you chad



I guess I am still confused in some respects, 1st of all I am not sure how I would go about setting up individual nodes for each filesystem (never have done that b4) and secondly where would I stick the -incrbydate option in at, would that go in the DSM.SYS file.



Just for clarification, about 80% of the TSM servers and clients are running on AIX and the other 20% are linux based, no windows clients at all being backed up. I do believe that linux with TSM does not support Journal based backups at this time but hopefully it will do so in the future.



In any case if you could provide some more details on this I would appreciate it.



BTW: in some cases on some of my TSM environments I have over 30 Filesystems mounted.



Thanks again chad, I do appreciate your help.

 

[chad_small]

First you register new names in TSM, so SERVERA would have another node called SERVERA_FS1. For unix you need to edit environmental settings in a script to point to a different dsm.opt that states a different servernam stanza in the dsm.sys and also lists just the one filesystem in the domain statement. It sounds harder to setup than it really is. For NT you can just create a new dsm.opt then point to the specific dsm.opt file for each filesystem (we do this with our clusters). In each case make sure you point to different dsmsched.log and dsmerror.log files (don't want to overwrite them). Here is an example of a dsm.sys with multiple SERVERNAME stanzas:



SErvername TSM01

MANAGEDSERVICES WEBCLIENT

COMMmethod TCPip

TCPPort 1500

TCPServeraddress 131.127.245.31

TCPWINDOWSIZE 256

TCPBUFFSIZE 16

TXNBYTELIMIT 2097152

TCPNODELAY YES

LARGECOMMBUFFERS YES

NODENAME SERVERA

passwordaccess generate

SCHEDLOGRETENTION 7

SCHEDLOGNAME /tmp/dsmsched.log

ERRORLOGRETENTION 7

ERRORLOGNAME /tmp/dsmerror.log

Inclexcl /usr/tivoli/tsm/client/ba/bin/inclexcl.list

QUERYSCHED 1

schedmode PROMPT





SErvername TSM02

COMMmethod TCPip

TCPPort 1500

TCPServeraddress 131.127.245.31

TCPWINDOWSIZE 256

TCPBUFFSIZE 16

TXNBYTELIMIT 2097152

TCPNODELAY YES

LARGECOMMBUFFERS YES

NODENAME SERVERA_FS1

passwordaccess generate

SCHEDLOGRETENTION 7

SCHEDLOGNAME ./dsmsched.log

ERRORLOGRETENTION 7

ERRORLOGNAME ./dsmerror.log

QUERYSCHED 1

schedmode PROMPT





What the scheduler does is start using one of the stanzas that you choose in the dsm.opt file that was referenced. So your UNIX script would look like this:



#!/bin/ksh



EXPORT DSM_CONFIG=/usr/tivoli/tsm/client/ba/bin/dsm_fs1.opt



/usr/tivoli/tsm/client/ba/bin/dsmc sched 2> /dev/null &



This will switch TSM from using the default dsm.opt file and use the one stated. In the one stated you would put the following:



SERVERNAME TSM02

DOMAIN /FS1



You might have to tell TSM NOT to backup the other FS's by listing them with a - in front of them like -/FS2 . This process will start a scheduler process under the second stanza in the dsm.sys file using the alternate nodename and have it backup only the /FS1 filesystem. For NT its similar but not as hard to setup.

 

[chad_small]

Almost forgot....for incremental by date you set a parameter on the command line or in your schedule. The option is -incrbydate and you put it on the command line when doing a backup or in a client schedule you put it in the options line with the - included. I'll give an example when doing an immediate action schedule.



def clientact NODENAME DO=* ACT=INC OPT=-INCRBYDATE



When using the GUI the -incrbydate option is the backup type called Incremental (Date Only) from the drop down menu.

 

[poolgod2004]

This is great chad, just a quick couple of questions about setting it up:



1) the incrbydate option - would I just add a line like this to my dsm.sys file: I am also pasting in my dsm.opt and sys files here so that you can see what I have.



incrbydate yes



2) What kind of pitfalls/problems arise from using that.



cat dsm.sys

************************************************************************

* Tivoli Storage Manager *

* *

* Sample Client System Options file for AIX and SunOS (dsm.sys.smp) *

************************************************************************



* This file contains the minimum options required to get started

* using TSM. Copy dsm.sys.smp to dsm.sys. In the dsm.sys file,

* enter the appropriate values for each option listed below and

* remove the leading asterisk (*) for each one.



* If your client node communicates with multiple TSM servers, be

* sure to add a stanza, beginning with the SERVERNAME option, for

* each additional server.



************************************************************************



SErvername TSM

COMMmethod TCPip

TCPPort 1500

TCPServeraddress 10.10.151.97

TCPWindowsize 32

TCPBuffsize 32

LargeCommBuffers no

passwordaccess generate

schedmode polling

querysch 1

txnbytelimit 25600

memoryefficientbackup no

inclexcl /opt/tivoli/tsm/client/ba/bin/inclexcl.txt

schedlogname /opt/tivoli/tsm/client/ba/bin/dsmsched.log

errorlogname /opt/tivoli/tsm/client/ba/bin/dsmerror.log

schedlogretention 4

errorlogretention 7

ManagedServices Schedule WebClient

PreSchedulecmd "/opt/tivoli/tsm/client/ba/bin/precmd.sh"

* PostSchedulecmd "/opt/tivoli/tsm/client/ba/bin/postcmd.sh"

resourceutilization 10

nodename tsmemag





dsm.opt

************************************************************************

* Tivoli Storage Manager *

* *

* Sample Client User Options file for AIX and SunOS (dsm.opt.smp) *

************************************************************************



* This file contains an option you can use to specify the TSM

* server to contact if more than one is defined in your client

* system options file (dsm.sys). Copy dsm.opt.smp to dsm.opt.

* If you enter a server name for the option below, remove the

* leading asterisk (*).



************************************************************************



* SErvername A server name defined in the dsm.sys file

servername TSM

domain all-local all-nfs





Thanks again with the assistance. I really appreciate it.

 

[PJ]

You'll want to be VERY careful with incrbydate. Since it only receives the timestamp of the last backup, it will NOT find newly created files or even directories with old timestamps (yes, these exist) and moved or renamed files nor will it detect deleted objects and consequently will never expire anything from you repository besides extra versions. I'd follow the multiple-node-approach or JB Backup or something completely different like snapshots or even full backups. If all that seems impossible and you end up using incrbydate, try to slip in a "normal" incremental at least once a week or so.



Peter