How to find how much of the backup of a file system is completed

[unixandtsm]

Hi all,

Anyway to find out how much of the filesystem is been backedup ?

I am having a full backup job of 22TB filesystem running from about 73 days now. Still not sure how many more days this is going to take ???

[unixandtsm]

Hi all,

Does "bytes recvd" in the output of query session command will say anything about how much of the backup job is complete ?
Currently this query on the backup job says that it received 11.3TB so far, does this mean it completed backup of 11.3TB of data of total 22TB ?

q se 18626 f=d

              Sess Number: 18,626
             Comm. Method: Tcp/Ip
               Sess State: Run
                Wait Time: 0 S
               Bytes Sent: 3.6 M
              Bytes Recvd: 11.3 T
                Sess Type: Node
                 Platform: Linux86
              Client Name: Linux09
      Media Access Status: Current output volumes:  BKUPJA,C01218,(6449 Seconds)
                User Name:
Date/Time First Data Sent: 08/30/12   00:12:05
   Proxy By Storage Agent:

[Trident]

If this is the first backup, and you do not have client compression enabled, then this is true.

[rallingham]

You have to admit that this is a pretty poor transfer rate for the backup. It works out to around 154 GB per day.

[unixandtsm]

Thanks Trident.

Hi rallingham, yes this is very poor transfer rate.
How can i improve things here ?
What are the things i should be checking to improve the TSM performance ?

[rallingham]

Hard to tell right now. I would suspect your network connection as the transfer rate is so poor. But it is impossible to tell without a lot of details. For example are the connections through GB ethernet or lower?

[MrOdysseus]

You are also backing up straight to tape it seems. A few factors to consider
Do you have Collocation defined for that node?
Are there are processes using drives?
The size of the data is a factor as well. Are these millions of objects or a few big files
... and so on

You can determine how many tapes you've used so far by trying something like this, fill in for X

select count(volume_name) as vol_count from volumeusage where node_name='Linux09' and stgpool_name='XXXX'

You can also try

Query OCCupancy node_name

[Trident]

If you have the possibility, you can start multiple threads from the client.

dsmc i d:\dir1\* -subdir=yes
dsmc i d:\dir2\* -subdir=yes
..

And so on.

For small files, direct them to a DISK type devclass.

There are aslo some tuning parameters dsm.opt that can be used.

My favorite is to set TCPWindowSize to 0 (let the os handle windowscaling).

[unixandtsm]

thanks rallingham and Trident,

Hi MrOdysseus,

looks like data is going straight to tapes only (from what i see of q stg commmand's output; the HI and LOW mig of stgpool are being reset to 0 and 0)

How to check if collocation is defined ?

Can you explain little more on "any processess using drives" ? you mean tape drives ? i don't see any process related this particular bacup job but there is a stgpool backup running at the moment and q mount command shows that 7 out of 8 tape drives are being used.

select command shows zero count

select count(volume_name) as vol_count from volumeusage where node_name='Linux09' and stgpool_name='BKUPJA'

  VOL_COUNT
-----------
          0

but below is the q occ for this filespace

query occ Linux09 /DATA

Node Name      Type     Filespace       FSID     Storage        Number of      Physical       Logical
                        Name                     Pool Name          Files         Space         Space
                                                                               Occupied      Occupied
                                                                                   (MB)          (MB)
----------     ----     ----------     -----     ----------     ---------     ---------     ---------
Linux09     Bkup     /DATA         34     BKUPPOOL            72         69.79         69.79
 
 Linux09     Bkup     /DATA         34     BKUPJA         5,012,713     18,976,99     18,972,23
                                                                         9.96          4.40

Linux09     Bkup     /DATA         34     CPOFFJA      4,378,495     16,968,44     16,967,68
                                                                         7.93          8.60
 
Linux09   Bkup     /DATA         34     CPONJA       4,718,526     17,742,62     17,741,86
                                                                          9.29          9.97

[MrOdysseus]

Hi there...
Yes, I wanted to ask if there was anything else using the Tape Drives. You could end up with tape contention, slowing down the backup. I'm not sure if that is the case.

Can you try the other storage pool names? (CPOFFJA, and CPONJA). This will only give you an idea of how many tapes you are using for that node, and check if that number grows.

I take it CPOFF is the offsite copy and CPON is the onsite.

Is BKUPJA the disk storage pool?

It does seem like you've backed up 18TB already.

The way you do collocation is on the Offsite/Onsite copies You would either define to collocate by node or group, those storage pool wide settings.

When you do Q STG CPOFFJA you can see a option for

Collocate?:

Then you can define collocation groups and add members to those groups

q collocg or
q collocg f=d

If the stg is defined to collocate by node, the backups to tape could go faster. I don't recommend making changes as that could lead to heavy reclaims etc... this could be something to watch out for down the road.


Do you have enough scratch tapes? or tapes allocated to the storage pool. It does seem like a very long time to backup 22TB
Is the server itself using on a 100MB NIC card instead of 1GB NIC?


Hope this info helps.

[unixandtsm]

Hi MrOdysseus,

yes, CPOFFJA and CPONJA are offsite and onsite copy pools.
BKUPJA is a primay tape pool and BKUPPOOL is the disk pool.
When i query BKUPJA stgpool, looks like collocation is set to "group".
yes, there are few scratch tapes (12 now). i will always try to keep atleast 10 scratch tapes available.

Does backing up data directly to tape take long time ?
I suspect that the data is going directly to tape instead of first going to disk pool BKUPPOOL (because the HI pct migr and LO pct migr values are being reset every night to 0 and 0. This causing "migration" process to run all the time and also using the tape drives. Every morning i am changing HI and LO values of migration to 90 and 10. I am having hard time to find why this is being reset 0 and 0)

[MrOdysseus]

Hi UnixandTSM ...

Sounds like a scheduled task is resetting it for you.. It could either be an external task or a administrative one.

try q event * t=a or

q event * t=a begind=-1

You could also dig through the logs and see what changing those thresholds for you

q act begint=-24 search=HIghmig


If youh have tons of scripts, you can output everything into a file and search that file.

q script * > c:\scripts.txt

[unixandtsm]

Hi MrOdysseus,

Just an hour ago i updated the stgpool to values HI=90 and LO=10 but then again it reset.
below is the actlog query.

tsm>q actlog begint=-1 search=HIghmig

Date/Time                Message
--------------------     ----------------------------------------------------------
11/12/12   10:19:51      ANR1000I Migration process 31403 started for storage pool
                          BACKUPPOOL automatically, highMig=0, lowMig=0,
                          duration=No. (PROCESS: 31403)

Not sure what is making the migration to run automatically?

[smajl]

q act s=BACKUPPOOL
q act s="update stg"
q act s="PROCESS: 31403"
and look what/who updated the stg pool