1. Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING) Click the link to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This message will disappear after you have made at least 12 posts. Thank you for your cooperation.

How to find how much of the backup of a file system is completed

Discussion in 'Backup / Archive Discussion' started by unixandtsm, Nov 1, 2012.

  1. unixandtsm

    unixandtsm New Member

    Joined:
    Feb 9, 2011
    Messages:
    307
    Likes Received:
    0
    Hi all,

    Anyway to find out how much of the filesystem is been backedup ?

    I am having a full backup job of 22TB filesystem running from about 73 days now. Still not sure how many more days this is going to take ???
     
  2.  
  3. unixandtsm

    unixandtsm New Member

    Joined:
    Feb 9, 2011
    Messages:
    307
    Likes Received:
    0
    Hi all,

    Does "bytes recvd" in the output of query session command will say anything about how much of the backup job is complete ?
    Currently this query on the backup job says that it received 11.3TB so far, does this mean it completed backup of 11.3TB of data of total 22TB ?

    Code:
    q se 18626 f=d
    
                  Sess Number: 18,626
                 Comm. Method: Tcp/Ip
                   Sess State: Run
                    Wait Time: 0 S
                   Bytes Sent: 3.6 M
                 [B] Bytes Recvd: 11.3 T[/B]
                    Sess Type: Node
                     Platform: Linux86
                  Client Name: Linux09
          Media Access Status: Current output volumes:  BKUPJA,C01218,(6449 Seconds)
                    User Name:
    Date/Time First Data Sent: 08/30/12   00:12:05
       Proxy By Storage Agent:
    
    
     
  4. Trident

    Trident Senior Member

    Joined:
    Apr 2, 2007
    Messages:
    289
    Likes Received:
    27
    Occupation:
    IT operations
    Location:
    Oslo, Norway
    If this is the first backup, and you do not have client compression enabled, then this is true.
     
  5. rallingham

    rallingham Senior Member

    Joined:
    Dec 16, 2003
    Messages:
    516
    Likes Received:
    22
    Occupation:
    Sr. TSM Solutions Specialist
    Location:
    Greater Toronto Area Canada
    You have to admit that this is a pretty poor transfer rate for the backup. It works out to around 154 GB per day.
     
  6. unixandtsm

    unixandtsm New Member

    Joined:
    Feb 9, 2011
    Messages:
    307
    Likes Received:
    0
    Thanks Trident.

    Hi rallingham, yes this is very poor transfer rate.
    How can i improve things here ?
    What are the things i should be checking to improve the TSM performance ?
     
  7. rallingham

    rallingham Senior Member

    Joined:
    Dec 16, 2003
    Messages:
    516
    Likes Received:
    22
    Occupation:
    Sr. TSM Solutions Specialist
    Location:
    Greater Toronto Area Canada
    Hard to tell right now. I would suspect your network connection as the transfer rate is so poor. But it is impossible to tell without a lot of details. For example are the connections through GB ethernet or lower?
     
  8. MrOdysseus

    MrOdysseus New Member

    Joined:
    May 16, 2012
    Messages:
    34
    Likes Received:
    1
    Occupation:
    SAN/TSM Administrator
    Location:
    TSM World
    You are also backing up straight to tape it seems. A few factors to consider
    Do you have Collocation defined for that node?
    Are there are processes using drives?
    The size of the data is a factor as well. Are these millions of objects or a few big files
    ... and so on

    You can determine how many tapes you've used so far by trying something like this, fill in for X

    select count(volume_name) as vol_count from volumeusage where node_name='Linux09' and stgpool_name='XXXX'

    You can also try

    Query OCCupancy node_name
     
  9. Trident

    Trident Senior Member

    Joined:
    Apr 2, 2007
    Messages:
    289
    Likes Received:
    27
    Occupation:
    IT operations
    Location:
    Oslo, Norway
    If you have the possibility, you can start multiple threads from the client.

    dsmc i d:\dir1\* -subdir=yes
    dsmc i d:\dir2\* -subdir=yes
    ..

    And so on.

    For small files, direct them to a DISK type devclass.

    There are aslo some tuning parameters dsm.opt that can be used.

    My favorite is to set TCPWindowSize to 0 (let the os handle windowscaling).
     
  10. unixandtsm

    unixandtsm New Member

    Joined:
    Feb 9, 2011
    Messages:
    307
    Likes Received:
    0
    thanks rallingham and Trident,

    Hi MrOdysseus,

    looks like data is going straight to tapes only (from what i see of q stg commmand's output; the HI and LOW mig of stgpool are being reset to 0 and 0)

    How to check if collocation is defined ?

    Can you explain little more on "any processess using drives" ? you mean tape drives ? i don't see any process related this particular bacup job but there is a stgpool backup running at the moment and q mount command shows that 7 out of 8 tape drives are being used.

    select command shows zero count
    Code:
    select count(volume_name) as vol_count from volumeusage where node_name='Linux09' and stgpool_name='BKUPJA'
    
      VOL_COUNT
    -----------
              0
    
    
    but below is the q occ for this filespace

    Code:
    query occ Linux09 /DATA
    
    Node Name      Type     Filespace       FSID     Storage        Number of      Physical       Logical
                            Name                     Pool Name          Files         Space         Space
                                                                                   Occupied      Occupied
                                                                                       (MB)          (MB)
    ----------     ----     ----------     -----     ----------     ---------     ---------     ---------
    Linux09     Bkup     /DATA         34     BKUPPOOL            72         69.79         69.79
     
     Linux09     Bkup     /DATA         34     BKUPJA         5,012,713     18,976,99     18,972,23
                                                                             9.96          4.40
    
    Linux09     Bkup     /DATA         34     CPOFFJA      4,378,495     16,968,44     16,967,68
                                                                             7.93          8.60
     
    Linux09   Bkup     /DATA         34     CPONJA       4,718,526     17,742,62     17,741,86
                                                                              9.29          9.97
     
    
     
  11. MrOdysseus

    MrOdysseus New Member

    Joined:
    May 16, 2012
    Messages:
    34
    Likes Received:
    1
    Occupation:
    SAN/TSM Administrator
    Location:
    TSM World
    Hi there...
    Yes, I wanted to ask if there was anything else using the Tape Drives. You could end up with tape contention, slowing down the backup. I'm not sure if that is the case.

    Can you try the other storage pool names? (CPOFFJA, and CPONJA). This will only give you an idea of how many tapes you are using for that node, and check if that number grows.

    I take it CPOFF is the offsite copy and CPON is the onsite.

    Is BKUPJA the disk storage pool?

    It does seem like you've backed up 18TB already.

    The way you do collocation is on the Offsite/Onsite copies You would either define to collocate by node or group, those storage pool wide settings.

    When you do Q STG CPOFFJA you can see a option for

    Collocate?:

    Then you can define collocation groups and add members to those groups

    q collocg or
    q collocg f=d

    If the stg is defined to collocate by node, the backups to tape could go faster. I don't recommend making changes as that could lead to heavy reclaims etc... this could be something to watch out for down the road.


    Do you have enough scratch tapes? or tapes allocated to the storage pool. It does seem like a very long time to backup 22TB
    Is the server itself using on a 100MB NIC card instead of 1GB NIC?


    Hope this info helps.
     
  12. unixandtsm

    unixandtsm New Member

    Joined:
    Feb 9, 2011
    Messages:
    307
    Likes Received:
    0
    Hi MrOdysseus,

    yes, CPOFFJA and CPONJA are offsite and onsite copy pools.
    BKUPJA is a primay tape pool and BKUPPOOL is the disk pool.
    When i query BKUPJA stgpool, looks like collocation is set to "group".
    yes, there are few scratch tapes (12 now). i will always try to keep atleast 10 scratch tapes available.

    Does backing up data directly to tape take long time ?
    I suspect that the data is going directly to tape instead of first going to disk pool BKUPPOOL (because the HI pct migr and LO pct migr values are being reset every night to 0 and 0. This causing "migration" process to run all the time and also using the tape drives. Every morning i am changing HI and LO values of migration to 90 and 10. I am having hard time to find why this is being reset 0 and 0)
     
  13. MrOdysseus

    MrOdysseus New Member

    Joined:
    May 16, 2012
    Messages:
    34
    Likes Received:
    1
    Occupation:
    SAN/TSM Administrator
    Location:
    TSM World
    Hi UnixandTSM ...

    Sounds like a scheduled task is resetting it for you.. It could either be an external task or a administrative one.

    try q event * t=a or

    q event * t=a begind=-1

    You could also dig through the logs and see what changing those thresholds for you

    q act begint=-24 search=HIghmig


    If youh have tons of scripts, you can output everything into a file and search that file.

    q script * > c:\scripts.txt
     
  14. unixandtsm

    unixandtsm New Member

    Joined:
    Feb 9, 2011
    Messages:
    307
    Likes Received:
    0
    Hi MrOdysseus,

    Just an hour ago i updated the stgpool to values HI=90 and LO=10 but then again it reset.
    below is the actlog query.
    Code:
    tsm>q actlog begint=-1 search=HIghmig
    
    Date/Time                Message
    --------------------     ----------------------------------------------------------
    11/12/12   10:19:51      ANR1000I Migration process 31403 started for storage pool
                              BACKUPPOOL automatically, highMig=0, lowMig=0,
                              duration=No. (PROCESS: 31403)
    
    
    Not sure what is making the migration to run automatically?
     
  15. smajl

    smajl Senior Member

    Joined:
    Dec 10, 2009
    Messages:
    148
    Likes Received:
    19
    Location:
    Brno
    q act s=BACKUPPOOL
    q act s="update stg"
    q act s="PROCESS: 31403"
    and look what/who updated the stg pool
     

Share This Page