1. Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING) Click the link to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This message will disappear after you have made at least 12 posts. Thank you for your cooperation.

GPFS mmbackup

Discussion in 'Backup / Archive Discussion' started by bhaven41, Jun 29, 2009.

  1. bhaven41

    bhaven41 New Member

    Joined:
    Feb 27, 2007
    Messages:
    70
    Likes Received:
    1
    Occupation:
    Storage/Backup Admin
    Location:
    Singapore
    Hi,

    We have production GPFS Linux systems (22 in cluster).
    We take falshcopy for GPFS filesystems (DS8K) and mount it to a dedicated Linux mounting server(1 node GPFS cluster) for TSM to backup.

    This is the FS mounted, 1.9TB with around 10 millions small files. (and growing)
    /dev/rsidev 4.3T 1.9T 2.5T 44% /opt/rsi

    This is not production server itself however I can not use ba client (only 1 filesystem - no multisession) and I can not use Image backup (not supported).

    So using following mmbackup command to back it up.

    /usr/lpp/mmfs/bin/mmbackup /dev/rsidev -n /opt/tivoli/tsm/client/ba/bin/<project_name>/mmbackup.ctrl -t full -s /var/tmp


    # more /opt/tivoli/tsm/client/ba/bin/<project_name>/mmbackup.ctrl
    # backup server
    serverName=<<servername in dsm.sys)
    # backup clients
    clientName=<<gpfs client name>
    # number of processes per client
    numberOfProcessesPerClient=6


    This backup is running for 40+ hours using Lan-free and 6 sessions. (PROBLEM #1)

    At last it fails with following error. (PROBLEM #2)

    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.

    <<client name>>: TSM dsmc selective command failed to run.

    Process 2 on client <<client name>> failed in processing its list of files.

    <<client name>>: TSM dsmc selective command failed to run.

    Process 3 on client <<client name>> failed in processing its list of files.

    <<client name>>: TSM dsmc selective command failed to run (see file /var/mmfs/mmbackup/opt/rsi_090624_05:54:28_4/dsmerror.log)
    .

    Process 4 on client <<client name>> failed in processing its list of files.
    *** glibc detected *** /usr/lpp/mmfs/bin/tsbackup: realloc(): invalid next size: 0x0000000055a9b000 ***
    ======= Backtrace: =========
    /lib64/libc.so.6[0x3be0a73adb]
    /lib64/libc.so.6(realloc+0x1d0)[0x3be0a75c30]
    /lib64/libc.so.6(realloc+0x3e)[0x3be0a75a9e]
    /usr/lpp/mmfs/bin/tsbackup[0x40009644]
    ======= Memory map: ========
    40000000-4000e000 r-xp 00000000 fd:00 2492080 /usr/lpp/mmfs/bin/tsbackup
    40110000-40112000 rwxp 00010000 fd:00 2492080 /usr/lpp/mmfs/bin/tsbackup
    40112000-4021f000 rwxp 40112000 00:00 0
    .......................
    ...................
    2480336 /usr/lib64/libstdc++.so.5.0.7
    2b03235e2000-2b03235eb000 rwxp 000c1000 fd:00 2480336 /usr/lib64/libstdc++.so.5.0.7
    2b03235eb000-2b03235fc000 rwxp 2b03235eb000 00:00 0
    2b03235fc000-2b0323603000 r-xp 00000000 fd:00 2492119 /usr/lpp/mmfs/lib/libgpfs.so
    2b0323603000-2b0323702000 ---p 00007000 fd:00 2492119 /usr/lpp/mmfs/lib/libgpfs.so
    2b0323702000-2b0323704000 rwxp 00006000 fd:00 2492119 /usr/lpp/mmfs/lib/libgpfs.so
    2b0323704000-2b0323707000 rwxp 2b0323704000 00:00 0
    7fff8761c000-7fff877a5000 rwxp 7fffffe76000 00:00 0 [stack]
    ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0 [vdso]
    /usr/lpp/mmfs/bin/mmbackup: line 379: 22148: Abort
    mmbackup: Command failed. Examine previous error messages to determine cause.


    In errorlog file,
    # more /var/mmfs/mmbackup/opt/rsi_090624_05:54:28_4/dsmerror.log
    06/25/2009 20:12:06 ANS1228E Sending of object '/opt/tivoli/tsm/client/ba/bin/<<project_name>>/*' failed
    06/25/2009 20:12:06 ANS4005E Error processing '/opt/tivoli/tsm/client/ba/bin/<<project_name>>/*': file not found
    06/25/2009 22:11:48 ANS1804E Selective Backup processing of '/opt/rsi/.mmbuTrans4' finished with failures

    I do not understand why mmbackup is trying to backup '/opt/tivoli/tsm/client/ba/bin/<<project_name>>/*'
    (PROBLEM #3)


    Have you seen these kind of problems earlier?

    Is the memory dump because of /opt/tivoli/tsm/client/ba/bin/<<project_name>>/*'?


    In TSM Server's actlog, I can see 6 sessions completed with messages

    06/16/2009 22:48:59 ANE4971I (Session: 1777, Node: <<node_name>.) LanFree data
    bytes: 225.72 GB (SESSION: 1777)

    All the sessions are getting completed with near to 225GB.
    Which means (225*6= 1.35TB) however my filesystems is of 1.9TB
    ((PROBLEM #4))

    As per my understanding, when starting 6 sessions with mmbackup, it equally devides data into 6 filelists.



    Please help me understand these.

    Thank you in advance.
     
  2.  
  3. tivolimistery

    tivolimistery New Member

    Joined:
    Oct 10, 2009
    Messages:
    22
    Likes Received:
    0
    Permission Denied - mmexectsmcmd

    Hi,

    I have a problem that seems to be similar to your problem.
    Do you have solved it? Can you help me?

    Thanks.
    Mauro Tridici

    ************************************************
    Permission denied.
    Permission denied.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.
    Permission denied.
    mmexectsmcmd: Command failed. Examine previous error messages to determine cause.

    n11-ml0:

    Process 0 on client n11-ml0 failed in processing its list of files.

    n21-ml0:

    Process 1 on client n21-ml0 failed in processing its list of files.

    n22-ml0:

    Process 2 on client n22-ml0 failed in processing its list of files.

    n11-ml0:

    Process 3 on client n11-ml0 failed in processing its list of files.

    n21-ml0:

    Process 4 on client n21-ml0 failed in processing its list of files.

    n22-ml0:

    Process 5 on client n22-ml0 failed in processing its list of files.

    n11-ml0:

    Process 6 on client n11-ml0 failed in processing its list of files.

    n21-ml0:

    Process 7 on client n21-ml0 failed in processing its list of files.

    n22-ml0:

    Process 8 on client n22-ml0 failed in processing its list of files.
    doFSBackup() returned rc = 2
     
  4. sipocz

    sipocz New Member

    Joined:
    Mar 23, 2006
    Messages:
    3
    Likes Received:
    0
    I would simplify things to try to resolve the issue. Try only one process for the server. Since the files are small, I would not use LAN free.
     
  5. bhaven41

    bhaven41 New Member

    Joined:
    Feb 27, 2007
    Messages:
    70
    Likes Received:
    1
    Occupation:
    Storage/Backup Admin
    Location:
    Singapore
    Sorry I missed the previous one............

    I tried everything and finally gave up ... :( :( :down:

    Not using TSM for GPFS now.
     
: gpfs

Share This Page