Migration "Hangs"

[Wfitzgerald]

I have recently had a number of occurrences of TSM migration of data from disk pool to tape pool seem to Freeze. I say freeze because there is no indication in the logs, in the errrpt, the tape library or the tape drive logs that show any reason for the situation.
The problem is intermittent. I has been going on for a month with no real pattern.

TSM server

AIX 5.3 ML9
Tape Library 3584
Tape Drives 3588 an combination of LTO3 and LTO4
Disk space DS4300

TSM 5.4.6 in the process of migrating all our clients to a level that will handle 6.1. Will then Migrate TSM to 6.1.3

Symptoms:
Data is being migrated from the disk pool to the primary tape pool. Migration Process starts, but then does not complete. Example, 1.5 gig files should take a few minutes to move, Job sat for 65 minutes before I had to bounce TSM to get it to release. Cancel does not work in this situation.

Fact: direct backup to tape works OK.
Copy from Tape pool to tape pool works OK.

I updated the TSM server to 5.4.6 to cure this issue. it has not helped.

Anyone here have this kind of situation? Any answers? Suggestions?

Thanks

Bill

 

[Kelvin]

Actlog is your friend

 

[Wfitzgerald]

That is part of the problem, there is nothing in the actlog or any other log that hints as to what is happening.

 

[cheffern]

are you running caching on the diskpool?

 

[Wfitzgerald]

no, no caching.

 

[cheffern]

output of 'q db f=d' ?

 

[Wfitzgerald]

Available Space (MB): 253,144
Assigned Capacity (MB): 253,144
Maximum Extension (MB): 0
Maximum Reduction (MB): 56,524
Page Size (bytes): 4,096
Total Usable Pages: 64,804,864
Used Pages: 49,609,723
Pct Util: 76.6
Max. Pct Util: 76.7
Physical Volumes: 32
Buffer Pool Pages: 131,072
Total Buffer Requests: 640,422,431
Cache Hit Pct.: 99.10
Cache Wait Pct.: 0.00
Backup in Progress?: Yes
Type of Backup In Progress: Full
Incrementals Since Last Full: 0
Changed Since Last Backup (MB): 379.43
Percentage Changed: 0.20
Last Complete Backup Date/Time: 06/02/10 15:08:14
Estimate of Recoverable Space (MB): 16,879
Last Estimate of Recoverable Space (MB): 02/02/07 06:46:40

 

[Wfitzgerald]

all back end disk space is defined in 8 gig volumes. we are using DS4300 disks

 

[TonyB]

Sounds like a locking issue...try running a "show lock" before recycling the instance, might tell you something useful.

 

[Wfitzgerald]

I will give that a try next time

 

[Wfitzgerald]

This is what I have from the show lock

Lock hash table contents (slots=510):
Note: Enabling trace class TMTIMER will provide additional timing info on the following locks
slot -> 109:
LockDesc: Type=34040(ss segment), NameSpace=38869, SummMode=xLock, Key='287.0'
Holder: (ssalloc.c:1684 Thread -1) Tsn=0:4262539828, Mode=xLock
slot -> 229:
LockDesc: Type=34040(ss segment), NameSpace=38871, SummMode=xLock, Key='287.0'
Holder: (ssalloc.c:1684 Thread -1) Tsn=0:4262539828, Mode=xLock
slot -> 403:
LockDesc: Type=34040(ss segment), NameSpace=38870, SummMode=xLock, Key='95.0'
Holder: (ssalloc.c:1684 Thread -1) Tsn=0:4262539828, Mode=xLock

 

[Wfitzgerald]

at this time it is not completely froze, but it is moving the data very slowly.
Any Ideas?

 

[GregE]

Are the disks used by your TSM disk pools shared with any other applications? Can you or your SAN storage admins look at disk activity?

 

[Wfitzgerald]

the disk is only used by the this instance of TSM. The space is used by all clients not backing up directly to tape.

 

[Wfitzgerald]

when TSM is inactive there is no activity on these disks, The thing is, I have been using these disks for the TSM disk pool for over 3 years with out issue. The zoning is set so only TSM can see it. I have 2 TSM environments, but they use separate disk space completely.

 

[Wfitzgerald]

today I got this, again it is moving, but very slowly
Lock hash table contents (slots=510):
Note: Enabling trace class TMTIMER will provide additional timing info on the following locks
slot -> 13:
LockDesc: Type=34040(ss segment), NameSpace=38871, SummMode=xLock, Key='31.0'
Holder: (ssalloc.c:1684 Thread -1) Tsn=0:4267507210, Mode=xLock
slot -> 133:
LockDesc: Type=34040(ss segment), NameSpace=38873, SummMode=xLock, Key='31.0'
Holder: (ssalloc.c:1684 Thread -1) Tsn=0:4267507210, Mode=xLock

 

[Wfitzgerald]

I have noticed that as the number of files associated with the dataspace are backed up, each time a file completes, the number of locks increase. 2 files 2 locks, 3 files 3 locks, 4 files 4 locks. I was thinking that the number of locks might increase the processing time. so as each lock is added the processing slows. I have a call to IBM in ans will let you know what I find