tsm crash - TS3200

[jeszi]

Hello,

Please help me.

server: win2003 32 bit sp1
tsm server: Version 5, Release 5, Level 4.3

I must configure new ts3200 library with 4 drive, but tsm crashing while touching the new library.
I set up the new stg as the nextstg for the diks pool and then I start the "migrate stg DISKPOOL lo=0 w=y"

TSM server log:

ANR2017I Administrator ADMIN issued command: MIGRATE STGPOOL DISKPOOL lo=0
w=y
ANR0984I Process 1 for MIGRATION started in the FOREGROUND at 15:18:08.
ANR2110I MIGRATE STGPOOL started as process 1.
ANR1000I Migration process 1 started for storage pool DISKPOOL manually,
highMig=80, lowMig=0, duration=No.
Entering exception handler.
Leaving exception handler.
C:\Program Files\Tivoli\tsm\server>

dsmserv.err:
------------
====================================================================

Crash dump file:

C:\Program Files\Tivoli\tsm\server\dsmserv.dmp

Exception code:

C0000005 (ACCESS_VIOLATION) at 001B:1047AC43

Registers:

EAX=00000000 EBX=00000000 ECX=0CD94E38 EDX=0CD71E30 ESI=0CD68160
EDI=0CD8F250 EBP=00000018 ESP=1B3BDEDC EIP=1047AC43 FLG=00010246
CS =0000001B DS =00000023 SS =00000023 ES =00000023 FS =0000003B
GS =00000000

Call stack:

Address Parameters F()+Offset, File(Line#)
------------------------------------------------------------------------------
1047AC43 (0x0CD78FC0 0x00000000 0x1B3BE068 0x00000000) ScsiCheckAutoID()+783, Unavailable
1047B167 (0x00000000 0x00000000 0x00000000 0x00000000) ScsiDeleteVolume()+347, Unavailable
0CD78FC0 (0x00000000 0x00000000 0x0CD78FD0 0x00000000) Unavailable, Unavailable

Exception handler complete. RC: 126




library:
--------
q library 3573L5 f=d

Library Name: 3573L5
Library Type: SCSI
ACS Id:
Private Category:
Scratch Category:
WORM Scratch Category:
External Manager:
RSM Media Type:
Shared: No
LanFree:
ObeyMountRetention:
Primary Library Manager:
WWN:
Serial Number: 00L4U78N0222_LL0
AutoLabel: Yes
Reset Drives: No
Relabel Scratch:

library path:
-------------
q path IGA_TSM 3573L5 srctype=SERVER desttype=LIBRARY f=d

Source Name: IGA_TSM
Source Type: SERVER
Destination Name: 3573L5
Destination Type: LIBRARY
Library:
Node Name:
Device: \\.\Changer0
External Manager:
LUN:
Initiator: 0
Directory:
On-Line: Yes


1 of the 4 drive:
-----------------
tsm: IGA_TSM>q drive 3573L5 DRIVE1 f=d

Library Name: 3573L5
Drive Name: DRIVE1
Device Type: LTO
On-Line: Yes
Read Formats: ULTRIUM5C,ULTRIUM5,ULTRIUM4C,ULTRIUM4,ULTRIUM3C,ULTRIUM3
Write Formats: ULTRIUM5C,ULTRIUM5,ULTRIUM4C,ULTRIUM4
Element: 256
Drive State: EMPTY
Volume Name:
Allocated to:
WWN:
Serial Number: 1068008035
Cleaning Frequency (Gigabytes/ASNEEDED/NONE): NONE


1 of the 4 drive path:
----------------------
q path IGA_TSM DRIVE1 srctype=SERVER desttype=DRIVE library=3573L5 f=d

Source Name: IGA_TSM
Source Type: SERVER
Destination Name: DRIVE1
Destination Type: DRIVE
Library: 3573L5
Node Name:
Device: \\.\Tape4801101
External Manager:
LUN:
Initiator: 0
Directory:
On-Line: Yes


device class:
-------------
q devclas LTOCLASS5 f=d

Device Class Name: LTOCLASS5
Device Access Strategy: Sequential
Storage Pool Count: 1
Device Type: LTO
Format: DRIVE
Est/Max Capacity (MB):
Mount Limit: DRIVES
Mount Wait (min): 40
Mount Retention (min): 10
Label Prefix: ADSM
Drive Letter:
Library: 3573L5
Directory:
Server Name:
Retry Period:
Retry Interval:
Twosided:
Shared:
High-level Address:
Minimum Capacity:
WORM: No
Drive Encryption: Allow
Scaled Capacity:


stg:
----
q stg STGNAME f=d

Storage Pool Name: STGNAME
Storage Pool Type: Primary
Device Class Name: LTOCLASS5
Estimated Capacity: 0.0 M
Space Trigger Util:
Pct Util: 0.0
Pct Migr: 0.0
Pct Logical: 0.0
High Mig Pct: 90
Low Mig Pct: 70
Migration Delay: 0
Migration Continue: Yes
Migration Processes: 1
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold: No Limit
Access: Read/Write
Description: 3573 LTO5 Library
Overflow Location:
Cache Migrated Files?:
Collocate?: No
Reclamation Threshold: 60
Offsite Reclamation Limit:
Maximum Scratch Volumes Allowed: 100
Number of Scratch Volumes Used: 0
Delay Period for Volume Reuse: 0 Day(s)
Migration in Progress?: No
Amount Migrated (MB): 0.00
Elapsed Migration Time (seconds): 0
Reclamation in Progress?: No
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?: Yes
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:

 

[Harry_Redl]

Hello Jeszi,

first thing - do not hijack other threads - start new ones. I moved your post to new thread.

Now to your problem:
Are you sure this is related to your library? You say that it happen when TSM "touches" the library. All I can see now it crashes when you run migration (to that library).
Please test these:
a) run audit your library (so we see if it crashes when contacting it)
b) try to delete the library (paths, drives, library) and re-create it again (use TSM names - like lb1.0.0.2, not \\.\Changer0)
(are you using IBM device driver? not TSM driver .... that is wrong for IBM devices)
c) try do define new primary stgpool and try to migrate some data from DISKPOOL to it - so we see if the migration works (and get TS3200 out of the picture) - I recommend using DISK based stgpool first (so you can easily migrate the data back) - then you can try FILE

When did it start? You are at 5.5.4.3 - it is the latest in the 5.5 line at the moment - did it start after upgrade? Was there anything changed? Is Windows Removable Storage service disabled?

Hope it helps

Harry

 

[jeszi]

audit library

Hello Harry,

I'm beginer in TSM so it is hard to resolve this issu. Thank you for your help.

I updated the library path to use TSM names - like lb1.0.0.2. (crash is still present when start migrate process)
Yes, I'm using IBM device driver.
Yes, the Windows Removable Storage service is disabled.

Audit library completed without crash but the 25 tape config definitly wrong. Could you help me how to add these tapes to TSM's inventory database?

Audit library:
--------------
ANR2017I Administrator ADMIN issued command: AUDIT LIBRARY 3573L5
ANR0984I Process 1 for AUDIT LIBRARY started in the BACKGROUND at 22:17:23.
ANR8457I AUDIT LIBRARY: Operation for library 3573L5 started as process 1.
ANR2017I Administrator ADMIN issued command: QUERY PROCESS
ANR8941W The volume from slot-element 4096 in drive DRIVE1 (\\.\Tape4801101) in library 3573L5 is blank.
ANR8941W The volume from slot-element 4097 in drive DRIVE2 (\\.\Tape4801102) in library 3573L5 is blank.
ANR8941W The volume from slot-element 4098 in drive DRIVE3 (\\.\Tape4801103) in library 3573L5 is blank.
.
.etc
.
ANR8941W The volume from slot-element 4138 in drive DRIVE3 (\\.\Tape4801103) in library 3573L5 is blank.
ANR8941W The volume from slot-element 4139 in drive DRIVE1 (\\.\Tape4801101) in library 3573L5 is blank.
ANR8941W The volume from slot-element 4140 in drive DRIVE4 (\\.\Tape4801104) in library 3573L5 is blank.
ANR8455E Volume CG0001L5 could not be located during audit of library 3573L5. Volume has been removed from the library inventory.
ANR8455E Volume CG0002L5 could not be located during audit of library 3573L5. Volume has been removed from the library inventory.
ANR8455E Volume CG0003L5 could not be located during audit of library 3573L5. Volume has been removed from the library inventory.
.
.etc
.
ANR8455E Volume CG0023L5 could not be located during audit of library 3573L5. Volume has been removed from the library inventory.
ANR8455E Volume CG0024L5 could not be located during audit of library 3573L5. Volume has been removed from the library inventory.
ANR8455E Volume CG0025L5 could not be located during audit of library 3573L5. Volume has been removed from the library inventory.
ANR8883W The volume in slot 4096 is in the library 3573L5 but not in the TSM's inventory database. Please either check in this volume or remove it from the library;
ANR8883W The volume in slot 4097 is in the library 3573L5 but not in the TSM's inventory database. Please either check in this volume or remove it from the library;
ANR8883W The volume in slot 4098 is in the library 3573L5 but not in the TSM's inventory database. Please either check in this volume or remove it from the library;
.
.etc
.
ANR8883W The volume in slot 4138 is in the library 3573L5 but not in the TSM's inventory database. Please either check in this volume or remove it from the library;
ANR8883W The volume in slot 4139 is in the library 3573L5 but not in the TSM's inventory database. Please either check in this volume or remove it from the library;
ANR8883W The volume in slot 4140 is in the library 3573L5 but not in the TSM's inventory database. Please either check in this volume or remove it from the library;
ANR8461I AUDIT LIBRARY process for library 3573L5 completed successfully.
ANR0985I Process 1 for AUDIT LIBRARY running in the BACKGROUND completed with completion state SUCCESS at 22:46:45.

 

[jeszi]

Is it ok for preparing tapes for the library? :

ANR2017I Administrator ADMIN issued command: AUDIT LIBRARY 3573L5 checkl=barc
ANR0984I Process 1 for AUDIT LIBRARY started in the BACKGROUND at 00:10:31.
ANR8457I AUDIT LIBRARY: Operation for library 3573L5 started as process 1.
ANR8461I AUDIT LIBRARY process for library 3573L5 completed successfully.
ANR0985I Process 1 for AUDIT LIBRARY running in the BACKGROUND completed with completion state SUCCESS at 00:10:32.
ANR2017I Administrator ADMIN issued command: CHECKIN LIBVOLUME 3573L5 status=private search=yes checkl=barc
ANR0984I Process 2 for CHECKIN LIBVOLUME started in the BACKGROUND at 00:16:09.ANR8422I CHECKIN LIBVOLUME: Operation for library 3573L5 started as process 2.
ANR8430I Volume CG0023L5 has been checked into library 3573L5.
ANR8430I Volume CG0022L5 has been checked into library 3573L5.
.
.
.
ANR8430I Volume CG0010L5 has been checked into library 3573L5.
ANR8430I Volume CG0020L5 has been checked into library 3573L5.
ANR8431I CHECKIN LIBVOLUME process completed for library 3573L5; 25 volume(s) found.
ANR0985I Process 2 for CHECKIN LIBVOLUME running in the BACKGROUND completed with completion state SUCCESS at 00:16:10.


q libvol

Library Name Volume Name Status Owner Last Use Home Device
Element Type
------------ ----------- ---------------- ---------- --------- ------- ------
3573L5 CG0001L5 Private 4,132
3573L5 CG0002L5 Private 4,128
.
.

 

[Harry_Redl]

Hi,

it is not correct - you have to label the tapes before checking them in - until they are labeled they cannot be used.
First - check all these volumes out of the library (checkout libvol 3573L5 <XYZ> checklabel=no remove=no) and use "label libvolume 3573l5 search=yes labelsource=barcode checkin=scratch"

Update the tape paths to "IBM naming" - for example: mt0.0.0.2 as well.
What about migrating to another (not Tape) stgpool?

Hope it helps

Harry

 

[jeszi]

First I run:
checkout libvol 3573L5 CG0023L5 checklabel=no remove=no
..
ANR8434I CHECKOUT LIBVOLUME: Operation for volume CG0020L5 in library 3573L5
started as process 31.
ANR8829I Remove volume CG0020L5 from slot 4140 of library 3573L5 at your convenience.
ANR8438I CHECKOUT LIBVOLUME for volume CG0020L5 in library 3573L5 completed
successfully.
ANR0985I Process 31 for CHECKOUT LIBVOLUME running in the BACKGROUND completed
with completion state SUCCESS at 11:31:11.

Then:
ANR2017I Administrator ADMIN issued command: LABEL LIBVOLUME 3573l5
search=yes labelsource=barcode checkin=scratch
ANR0984I Process 32 for LABEL LIBVOLUME started in the BACKGROUND at 11:32:51.
ANR8799I LABEL LIBVOLUME: Operation for library 3573L5 started as process 32.
ANR8816E LABEL LIBVOLUME: Volume CG0023L5 in library 3573L5 cannot be labeled
because it is currently defined in a storage pool or in the volume history
file.


I tryed:
del volhist todate=today type=remote volume=CG0023L5 force=yes
Do you wish to proceed? (Yes (Y)/No (N)) y
ANR2404E DELETE VOLHISTORY: Volume CG0023L5 is not available.
ANS8001I Return code 3.

q vol stg=NEWSTG
Volume Name Storage Device Estimated Pct Volume
Pool Name Class Name Capacity Util Status
------------------------ ----------- ---------- --------- ----- --------
CG0001L5 NEWSTG LTOCLASS5 0.0 M 0.0 Empty
CG0002L5 NEWSTG LTOCLASS5 0.0 M 0.0 Empty
.


There is a first backup to a DIKS type stg. The DISK type stg has a tape type nextstg which is an old library (4560). The migrate process is successful to the 4560 library.

I modified the nextstg to the new library (3573L5) and the run the migrate process that make a tsm crash.

 

[jeszi]

I could run "delete volume volumename" to all tape and now the label libvolume process is in progress...

For now label libvolume process is completed. I guess I shoult run "def volume NEWSTG volumename".

 

[Harry_Redl]

Hi,

so the label libvolume command finished (and labeled the tapes) correctly? (can you attach the activity log part?)
You do not need to define volumes to that stgpool - you can use scratch tapes ...

Can you migrate from stgpool based on 4650 library to TS3200?
Can you backup your DB to LTOCLASS5 devclass? Can you put any data on LTO5 tapes? (try direct backup to that stgpool - no migration)

Are device drivers in the Windows Device manager set correctly?

Can be multiple things - TSM and IBM device driver coexistence problem, wrong driver assignment, program error ....

Harry

 

[jeszi]

Hi,

so the label libvolume command finished (and labeled the tapes) correctly? (can you attach the activity log part?)
You do not need to define volumes to that stgpool - you can use scratch tapes ...

Can you migrate from stgpool based on 4650 library to TS3200?
Can you backup your DB to LTOCLASS5 devclass? Can you put any data on LTO5 tapes? (try direct backup to that stgpool - no migration)

Are device drivers in the Windows Device manager set correctly?

Can be multiple things - TSM and IBM device driver coexistence problem, wrong driver assignment, program error ....

Harry

label libvolume:
---------------
2010.10.24 12:44:46 ANR0984I Process 36 for LABEL LIBVOLUME started in the
BACKGROUND at 12:44:46. (SESSION: 650, PROCESS: 36)
2010.10.24 12:44:46 ANR8799I LABEL LIBVOLUME: Operation for library 3573L5
started as process 36. (SESSION: 650, PROCESS: 36)
2010.10.24 13:19:35 ANR8801I LABEL LIBVOLUME process 36 for library 3573L5
completed; 25 volume(s) labeled, 25 volume(s) checked-in.
(SESSION: 650, PROCESS: 36)
2010.10.24 13:19:35 ANR0985I Process 36 for LABEL LIBVOLUME running in the
BACKGROUND completed with completion state SUCCESS at
13:19:35. (SESSION: 650, PROCESS: 36)

I have already defined volumes to that stgpool. I hope it isn't bad.
In Windows Device manager I use TSM driver for old drives (4560 library) and I use ibm device driver for new drives (3573 library).

You wrote:
"Can you migrate from stgpool based on 4650 library to TS3200?
Can you backup your DB to LTOCLASS5 devclass? Can you put any data on LTO5 tapes? (try direct backup to that stgpool - no migration)"
Could you give me an example, please? (For example if I want to backup the DB to LTOCLASS5.)
I gues: backup db t=f devc=LTOCLASS5 , but I'm not sure.

 

[Harry_Redl]

Hi,

I just prefer DBSnapshot for testing - but except that you are correct
backup db type=dbs devc=ltoclass5

migration from 4650 to ts3200:
upd stg <4650POOL_NAME> next=stgname
migr stgpool <4650POOL_NAME> lowmig=XYZ

backing up directly to that pool:
You may define new policy domain (and policy set, mgmtclass, copygroup ....) and define new test node to it or you can vary all DISKPOOL volumes to offline and start backup (nextstgpool of diskpool must be TS3200 stgpool)

Hope it helps

Harry

 

[jeszi]

I run "backup db type=dbs devc=ltoclass5" but there was no scratch volume. Oh
I deleted volume from stg again.
I run label libvolume again - log told me tapes already labeled

I decided to run:
checkin libvol 3573L5 CG0001L5 checklabel=barcode status=scratch
But it is not working. Log:
ANR8422I CHECKIN LIBVOLUME: Operation for library 3573L5 started as process 92.A
NR8323I 006: Insert LTO volume CG0001L5 R/W into entry/exit port of library
3573L5 within 60 minute(s); issue 'REPLY' along with the request ID when ready.A
NR8323I 006: Insert LTO volume CG0001L5 R/W into entry/exit port of library
3573L5 within 59 minute(s); issue 'REPLY' along with the request ID when ready.A
NR8323I 006: Insert LTO volume CG0001L5 R/W into entry/exit port of library
3573L5 within 58 minute(s); issue 'REPLY' along with the request ID when ready.A

I really don't understand what is happening. I can't see the new tapes: q vol

 

[hogmaster]

Nearly right . Just add searh=yes (see below) , other wise the commadn defaults to the entry/exit port in the library.
checkin libvol 3573L5 CG0001L5 checklabel=barcode status=scratch search=yes

 

[Harry_Redl]

Hi,

do not mix two terms -
"volumes" - members of stgpool
"libvolumes" - inventory of the library

You already checked the tapes in (during label libvolume process)

2010.10.24 13:19:35 ANR8801I LABEL LIBVOLUME process 36 for library 3573L5
completed; 25 volume(s) labeled, 25 volume(s) checked-in.

issue "q libvol 3573L5" to see them (do you see them?)
What error you got when running the DBS backup? Actlog part please ...

Hope it helps

Harry

 

[jeszi]

Thank you!

It was the clue
checkin libvol 3573L5 checklabel=barcode status=scratch search=yes

 

[jeszi]

That's it!
The first step I can use the new device:
ANR2017I Administrator ADMIN issued command: BACKUP DB type=dbs
devc=ltoclass5
ANR0984I Process 95 for DATABASE BACKUP started in the BACKGROUND at 20:44:44.
ANR8337I LTO volume CG0001L5 mounted in drive DRIVE1 (mt0.0.0.3).
ANR0513I Process 95 opened output volume CG0001L5.
ANR1360I Output volume CG0001L5 opened (sequence number 1).
ANR4554I Backed up 344000 of 776184 database pages.
ANR4554I Backed up 710080 of 776184 database pages.
ANR1361I Output volume CG0001L5 closed.
ANR0515I Process 95 closed volume CG0001L5.
ANR4550I Full database backup (process 95) complete, 776184 pages copied.
ANR0985I Process 95 for DATABASE BACKUP running in the BACKGROUND completed
with completion state SUCCESS at 20:46:43.
ANR8336I Verifying label of LTO volume CG0001L5 in drive DRIVE1 (mt0.0.0.3).
ANR8468I LTO volume CG0001L5 dismounted from drive DRIVE1 (mt0.0.0.3) in
library 3573L5.

You are great guys!

Could you help me to migrate everything from old library to new? I must remove the old library from the system.

tsm: IGA_TSM>q stg

Storage Device Estimated Pct Pct High Low Next Stora-
Pool Name Class Name Capacity Util Migr Mig Mig ge Pool
Pct Pct
----------- ---------- ---------- ----- ----- ---- --- -----------
ALL_B_T LTOCLASS3 33,369 G 22.8 33.6 100 0
IGA_B_T LTOCLASS5 0.0 M 0.0 0.0 90 70
LOTUS_A_T LTO_02 763 G 30.4 50.0 90 70
WIN_A_T LTO_01 0.0 M 0.0 0.0 90 70
WIN_B_D DISK 70 G 100.0 100.0 80 10 ALL_B_T


IGA_B_T= new library

1. I gues first step is "disable sessions".
2. And then setup IGA_B_T as nextstg for ALL_B_T, and then "migrate stg ALL_B_T lo=0".
3. "upd stg WIN_B_D nextstg IGA_B_T"
4. migrate stg WIN_B_T lo=0

But I would be very glad if you could write to me the correct steps.

 

[jeszi]

Harry, the migration process is in progress without crash! Thank you for your help. It seems the missing label libvolume command was the crash problem. (or the drive and library device name/tsm name)

tsm: IGA_TSM>q proc

Process Process Description Status
Number
-------- -------------------- -------------------------------------------------
97 Migration Disk Storage Pool WIN_B_D, Moved Files: 12, Moved
Bytes: 32,737,120,256, Unreadable Files: 0,
Unreadable Bytes: 0. Current Physical File
(bytes): 20,451,377,152 Current output volume:
CG0002L5.

I'm really glad. Thanks again.