TSM 6.3, IBM TS3100, scratch tapes arent automatically assigned to pools as needed

[bluesman]

Hi,

I've been trying to get this library to work for the past week but it's not really working yet the way I want it to. Not to mention that if I enable I/O Station i get I/O errors for just about any action. But that's not really important right now.

The problem i'm trying to solve right now is that it won't use scratch tapes when I try to migrate data from my primary stgpool to next tape pool.

One thing that I did notice is that, the tapes are really LTO3 but the barcodes are apparently LTO4. Could the problem be Media Type LTO-4? If I manually assign libvol to backuptape pool, it will migrate (eg: define vol backuptape BIC001L4).

Any ideas?

If more info is need, just let me know. Thanks for the help.

tsm> migrate stg backuppool lo=0

Act log says:
02/15/2012 11:08:30 ANR1100I Migration started for volume
/opt/tsm/pools/backup/0000018F.BFS, storage pool
BACKUPPOOL (process number 3). (SESSION: 1, PROCESS: 3)
02/15/2012 11:08:30 ANR1102I Removable volume /opt/tsm/pools/backup/0000018F.-
BFS is required for migration. (SESSION: 1, PROCESS: 3)
02/15/2012 11:08:30 ANR1102I Removable volume /opt/tsm/pools/backup/00000190.-
BFS is required for migration. (SESSION: 1, PROCESS: 3)
02/15/2012 11:08:30 ANR1405W Scratch volume mount request denied - no scratch
volume available. (SESSION: 1, PROCESS: 3)
02/15/2012 11:08:30 ANR1126E Migration is ended for volume
/opt/tsm/pools/backup/0000018F.BFS. There is insufficient
space in subordinate storage pools. (SESSION: 1, PROCESS:
3)

tsm>q drive f=d

Library Name: TS3100
Drive Name: DRIVE1
Device Type: LTO
On-Line: Yes
Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTRIUM2,ULTRIUMC,ULTRIUM
Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTRIUM2
Element: 256
Drive State: EMPTY
Volume Name:
Allocated to:
WWN: 5000E11140891001
Serial Number: 1H10058862
Last Update by (administrator): TSMADMIN
Last Update Date/Time: 02/14/2012 19:17:49
Cleaning Frequency (Gigabytes/ASNEEDED/NONE): NONE


tsm:>q libvol f=d

Library Name Volume Name Status Owner Last Use Home Device Cleanings Left Media Type
Element Type
------------ ----------- ---------------- ---------- --------- ------- ------ -------------- ----------------
TS3100 BIC001L4 Scratch 4,097 LTO LTO-4
TS3100 BIC002L4 Scratch 4,098 LTO LTO-4
TS3100 BIC003L4 Scratch 4,099 LTO LTO-4
TS3100 BIC004L4 Scratch 4,100 LTO LTO-4
TS3100 BIC005L4 Scratch 4,101 LTO LTO-4
TS3100 BIC006L4 Scratch 4,102 LTO LTO-4
TS3100 BIC007L4 Scratch 4,103 LTO LTO-4
TS3100 BIC008L4 Scratch 4,104 LTO LTO-4
TS3100 BIC009L4 Scratch 4,105 LTO LTO-4
TS3100 BIC010L4 Scratch 4,106 LTO LTO-4
TS3100 BIC011L4 Scratch 4,110 LTO LTO-4
TS3100 BIC012L4 Scratch 4,109 LTO LTO-4
TS3100 BIC013L4 Scratch 4,108 LTO LTO-4
TS3100 BIC014L4 Scratch 4,107 LTO LTO-4
TS3100 BIC015L4 Scratch 4,114 LTO LTO-4
TS3100 BIC016L4 Scratch 4,113 LTO LTO-4
TS3100 BIC017L4 Scratch 4,112 LTO LTO-4
TS3100 BIC018L4 Scratch 4,111 LTO LTO-4
TS3100 BIC019L4 Scratch 4,118 LTO LTO-4
TS3100 BIC020L4 Scratch 4,117 LTO LTO-4
TS3100 BIC021L4 Scratch 4,116 LTO LTO-4
TS3100 BIC022L4 Scratch 4,115 LTO LTO-4

 

[smajl]

What is number of "Maximum Scratch Volumes Allowed" and "Number of Scratch Volumes Used" of target storage pool?
q stg <tapestg> f=d

 

[bluesman]

30 and 0.

tsm: SERVER1>q stg backuptape f=d

Storage Pool Name: BACKUPTAPE
Storage Pool Type: Primary
Device Class Name: LTO_CLASS_1
Estimated Capacity: 0.0 M
Space Trigger Util:
Pct Util: 0.0
Pct Migr: 0.0
Pct Logical: 0.0
High Mig Pct: 90
Low Mig Pct: 70
Migration Delay: 0
Migration Continue: Yes
Migration Processes: 1
Reclamation Processes: 1
Next Storage Pool:
Reclaim Storage Pool:
Maximum Size Threshold: No Limit
Access: Read/Write
Description:
Overflow Location:
Cache Migrated Files?:
Collocate?: No
Reclamation Threshold: 60
Offsite Reclamation Limit:
Maximum Scratch Volumes Allowed: 30
Number of Scratch Volumes Used: 0
Delay Period for Volume Reuse: 0 Day(s)
Migration in Progress?: No
Amount Migrated (MB): 0.00
Elapsed Migration Time (seconds): 0
Reclamation in Progress?: No
Last Update by (administrator): TSMADMIN
Last Update Date/Time: 02/14/2012 19:38:21
Storage Pool Data Format: Native
Copy Storage Pool(s):
Active Data Pool(s):
Continue Copy on Error?: Yes
CRC Data: No
Reclamation Type: Threshold
Overwrite Data when Deleted:
Deduplicate Data?: No
Processes For Identifying Duplicates:
Duplicate Data Not Stored:
Auto-copy Mode: Client
Contains Data Deduplicated by Client?: No

 

[smajl]

Can you physically verify labels on volumes in library?

 

[bluesman]

Yes, the barcodes are correctly read. I've checked it today, that's why I noticed the tapes are really LOT3. Now if the tapes are in proper slots that I can't be 100% sure, since I don't know how to map HomeElement to slot number. But I presume they are.

 

[smajl]

If I understand you correctly LTO-3 tapes you inserted have label ending "...L4" ?
TSM recognize volume types accorging to 2 last chars of barcode. xxxxxxL2, xxxxxxL3, xxxxxxL4.

 

[bluesman]

Yes, that's correct LTO3 tapes with L4 labels, just noticed that today. Is there a way to get around it? For example, disable barcode reader and manually labeling the tapes for example?

EDIT: Or maybe force Media Type to LTO-3? Cant find the command though.

 

[spidermila]

Have you labeled the tapes using the "label libvolume" command?

 

[bluesman]

Yes, though I had some problems with it. But eventually managed. I could try removing all volumes and history and start from scratch, log the commands and responses and post them. Would that be helpful?

 

[bluesman]

Redefined the whole thing and now i'm trying to label them and getting...
tsm> LABEL LIBVOLUME ts3100 search=bulk labelsource=barcode
ANR8826E No Entry/Exit found on library TS3100.

I'm lost...

 

[rore]

TSM recognize volume types accorging to 2 last chars of barcode. xxxxxxL2, xxxxxxL3, xxxxxxL4.

Can you provide documentation about? I am almost sure that is not true.
TSM can handle tapes with 6 or 8 character barcode (xxxxxxLx OR xxxxxx).

Redefined the whole thing and now i'm trying to label them and getting...
tsm> LABEL LIBVOLUME ts3100 search=bulk labelsource=barcode
ANR8826E No Entry/Exit found on library TS3100.

I'm lost...

Did that command work in the past?
Is the I/O station enabled?

If you enabled it after defining your library you have to audit it with the option REFRESHstate=yes OR recycle TSM.

Rudy

 

[bluesman]

Enabled the I/O station, audit the library and that fixed the No Entry/Exit problem. Now LABEL command asks:
ANR8373I 003: Fill the bulk entry/exit port of library TS3100 with all LTO volumes to be processed within 60 minute(s); issue 'REPLY' along with the request ID when ready.

So I do that and I get

ANR9756I No volumes were found.
ANR8801I LABEL LIBVOLUME process 6 for library TS3100 completed; 0 volume(s) labeled, 0 volume(s) checked-in.

tsm> show slots ts3100 -- strange, since this used to show all barcode values and matched the inventory status i see on the TS3100 WEB admin page

tsm: TSM>show slots ts3100
PVR slot information for library TS3100.
Library : TS3100
Product Id : 3573-TL
Support module : 2

Mount count : 0

Drives : 1
Slots : 23
Changers : 1
Import/Exports : 1

Device : /dev/IBMchanger0

Drive 0, element 256

Changer 0, element 1

ImpExp 0, element number 16
Slot 0, status Full, element number 4096, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 1, status Full, element number 4097, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 2, status Full, element number 4098, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 3, status Full, element number 4099, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 4, status Full, element number 4100, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 5, status Full, element number 4101, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 6, status Full, element number 4102, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 7, status Full, element number 4103, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 8, status Full, element number 4104, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 9, status Full, element number 4105, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 10, status Full, element number 4106, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 11, status Full, element number 4107, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 12, status Full, element number 4108, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 13, status Full, element number 4109, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 14, status Full, element number 4110, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 15, status Full, element number 4111, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 16, status Full, element number 4112, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 17, status Full, element number 4113, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 18, status Full, element number 4114, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 19, status Full, element number 4115, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 20, status Full, element number 4116, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 21, status Full, element number 4117, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY
Slot 22, status Full, element number 4118, barcode not present, barcode value <>, devT=LTO, mediaT=427, elemT=ANY

slot element range 4096 - 4118

 

[rore]

What was the complete audit command you executed? What was the result? (hint: q actlog).
Can you execute an inventory request on the TS3100 WEB admin using barcode? What is the result?

Are you using the lin_tape driver on your Linux server?

Rudy

 

[bluesman]

tsm> audit library ts3100 refreshstate=yes

02/15/2012 18:47:13 ANR2017I Administrator TSMADMIN issued command: AUDIT LIBRARY ts3100 refreshstate=yes (SESSION: 1)
02/15/2012 18:47:13 ANR0984I Process 6 for AUDIT LIBRARY started in the BACKGROUND at 06:47:13 PM. (SESSION: 1, PROCESS: 6)
02/15/2012 18:47:13 ANR8457I AUDIT LIBRARY: Operation for library TS3100 started as process 6. (SESSION: 1, PROCESS: 6)
02/15/2012 18:47:14 ANR8993I The library state refresh for TS3100 was successful. (SESSION: 1, PROCESS: 6)
02/15/2012 18:47:14 ANR8461I AUDIT LIBRARY process for library TS3100 completed successfully. (SESSION: 1, PROCESS: 6)
02/15/2012 18:47:14 ANR0985I Process 6 for AUDIT LIBRARY running in the BACKGROUND completed with completion state SUCCESS at 06:47:14 PM. (SESSION: 1, PROCESS: 6)

Executed Inventory Rescan on WEB admin. It displays all the tapes and barcodes correctly.

Yes, it's lin_tape driver on Centos 5.7, kernel 2.6.18-274.17.1.el5xen, driver version 1.41.1, also tried with latest from FixCentral, same thing...

 

[bluesman]

Missed the question before: yes, the label and checking worked before, but I'm not sure how I actually managed to get it running. (Automatically using scratch tapes never worked)

Also, when I try to label one tape which is in I/O Station and I'm getting:

02/15/2012 19:15:55 ANR2017I Administrator TSMADMIN issued command: LABEL LIBVOLUME ts3100 BIC001L4 overwrite=yes (SESSION: 2)
02/15/2012 19:15:55 ANR0984I Process 3 for LABEL LIBVOLUME started in the BACKGROUND at 07:15:55 PM. (SESSION: 2, PROCESS: 3)
02/15/2012 19:15:55 ANR8799I LABEL LIBVOLUME: Operation for library TS3100 started as process 3. (SESSION: 2, PROCESS: 3)
02/15/2012 19:15:55 ANR8323I 001: Insert ANY volume BIC001L4 R/W into entry/exit port of library TS3100 within 60 minute(s); issue 'REPLY' along with the request ID when ready. (SESSION: 2, PROCESS: 3)
02/15/2012 19:16:02 ANR2017I Administrator TSMADMIN issued command: REPLY 1 (SESSION: 2)
02/15/2012 19:16:02 ANR8499I Command accepted. (SESSION: 2)
02/15/2012 19:16:02 ANR8447E No drives are currently available in library TS3100. (SESSION: 2, PROCESS: 3)
02/15/2012 19:16:02 ANR8802E LABEL LIBVOLUME process 3 for library TS3100 failed. (SESSION: 2, PROCESS: 3)
02/15/2012 19:16:02 ANR0985I Process 3 for LABEL LIBVOLUME running in the BACKGROUND completed with completion state FAILURE at 07:16:02 PM. (SESSION: 2, PROCESS: 3)
02/15/2012 19:16:02 ANR1893E Process 3 for LABEL LIBVOLUME completed with a completion state of FAILURE. (SESSION: 2, PROCESS: 3)


The drive clearly is available:

tsm: TSM>q drive f=d

Library Name: TS3100
Drive Name: DRIVE1
Device Type: LTO
On-Line: Yes
Read Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTRIUM2,ULTRIUMC,ULTRIUM
Write Formats: ULTRIUM3C,ULTRIUM3,ULTRIUM2C,ULTRIUM2
Element: 256
Drive State: EMPTY
Volume Name:
Allocated to:
WWN:
Serial Number: 1H10058862
Last Update by (administrator): TSMADMIN
Last Update Date/Time: 02/15/2012 17:12:41
Cleaning Frequency (Gigabytes/ASNEEDED/NONE): NONE



tsm: TSM>q path f=d

Source Name: TSM
Source Type: SERVER
Destination Name: TS3100
Destination Type: LIBRARY
Library:
Node Name:
Device: /dev/IBMchanger0
External Manager:
ZOS Media Server:
Comm. Method:
LUN:
Initiator: 0
Directory:
On-Line: Yes
Last Update by (administrator): TSMADMIN
Last Update Date/Time: 02/15/2012 16:28:31

Source Name: TSM
Source Type: SERVER
Destination Name: DRIVE1
Destination Type: DRIVE
Library: TS3100
Node Name:
Device: /dev/IBMtape0
External Manager:
ZOS Media Server:
Comm. Method:
LUN:
Initiator: 0
Directory:
On-Line: Yes
Last Update by (administrator): TSMADMIN
Last Update Date/Time: 02/15/2012 17:13:45

 

[rore]

Try audit library ts3100 checklabel=barcode.

 

[bluesman]

tsm> audit library ts3100 checklabel=barcode.
Completes successfuly.

And trace of the label libvol command for a single tape. I pushed that tape in the I/O Station manually with WEB admin.
tsm>label libvolume ts3100 BIC001L4 overwrite=yes

19:42:06.172 [94][napath.c][586][naTestPath]:Testing path sourceName=TSM, destination=TS3100RIVE1, source type=2, destination type=1, initmode=0
19:42:06.172 [94][napath.c][596][naTestPath]:Starting a new transaction.
19:42:06.172 [94][nautil.c][150][NaLockUniverse]:Acquiring NA universe lock (sLock).
19:42:06.172 [94][napthcmd.c][1861][naOpenQueryPath]:Setting search bounds with srcName = $$LOCALSERVER$$
19:42:06.172 [94][napthcmd.c][1878][naOpenQueryPath]:Setting search bounds with destName = TS3100RIVE1
19:42:06.173 [94][mmsdrive.c][7039][MmsReleaseDriveList]:Releasing DriveList in library TS3100
19:42:06.173 [94][mmsdrive.c][2409][MmsBorrowDrive]:Couldn't borrow drive in lib=TS3100
19:42:06.173 [94][output.c][7227][PutConsoleMsg]:ANR8447E No drives are currently available in library TS3100.~
19:42:06.173 [94][mmsscsi.c][11537][ExtendPort]:Extending import/export station for library TS3100 using element address 16.

 

[rore]

If you do a show slots after the audit with barcode, you still get no barcodes?

At this point I would suggest to do a load/unload, read/write, inventory and library test with itdt. However, from all your posts it seems that the library and drive have no problem. Before that maybe you want to try restarting TSM and CentOS. When the OS comes back up check that the library and drive are well recognized. Then start TSM, check paths matches and see actlog for any unusual message in library initialization. If same behaviour, open a PMR.

Rudy

 

[bluesman]

That is correct, rebooted the server, started dsmserv in foreground to see any weird stuff. Everything OK. Still no barcodes.

Library tests are OK, I can easily manage it via WEB interface, I've done some checks with ITDT tool and it was all clear. So I'm gonna take a guess everything is OK with library.

I've installed lin_taped and checked the logs and I'm getting various errors depending on what I try do, as expected. But no errors if I simply do label libvol ts3100 search=yes labelsource=barcode, I just get the ANR8847E No LTO-type drives are currently available in library TS3100. I would understand this not working if I got driver scsi errors, which I don't for this particular case, neither does audit library... Strange stuff going on...

Think I'm gonna nuke DB and start over tomorrow, maybe even try 6.2 version...

 

[smajl]

Can you provide documentation about? I am almost sure that is not true.
TSM can handle tapes with 6 or 8 character barcode (xxxxxxLx OR xxxxxx).
...
Rudy

You can read something here (or on screenshot attached):
http://www.emc.com/collateral/hardware/white-papers/h8065-ibm-tsm-backup-data-domain-wp.pdf


Also here is something: http://www-01.ibm.com/support/docview.wss?uid=swg1IC43954
If the eighth position is "2" the cartridge will be treated as LTO 2 media.

And here: http://publib.boulder.ibm.com/infoc...com.ibm.storage.3584.doc/ipg_3584_mehlab.html
When read by the library's bar code reader, the bar code identifies the cartridge's VOLSER to the tape library. The bar code also tells the library whether the cartridge is a data, cleaning, or diagnostic cartridge. In addition, the bar code includes the two-character media-type identifier Lx, where x equals 1, 2, 3, 4, 5, T, U, or V. L identifies the cartridge as an LTO cartridge. 1 indicates that the cartridge is the first generation of its type; 2, 3, 4, or 5 indicates that the cartridge is the second, third, fourth, or fifth generation of its type; T indicates that the cartridge is generation 3 WORM cartridge; U indicates that the cartridge is generation 4 WORM cartridge; V indicates that the cartridge is a generation 5 WORM cartridge. Figure 1 shows a sample bar code label for the LTO Ultrium 3 Tape Cartridge.
You can order tape cartridges with the labels included, or you can order custom labels. The bar code must meet predefined specifications. They include (but are not limited to):

Eight uppercase alphanumeric characters, where the last two characters must be L1, L2, L3, L4, L5, LT, LU, or LV



Problem is that TSM queries library for volume information/identification (show slots). And if library scanned *L4 it assumes it is LTO-4 and returns mediaT=427. Based on this it asks for a drive that is able to read/write/manipulate LTO-4 format. But there is none like that -> "No drive available"

bluesman you can try to set up library to read only first 6 chars of barcode.