Problem restrieving data from a dead server to a new one

[davidgpv]

Hello to everybody.

I do not have all that much experience administering TSM but I have been forced by circunstances to learn a little. Right now I'm facing a problema de Google has not been able to help me with. We had a centraliced backup server where we would send files by FTP and them using TSM we would take them to tape. In this server we had configured several node names, one for each policy (SERVERNAME_DAILY_5YRS [runs daily and stores data in a 5 years stgpool], SEVERNAME_DAILY_1YRS [runs daily and stores data in a 1 year stgpool]). We changed this scheme to new architecture, and reused the server, preserving the nodes in TSM until the data will expire. There are backups and archives there. An user requested some data stored in one of those nodes and so I took the task to try and retrieve it (it's and archived directory). I took the easy route: I went and edited the dsm.opt and changed the NODENAME. It worked and it showed me the filespace all right... however that all I can see. It shows the dates from the archives but does not list any directories or files. I tried with the GUI Client and the dsmc but none worked.

I then used this query to list all the tapes used by the node:
select vu.node_name,vu.volume_name,vu.stgpool_name, v.access from volumeusage vu , volumes v where vu.node_name='BCR0106BAK01_DIARIO_H1A' and vu.volume_name = v.volume_name

Some appear with READWRITE access mode while others say "UNAVAILABLE", which is probably because our tape operator took it out of the library leaving it in that state (it has happened before). If I issue a "q con <volume name>", it shows the directories that I want to retrieve but the clientes keep giving me nothing.

Any ideas of what could I be doing wrong?

Sorry for the long rant (and the bad english ) and thank you for any help you can give me

PD:
I don't know how relevant this information is, but I'll put it here in case it helps you to understand the case:

===== TSM SERVER ======
TSM Version: 5.2
SO: Windows 2003 SE

==== DEAD SERVER =======
SO: Windows 2003 SE SP2
File space: \\servername\k$
TSM Client version: 5.2

===== TARGET SERVER FOR RETRIEVING =====
SO: Windows 2003 SE
Target folder: D:\Restore\
TSM Client version: 5.2
Note: This server lacks a K: drive. It just have C: and D:

 

[tariq.kaifi]

Hello,

Welcome to adsm.org!

Shortest troubleshooting first. Looks to me that you have data in your TSM server. Just see if those tapes are primary volumes and check-in them back to the library. Do a restore and see how it goes. Show us the actlog if possible just to see what goes around at that time.

 

[davidgpv]

Hello Tariq,

Thank you for your reply. For what I can tell the tapes all belong to primary pools. We do not have copy pools since we lack some resources and when we tried to implemente it was a scheduled jobs nightmare.
My main problem is that I query the tsm database for the list of volumes that belong to the node, then I take one volume and do a query content and I can see the directories that I need, but when I try to retrieve them in dsmc tsm says that no objects match my query and in the gui I just get a list of archive date, the file space but no directories of files!

I get something like this in the GUI:

BCR0106BAK01_DIARIO_H1A
|_[-]Archive Date: 01/01/2011
|_[ ]\\BCR0106BAK01\k$ (K
|_[-]Archive Date: 01/02/2011
|_[ ]\\BCR0106BAK01\k$ (K
|_[-]Archive Date: 01/03/2011
|_[ ]\\BCR0106BAK01\k$ (K
...

So, I'm at a loss with this. I always thought that TSM would list the files from the TSM Database and then when you want to retrieve/restore it ask for the tapes.

Anyway, I hope this info gives you some light about what my problem may be.

Thanks in advance for any help you can give me.


=========================== CMD PROMPT ==========================
C:\Program Files\Tivoli\tsm\baclient>dsmc -virtualnodename=BCR0106BAK01_DIARIO_H
1A
IBM Tivoli Storage Manager
Command Line Backup/Archive Client Interface - Version 5, Release 2, Level 2.0
(c) Copyright by IBM Corporation and other(s) 1990, 2003. All Rights Reserved.

Node Name: BCR0106BAK01_DIARIO_H1A
Please enter your user id <BCR0106BAK01_DIARIO_H1A>: admin

Please enter password for user id "ADMIN": *****

Session established with server BCR0106TEM11: Windows
Server Version 5, Release 2, Level 2.0
Server date/time: 09/09/2011 11:00:20 Last access: 09/09/2011 10:54:19

tsm> retrieve \\BCR0106BAK01\k$\ARCHIVE.H1A\* d:\restore\ -pick
Retrieve function invoked.

ANS1302E No objects on server match query

tsm>


=========================== TSM Admin Page ==========================
>q act log
...

09/09/2011 10:57:14 ANR0406I Session 22379 started for node BCR0106BAK01_DIAR-
IO_H1A (WinNT) (Tcp/Ip BCR0106TEM11(2175)). (SESSION:
22379)
09/09/2011 11:00:23 ANR0406I Session 22380 started for node BCR0106BAK01_DIAR-
IO_H1A (WinNT) (Tcp/Ip BCR0106TEM11(2176)). (SESSION:
22380)
09/09/2011 11:01:57 ANR2121W ATTENTION: More than 226 MB of the database has
changed and the last database backup was more than 24
hours ago. Use the BACKUP DB command to provide for
database recovery.
09/09/2011 11:01:57 ANR2841W Server is NOT IN COMPLIANCE with license terms.
09/09/2011 11:02:03 ANR0984I Process 2283 for SPACE RECLAMATION started in the
BACKGROUND at 11:02:03. (PROCESS: 2283)
09/09/2011 11:02:03 ANR1040I Space reclamation started for volume C05962L1,
storage pool LTOPOOL (process number 2283). (PROCESS:
2283)
09/09/2011 11:02:03 ANR1044I Removable volume C07298L2 is required for space
reclamation. (PROCESS: 2283)
09/09/2011 11:02:03 ANR1044I Removable volume C05962L1 is required for space
reclamation. (PROCESS: 2283)
09/09/2011 11:02:03 ANR1420W Read access denied for volume C07298L2 - volume
access mode = "unavailable". (PROCESS: 2283)
09/09/2011 11:02:03 ANR0513I Process 2283 opened output volume C13207L2.
(PROCESS: 2283)
09/09/2011 11:02:03 ANR0985I Process 2283 for SPACE RECLAMATION running in the
BACKGROUND completed with completion state FAILURE at
11:02:03. (PROCESS: 2283)
09/09/2011 11:02:03 ANR1081W Space reclamation terminated for volume C05962L1
- storage media inaccessible.
09/09/2011 11:02:03 ANR1042I Space reclamation for storage pool LTOPOOL will
be retried in 60 seconds.
09/09/2011 11:02:43 ANR0407I Session 22381 started for administrator ADMIN
(WebBrowser) (HTTP 127.0.0.1(2177)). (SESSION: 22381)
09/09/2011 11:02:43 ANR2017I Administrator ADMIN issued command: QUERY ACTLOG
(SESSION: 22381)



===NOTE===
I tried to put in this post the results from a q stgpool and a q con, but the forum has some problems with the text and returns an error. If you seem to need it, I could send it to you in antoher way if you find that you need the info.

 

[tariq.kaifi]

Hi,

Is the account you used to retrieve has same privilege as compared to the one used to archive? Sometimes, just permission issues dont let you see the data.

Also, I would like to see the "query content" output.

 

[davidgpv]

In fact I used both users, the admin and the user used for the backup (which has the same name as the node), with the same result.

Here some rows from a query content from one of the volumnes (the report is a little long).:

>q con C10018L2

Node Name Type Filespace FSID Client's Name for File
Name
--------------- ---- ---------- ---- --------------------------------------
BCR0106BAK01_D- Bkup \\bcr0106- 1 \BACKUPS.H1A\DIARIO\BACKUPFILES\BCR01-
IARIO_H1A bak01\k$ 06LTN63\LOTUS\DOMINO\DATA\ GCI.NSF
BCR0106BAK01_D- Bkup \\bcr0106- 1 \BACKUPS.H1A\DIARIO\BACKUPFILES\BCR01-
IARIO_H1A bak01\k$ 06LTN63\LOTUS\DOMINO\DATA\ GCI.NTF
BCR0106BAK01_D- Bkup \\bcr0106- 1 \BACKUPS.H1A\DIARIO\BACKUPFILES\BCR01-
IARIO_H1A bak01\k$ 06LTN63\LOTUS\DOMINO\DATA\ GCI41G.NTF
BCR0106BAK01_D- Bkup \\bcr0106- 1 \BACKUPS.H1A\DIARIO\BACKUPFILES\BCR01-
IARIO_H1A bak01\k$ 06LTN63\LOTUS\DOMINO\DATA\
GCIARCHIVO.NTF

BCR0106BAK01_D- Arch \\bcr0106- 1 \ARCHIVES.H1A\ DIARIO
IARIO_H1A bak01\k$
BCR0106BAK01_D- Arch \\bcr0106- 1 \ARCHIVES.H1A\DIARIO\ WEBSPHERE
IARIO_H1A bak01\k$
BCR0106BAK01_D- Arch \\bcr0106- 1 \ARCHIVES.H1A\DIARIO\WEBSPHERE\
IARIO_H1A bak01\k$ BCR0106AWS43
BCR0106BAK01_D- Arch \\bcr0106- 1 \ARCHIVES.H1A\DIARIO\WEBSPHERE\
IARIO_H1A bak01\k$ BCR0106AWS44



I need to retrive this directory: "\ARCHIVES.H1A\DIARIO\WEBSPHERE\"

 

[unpaired]

Dear davidgpv,

you need retrive "\ARCHIVES.H1A\DIARIO\WEBSPHERE\" file ?
1) Login client
2) q arch "\ARCHIVES.H1A\DIARIO\WEBSPHERE\" -subdir=yes then look archive information. if archive file is there run the retrieve command.
3)login tsm server and run "q req" you will see request volume if volume is offsite.

kind regards.

 

[cwilloug]

Was the FTP server a windows server or other OS? You can not restore UNIX to a windows OS, or vice versa (windows to UNIX/Linux).

And I'd suggest, since you're on a windows machine, I'd suggest trying to restore files using the gui vs. the command line, just check the "show active and inactive versions" box in the gui to try to restore your files.

 

[davidgpv]

I used dsmc to run your instruction and it gave me this:

C:\Program Files\Tivoli\TSM\baclient>dsmc -virtualnodename=BCR0106BAK01_DIARIO_H1A
IBM Tivoli Storage Manager
Command Line Backup/Archive Client Interface - Version 5, Release 2, Level 2.0
(c) Copyright by IBM Corporation and other(s) 1990, 2003. All Rights Reserved.

Node Name: BCR0106BAK01_DIARIO_H1A
Please enter your user id <BCR0106BAK01_DIARIO_H1A>: admin

Please enter password for user id "ADMIN": **************

Session established with server BCR0106TEM11: Windows
Server Version 5, Release 2, Level 2.0
Server date/time: 09/13/2011 15:52:53 Last access: 09/13/2011 15:49:31

tsm> q arch "\ARCHIVE.H1A\DIARIO\WEBSPHERE\" -subdir=yes
ANS1083E No files have previously been archived for '\ARCHIVE.H1A\DIARIO\WEBSPHE
RE\'
tsm>
=======================================================
It didn't even ask for a tape, it just sent that message.

 

[davidgpv]

All the server involved in this process are Windows. I have tried with the GUI and the dsmc, but in the gui it shows al the archives that were done daily, but when you try to expand and see directories and files, it shows nothing past the filespace "\\BCR0106BAK01\k$\". As you can see I have queried the database to get the tapes and the queried the content to see if the info is there and I see the directories, but then I use the dsmc it just says that there are no files archived or that match the pattern in the retrieve. I just don't getting... What am I doing wrong?
When we had the original server working, I could retrieve the files without problem, but now it says that there are no files even when I query the tapes and it looks like the data is in them...

 

[Harry_Redl]

Hi,

show us "q files" from the client side ...
Try:
q archive \\bcr0106bak01\k$\ARCHIVE.H1A\* -sub=yes
q archive \\bcr0106bak01\k$\ARCHIVE.H1A\DIARIO\WEBSPHERE\* -sub=yes

If it does not help, we can try selects against server database to get the exact HL_NAME/LL_NAME of the files ... the we can try "direct" restore.

Harry

 

[davidgpv]

Hi Harry_Redl,
The following is the output for both q arch's:
========================================================================

tsm> q arch \\BCR0106BAK01\k$\ARCHIVE.H1A\* -sub=yes
Session established with server BCR0106TEM11: Windows
Server Version 5, Release 2, Level 2.0
Server date/time: 09/13/2011 16:20:01 Last access: 09/13/2011 15:49:31

ANS1092W No files matching search criteria were found
tsm> q arch \\BCR0106BAK01\k$\ARCHIVE.H1A\DIARIO\WEBSPHERE\* -sub=yes
ANS1092W No files matching search criteria were found
tsm>
================================================
Which table can I query to get the HL_NAME/LL_NAME?

Thank you for your help!

 

[Harry_Redl]

Hi,

output can be long ...

select node_name,filespace_name,type,hl_name,ll_name,archive_date from archives where node_name='BCR0106BAK01_DIARIO_H1A'

Harry

EDIT: make sure the client level you are using for restore is the same or higher than the original!!!!

 

[davidgpv]

Hi Harry,

I got several lines from this query. Between them is this:
===========================================
...
NODE_NAME: BCR0106BAK01_DIARIO_H1A
FILESPACE_NAME: \\bcr0106bak01\k$
TYPE: DIR
HL_NAME: \ARCHIVES.H1A\DIARIO\
LL_NAME: WEBSPHERE
ARCHIVE_DATE: 2010-11-28 02:22:27.000000
...
NODE_NAME: BCR0106BAK01_DIARIO_H1A
FILESPACE_NAME: \\bcr0106bak01\k$
TYPE: FILE
HL_NAME: \ARCHIVES.H1A\DIARIO\WEBSPHERE\BCR0106AWS43\
LL_NAME: PRODUCCION_BANCO_CUSTOM_01DEC10_0400.LOG.Z
ARCHIVE_DATE: 2010-12-02 02:12:38.000000
.....
==========================================

Does this mean that I really have the data in the tapes?

Thanks

David

 

[Harry_Redl]

Hi,

this is just a directory - can you post any example of TYPE:FILE ?
What about the B/A client level? Do you have the same level or higher? You are using 5.2.x - can you use 5.3?

Harry

P.S. what about the "q files" from the client?


EDIT: sorry - there is the example I asked for ...
try

retrieve {\\bcr0106bak01\k$}\ARCHIVES.H1A\DIARIO\WEBSPHERE\BCR0106AWS43\PRODUCCION_BANCO_CUSTOM_01DEC10_0400.LOG.Z C:\

Harry

 

[davidgpv]

Hi,

Here is the output from dsmc:

===================================
tsm> retrieve {\\bcr0106bak01\k$}\ARCHIVES.H1A\DIARIO\WEBSPHERE\BCR0106AWS43\PRODUCCION_BANCO_CUSTOM_01DEC10_0400.LOG.Z C:\
Retrieve function invoked.

Session established with server BCR0106TEM11: Windows
Server Version 5, Release 2, Level 2.0
Server date/time: 09/13/2011 16:54:25 Last access: 09/13/2011 16:31:17

ANS1302E No objects on server match query

tsm>

=====================================================================
About your questions:
>>What about the B/A client level? Do you have the same level or higher?
I'm trying to do the restore in the TSM Server using the TSM Client, so the client and server versions are the same. I don't remember the client version that web used in the dead server, but I think that it was newer. Does it affect the process?
>>You are using 5.2.x - can you use 5.3?
I don't think it is possible to update it right now. We currently have TSM as a mean to retrieve the backups and archives that we did when it was our main backup solution, so If it takes money it could take some time. Then there is the fact that I'm not very experimented in this plataform, so I should probably need some help :redface:.

About the "q files", what do you mean by the client? What I did was to open a "dsmc -virtualnodename=BCR0106BAK01_DIARIO_H1A" and then ran the instructions. Should I have used another command?

Thank you for all your help

 

[Harry_Redl]

Hi,

from all you have posted here I can say three things:
a) data is there - TSM server knows about it => it can be retrieved
b) There is no nasty error in the server activity log during the attempt to retrieve the data (you did not mention any) - so it seems the server has no problem - you do NOT need to touch it (upgrade it)
c) This leads me to the conclusion that the problem is in the client. These "No objects in server match query" messages can happen in two situations - 1) data is not there, 2) newer client was used for backup/archive than is used for restore/retrieve
The newer client updates the filespace on the server during the backup/archive operation and the older client does not understand the format.
Try to install newer client (can be on your laptop) and start it using "dsmc -virtualnodename=BCR0106BAK01_DIARIO_H1A" - then try to (first) query and then retrieve the data - if it does not work with 5.3, try 5.4 etc. until you find working version (the one you used for archive).

Harry

 

[davidgpv]

Hi Harry,

Sorry that I hadn't replied before but I came down with the flu and was sent home. It makes a lot of sense what you states. In fact I can say that the backups were down with a newer version that the one I'm using to restore. To be honest I didn't knew there was a difference. I'll follow your advice and ask for a newer version on Monday to try and do the recovery. Whatever the results are I'll be posting them here.

Thank you again for all your help.