TSM Server instance rebuild from mirrored SAN disks to different pre-existing TSM Server

[JuniorStorageGuy]

Hi All

I'm busy testing a faster DR solution for our two TSM servers so that we can DR faster and carry on working while a fresh TSM Server is being rebuilt. Our environment and solution is as follows:

We have 2 TSM 7.1 servers (AIX) in seperate buildings running 2 seperate instances:
Building 1 = TSM Server1 = instance1
Building 2 = TSM Server2 = instance2
We would like have the opposing TSM Servers to act as the DR locations for their respective primary instances.
Building 1 = TSM Server1 = instance1(PRI) + instance2(DR)
Building 2 = TSM Server2 = instance2(PRI) + instance1(DR)

In a DR situation, we would like to fail over as soon as possible rather than wait for a TSM DB restore.
So we tried the following:

• Mirroring the TSM DB volumes, log volumes and instance home directories at SAN level.We kinda preplanned doing this and gave each of the file systems names that were unique.
• On a test environment we restored instance1 successfully.
• We created a new a TSM instance using the /opt/tivoli/tsm/server/bin/dsmicfgx wizard. This new instance was created to resemble instance2, i.e instance name, file systems, db locations, log locations, etc. The only difference was that it was started on a different port. This was successful so we halted this new instance after ensuring success.
  
• Broke the SAN mirror and zoned the mirrored disks to the test TSM environment. We then assigend the disks to the neessary file systems, retaining the data on them. I then edited the dsmserv.opt to use port 1502.
  
• Tried starting the TSM instance2 using the instance owner. Which failed with the follwing error: DBI1306N The instance profile is not defined.

We recreated everything as is on the instance2, including instance directories and instance users and instance user groups. I think this should have worked as TSM calls any instance it creates TSMDB1.

Is this even possible? If not, is there a similar solution that can be achieved?

Thank You

 

[marclant]

Probably best if you use HADR instead. So let DB2 handle the mirroring of the DB. It's actually not mirroring though, DB2 sends the logs from one instance to the other, and the logs are applied. So essentially all transactions that happen on one instance are also executed on the other instance.

Alternatively, you could have just 1 instance on each server, and use node replication. You may need more hardware for this though.

 

[Phil_02GT]

A client of mine also replicates the TSM setup at the SAN level like you described it except that the DR server does not have it's own TSM instance. It is only used as a DR server.
In my case, I did not run dsmicfgx since all the information that DB2 needs is already there. My guess is that you probably wiped out some info that DB2 needed by running it

A few things you might want to check for, (at least they applied to my setup):

• Not running dsmicfgx I was missing the necessary DB2 related entries in /etc/services
  
  
• I ran into a problem where the DB2 database was in an inconsistent state, which you could also run into.
  tsminst1@xxxxxx(/home/tsminst1)$ db2 connect to tsmdb1
  SQL1015N The database is in an inconsistent state. SQLSTATE=55025
  
  tsminst1@xxxxxx(/home/tsminst1)$ db2 RESTART DATABASE tsmdb1
  DB20000I The RESTART DATABASE command completed successfully.
  
• As a worst case scenario, I also have a database backup on tape at the DR site that I cas use.
  
  
• After that you have to fix the usual stuff like redefining the tape library, audit volumes, etc

Some interesting references. Whether you decide to go with HADR or not, these helped me to understannd the whole thing:

• IBM pdf: Configuring an IBM Tivoli Storage Manager cluster with IBM Tivoli System Automation for Multiplatforms
• https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/Electronic%20vaulting%20using%20deduplicated%20remote%20copy%20storage%20pools?section=Introduction

 

[nenifled]

I would move my TSM to GPFS. Have the filesystems on both sites. Like this:

http://tsm2015.uni-koeln.de/10191.html#c1948


GPFS as platform for TSM Server - a fortunate marriage
Jochen Zeller

Brief introduction in GPFS; advantages for TSM server when using GPFS as filesystem platform (easy filesystem layout, high availability, flexibility in using storage capacities, cloudlike); TSM storage pools on IBM ElasticStorageServer (low cost, very fast, flexible); experiences gained from a 1 Pbyte project; hints & tips for configuring a TSM on GPFS environment;

...that REALLY rocks!