Setting up TSM for Data Domain and Data Domain for TSM

PREDATAR Control23

Sorry but I cannot share the script as this is part of company rules.

However, I can guide you on how to put one together.

Primary site:

1. Setup the DD device to replicate to the DD on the DR site
2. Setup the TSM on the primary side to backup the DB to the DD device (through NFS) plus the RP, volhist and devconfig files.

DR site:

1. Setup the DD device to receive replication from the primary site
2. Setup a TSM instance that has the same configuration as the original TSM instance
3. Create a script that:

- will create snapshots of the primary DD into the DR DD
- mount (NFS) the snapshots into the DR TSM server - the snapshots are normally created by the DD device with dates in it
- the date-defined snapshots will be a placeholder to easily determine which DBB volumes are needed for recovery

4. append to the script TSM restore functions:

- extract the necessary data from the RP File, etc
- restore the TSM instance via dsmserv restore ... (since the DB, logs, configuration, etc. can be seen by the DR instance via the snapshot mounts)

After this works, you can put this into CRON and you have an automated recovery mechanism.


Wanted to ask you, how would you test to make sure the DR TSM server works with DR DataDomain, while your Prod TSM server is up. If you do a DSMSERV restore, DR will get the Prod Name and in DNS will have same name but 2 ips ( Prod and DR).
 
PREDATAR Control23

Wanted to ask you, how would you test to make sure the DR TSM server works with DR DataDomain, while your Prod TSM server is up. If you do a DSMSERV restore, DR will get the Prod Name and in DNS will have same name but 2 ips ( Prod and DR).

They may have the same name but different IP addresses.

Just make sure that the IP address of the DR is not routed to the PRODUCTION environment.

I bring up the DR TSM server from time-to-time and without the IP of the DR TSM server routed to the PROD, I don't have issues.
 
PREDATAR Control23

ok Thanks!! One more question as I am on my way of eliminating tapes.. The question is how to delete TSM database backup volumes which will now go onto Data Domain, as when the backups were going on LTO media, the delete volhist t=dbb does the trick.
 
PREDATAR Control23

ok Thanks!! One more question as I am on my way of eliminating tapes.. The question is how to delete TSM database backup volumes which will now go onto Data Domain, as when the backups were going on LTO media, the delete volhist t=dbb does the trick.

The command "delete volhist t=dbb" is a generic one and applies for any type of backup media.
 
PREDATAR Control23

So today, I tried to perform a dsmserv db restore to the "Cold" TSM server attached to the DR Data Domain.
After fixing the Devconf.dat and Volhist.dat, I tried performing DBB restore, however, I am getting permission denied accessing TSM DBB vols.

Turns out the owner of the TSM DBB Volumes, is NX , 503 . ( This is default owner when TSM writes to Data Domain) and it does not want to give permission to tsminst1.


Has anyone seen the permission issue. I tried with source Data Domain as well but same permission issue.

Pls help, Thank you!




==================================================================================
[tsminst1@tsmserver inst1]$ /opt/tivoli/tsm/server/bin/dsmserv restore db todate=03/03/2016 totime=14:02:07
ANR7800I DSMSERV generated at 00:47:55 on Feb 22 2012.


Tivoli Storage Manager for Linux/x86_64

Version 6, Release 3, Level 1.000


Licensed Materials - Property of IBM


(C) Copyright IBM Corporation 1990, 2011.
All rights reserved.
U.S. Government Users Restricted Rights - Use, duplication or disclosure
restricted by GSA ADP Schedule Contract with IBM Corporation.


ANR7801I Subsystem process ID is 15955.
ANR0900I Processing options file /opt/tsm/inst1/dsmserv.opt.
ANR7814I Using instance directory /opt/tsm/inst1.
ANR4726I The ICC support module has been loaded.
ANR1636W The server machine GUID changed: old value (), new value (04.93.fc.19.fa.98.e5.11.92.4c.44.a8.42.47.31.1f).
ANR8200I TCP/IP Version 4 driver ready for connection with clients on port 1500.
ANR4634I Starting point-in-time database restore to date 03/03/2016 02:02:07 PM.
ANR4591I Selected backup series 4389 from 03/03/2016 and 02:02:07 PM as best candidate available for restore database processing.
ANR4592I Restore database backup series 4389 includes eligible operation 0 with volume /opt/tsm/ddrbkup/57031727.DBV
having sequence 10001 and using device class DDRTSMDB.
ANR4598I Validating database backup information for selected backup series 4389 and operation 0 using volume /opt/tsm/ddrbkup/57031727.DBV.
ANR9905W Unable to open file /opt/tsm/ddrbkup/57031727.DBV. Permission denied.
ANR8501E System return code 13 received opening volume /opt/tsm/ddrbkup/57031727.DBV with file name
/opt/tsm/ddrbkup/57031727.DBV
ANR1401W Mount request denied for volume /opt/tsm/ddrbkup/57031727.DBV - mount failed.
ANR4602E No volumes found for TODATE 03/03/2016 02:02:07 PM.
ANR4578E Database backup/restore terminated - required volume was not mounted.
[tsminst1@usbostsm01 inst1]$
 
PREDATAR Control23

To restore from the DR Data Domain, the snapshot MUST be owned by the same owner - this means that the PROD instance owner=DR instance owner. In other word, the DR MUST be a replica of the PROD.
 
PREDATAR Control23

Yes, I know that the owner needs to be instance owner tsminst1 tsmsrvrs that is why it is failing to mount the dbb volume and dsmserv db restore fails.

Upon looking at direc permissions of the datadomain NFS mounted on tsm server, I noticed the owner permission gets changed to some unknown user:group.

As strange as it sounds, the folder owner keeps changing when the folder is mounted. I am not sure why.
I have deleted the folder, recreated it, with tsminst1:tsmsrvrs, made sure it shows the owner as that. Then created mounts on fstab, as soon as I reboot and it gets mounted, the owner of that folder gets changed to someone unknown. And then I cannot change it to tsminst1.

Any suggestions.
 
PREDATAR Control23

Attention has separate CIFS and RMAN on 2 mtree because you have performance problems if you put everything on the same.
 
PREDATAR Control23

For the replica site to have the same permissions as the origin, you have make sure:

1. The source site has to have directory replication enabled and not MTREE IF you are sharing the MTREE with other instance/s;
2. The NFS mounts on a main directory like /DDmount1 with sub-directories like /DDmount1/tsmserver1/wintel where the owner of ../tsmserver1/wintel is the TSM instance, /DDmount1 is owned by root with 777 permissions set;
3. on the replica (DR) side, a snapshot is created and the same TSM instance should be setup: this means same TSM instance owner and TSM server name

With this setup, you should not have any owner naming changing.
 
Last edited:
PREDATAR Control23

Ok I was able to setup everything, tested and so I thought it would good to reply what I did for any other folks needing to do the same and eliminating tapes.

So as the Data Domain MTREE replication to the DR Data Domain keeps everything in "Read-Only". Therefore I created a FastCopy, and mounted that to the DR TSM server ( created Devcl t=File ). This will need to be done upon a actual failure of Both Primary DD and TSM server, if I want to continue to run backups from DR Site. If you just want read only, then just point to the mount for replicated Mtree snap.

In another senario, If just the TSM server is lost at the primary site and I want to use the Primary DD with DR TSM, then the mounts to the Primary DD, the ownership needs to be taken over by the instance id "tsminst1" , even though it is the same id on both.
So, chown -R tsminst1:tsmsrvrs /...to the nfs / needs to be done when using DR TSM, and then again when the Primary TSM is rebuild after Disaster because as soon as you take owership from DR TSM svr, it changes owner on Primary side to user defined on the linux svr. if you do a ls -al you will see the ownership changes.

The only thing I am not happy with eliminating tapes is that if Primary DD vol is corrupted the MTREE replicates corruption. Back in tape days, if a particular volume gets corrupted, it can restore that one vol, just like when a primary pool vol gets corrupted, a LTO media can be used to restore that vol.

And also wish that if TSM allowed the creation of "Copy Pool" to the DR Data Domain Mtree , BUT the replication between DD happend at DD level as it is happening now and not by Backup Stg pool ( from w/ in TSM) would send data in native format. Hopefully, they write DDboost for TSM integration to avoid this.
 
PREDATAR Control23

Hy

I recommend séparation between tsm and oracle because on one mtree you can not make quota and you risk filesystel full .
Secondly, it is not the same type of file, and i / o are not the same
 
PREDATAR Control23

In regards to "fixing the Devconf.dat and Volhist.dat", does this reference updating the DR TSM instance with the information for the DR datadomain hostname and pathanme/location of the DB backup? After DB restore, the recovered DR TSM instance would need to have its Device Configuration updated again, correct?
 
PREDATAR Control23

In regards to "fixing the Devconf.dat and Volhist.dat", does this reference updating the DR TSM instance with the information for the DR datadomain hostname and pathanme/location of the DB backup? After DB restore, the recovered DR TSM instance would need to have its Device Configuration updated again, correct?

The devconf and volhist file does not need to be fixed as the snapshots should have the same location as the original if you have step the replication correctly.
 
PREDATAR Control23

The devconf and volhist file does not need to be fixed as the snapshots should have the same location as the original if you have step the replication correctly.

If the DD was mounted via a CIFS share the DR DD would have a different hostname, how would you account for that? Also, I was thinking of the situation where you wanted to perform a DR test keeping replication processing. Here you would take a snapshot on the DD and mount via a different pathname, At least that is what is shown in the most recent (2013 EMC DD recovery document, see attached. Is there another process that would be better?
 

Attachments

  • h12254-tsm-datadomain-recovery-wp.pdf
    569.6 KB · Views: 8
PREDATAR Control23

In my case the path name is not an issue since I am running under Linux. All I have to do is create a link (the link name is the same as the source) to the DR Data Domain (different path name). In the eyes of the OS and TSM, it is looking at the same path on the DR side. Restores will carry on without changes in the devconf and volhist files.

You can also create symbolic link with Windows:

https://technet.microsoft.com/en-us/library/cc753194(v=ws.11).aspx
 
PREDATAR Control23

In my case the path name is not an issue since I am running under Linux. All I have to do is create a link (the link name is the same as the source) to the DR Data Domain (different path name). In the eyes of the OS and TSM, it is looking at the same path on the DR side. Restores will carry on without changes in the devconf and volhist files.

You can also create symbolic link with Windows:

https://technet.microsoft.com/en-us/library/cc753194(v=ws.11).aspx

Great, thank you.
 
PREDATAR Control23

HADR is for the DB only. You have to prime the DB on the standby server first by restoring it, then after that it's just replicating the logs and applying them. DD takes care of replicating your data.

Marclant,

Does it apply to Electronic Vaulting with DB2 HADR and VTL Mirroring?

There is a very old (2012?) PDF file explaining the basics of DRM (Performing Disaster Recovery with Tivoli Storage Manager by Dave Daun – IBM Advanced Technical Skills). The document mentions this technique but does not go into detail.

I'll try to upload it here.
 
Top