Veritas-bu

[Veritas-bu] Disaster Recovery of NBU

2007-10-03 11:22:25
Subject: [Veritas-bu] Disaster Recovery of NBU
From: <peterschmidt AT northwesternmutual DOT com>
To: <veritas-bu AT mailman.eng.auburn DOT edu>
Date: Wed, 3 Oct 2007 09:56:21 -0500

Good morning all -

I'm hoping to get some feedback on disaster recovery - but of NBU itself - for the cases where a site doesn't exist anymore.

For purposes of this question - assume that we have two sites, "East" and "West".
Master and media servers are HPUX running NBU 6.
Each site has - a handful of media servers, a library, and visibility to the other site.
The Symantec binaries and all associated files are stored on the SAN, which is mirrored in real time from site "East" to site "West".

This "disk" is mounted to a live master server in site "East", with the ability to be mounted to site "West" in case of site "East's" demise.

Both sites have a master server, with "East's" being live. In a disaster, "West's" would be renamed, the disks mounted, and then it would become the new master server.

We do not have clustering of any kind (in these systems), and the two sites are 50 miles from each other, connected via (fast) IP and DWDM, though each site is on a different subnet.

Having painted that picture - do any of you use/do similarly? I'm sure there are more deluxe versions of this, but...

This brings up a few questions…

First - since the disks are mirrored in real-time, it would seem to me that if the primary site's DB somehow became damaged, the DB has now been damaged at both sites in real time. I posed this question to Symantec and followed it with a question - "is there a way to 'check' NBU's DB before NBU is completely started… and if so… is there a way to resolve any discrepancies within the database?" The response was no on all accounts - no way to check (if the DB is good on start up, NBU will start, whereas, if it's bad, it will either continue "broken" or fail to start at all", and no way to repair. If it's broken, your only alternative is to recover from your last catalog backup. Not exactly what I wanted to hear of an "enterprise class" product… but even in that explanation, there seemed to be some glimmer of hope that it's doing some checks to know that it is or is not "corrupt".

Has anyone had this experience - where the DB has been damaged and NBU would not start? Was it repairable, or did you lose data?

A way around this would be to "snapshot" the disks/DB on a periodic interval…however, if you're not quiescing the DB, how good is your snapshot? Is there a way to quickly and safely quiesce the DB within NBU without shutting down the whole environment?

Next, with NBU 6, and the adoption/integration of a formal database, it would seem to me that there ought to be some roll (forward or back) ability within NBU, which would lead me to believe that somehow - you should be able to put the DB back to a stable state with -minimal- loss of data. If not, and you have a "disaster" which takes out your primary site, the tool you would use to bring back your enterprise is itself a liability. Thoughts?

How have others prepared NBU itself for disaster? From what I understand, NBU doesn't support active/active - which would mean there is still the core dependence on the DB - wherever it may be. Multiple master servers may be an option, but not without an enormous amount of work importing and exporting. There are always catalog hot backups I suppose, though still with a potential loss of hours of data (though even that is still better than days of loss).

Pete Schmidt
Northwestern Mutual Life
Peter.Schmidt AT northwesternmutual DOT com

This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications received via the Northwestern Mutual Secure Message Center are secure. Communications that are not received via the Northwestern Mutual Secure Message Center may not be secure and could be observed by a third party. Thank you for your cooperation.

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
<Prev in Thread] Current Thread [Next in Thread>