ADSM-L

Re: [ADSM-L] Database corruption

2010-05-20 11:04:25
Subject: Re: [ADSM-L] Database corruption
From: "Loon, EJ van - SPLXM" <Eric-van.Loon AT KLM DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Thu, 20 May 2010 17:02:25 +0200
Hoi Paul!
Thank you very much for your reply! 
I have an PMR open for several weeks, but Level 2 can't help me any
further too. They said "If it doesn't bother you during daily
operations, just ignore the corruptions". The problem is that I don't
know if it bothers me. I'm not 100% sure I can restore the primary pool
at the moment, nor am I 100% sure all clients can be restored. I think
so, but I wouldn't bet my life on it...
Kind regards,
Eric van Loon
KLM Royal Dutch Airlines

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Paul Zarnowski
Sent: donderdag 20 mei 2010 15:50
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: Database corruption

Rather than start with an empty server "B", start with a copy of your
corrupted database as server "B".  While you are running the audit on
server "A", direct your backups to server "B".  Hopefully the exports
you do after the audit on "A" completes from "B" back to "A" will work,
without importing any corruption.  After you have re-merged the recent
additions to "B" back into the cleaned up "A", you can discard "B" along
with its corruption.  I think this should work, but may depend on what
the corruption is.

We went through a similar experience about a decade ago.  We had
mirrored dbvols, so we just split the mirrors to clone a "B" off of our
damaged "A".

If you haven't already talked to support, they might also be able to
reach in and tweak your database to clean out the corruption, depending
on the nature of it and how widespread it is.  It's worth a shot.

Good luck.

..Paul


At 09:06 AM 5/20/2010, Loon, EJ van - SPLXM wrote:
>Hi Neil!
>It may be able to handle the load, but I don't have the storage
capacity
>on the backend on this server.
>When you direct the backups to the other server, all BA clients
>basically start all over again with a 'full' backup and I don't have
>that capacity on the spare server...
>Thank you very much for your reply!
>Kind regards,
>Eric van Loon
>KLM Royal Dutch Airlines
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
Of
>Strand, Neil B.
>Sent: donderdag 20 mei 2010 14:57
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: Re: Database corruption
>
>Eric,
>   I ran into a similar problem 2 years ago - Corrupted DB.
>Fortunately, I had a spare server that I quickly set up and moved all
>backup operations to that server.  It took a few days to run through
the
>DB fixes on the corrupt server.  After the DB was fixed, I just did
>server-server exports of all of the old data to the new server.
>   If your test server can temporarily handle the load, you may be able
>to run backups to it for the 3 or 4 days it takes to fix the corrupt DB
>and then swing everything back to the original server using server to
>server exports.
>
>Cheers,
>Neil Strand
>Storage Engineer - Legg Mason
>Baltimore, MD.
>(410) 580-7491
>Whatever you can do or believe you can, begin it.
>Boldness has genius, power and magic.
>
>
>-----Original Message-----
>From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
Of
>Loon, EJ van - SPLXM
>Sent: Thursday, May 20, 2010 7:08 AM
>To: ADSM-L AT VM.MARIST DOT EDU
>Subject: [ADSM-L] Database corruption
>
>Hi TSM-ers!
>One of our servers have database corruptions. An audit of a copy of the
>production database restored on our test environment revealed them.
>Since we do not know the impact of these errors (can all client data be
>restored, can I restore all primary volumes in case of a restore
>stgpool?) I definitely like to fix these errors.
>I ran a AUDITDB ARCHSTORAGE on this server, but that does not fix them,
>only a full audit does (I have proven this on the test server).
>The problem is that a full audit runs for 60 hours and I cannot afford
a
>60 hours downtime. Most of our Oracle and SAP clients haven't got
enough
>achivelog space to survive.
>I tried several things to trick TSM like create a snapshot on
>production, immediately followed by a full backup, then restore the
>snapshot on the test server, perform the full audit on this copy and
>than create a full backup on the fixed database. This way both fulls
>have the same sequence number, so I was hoping I could then restore the
>fixed copy on the production server and apply all incrementals made on
>the production server. Bad luck, TSM apparently stores timestamp
>information about previous full backups as part of the incremental
>backups:
>
>ANR4651E Restore of backup series 1733 operation restore is not in
>sequence; backup is part of another log epoch.
>
>Explanation:
>During a DSMSERV RESTORE DB, a backup volume was mounted that is not in
>the correct sequence. The current backup operation cannot be restored
in
>this series because it belongs to the same backup series from another
>point in time.
>
>Bummer... The only thing I can think of now is making a snapshot copy
>and restore it on the test server, perform a full audit here and freeze
>ALL housekeeping processes on the production server.
>On the production server perform an EXPORT SERVER FILEDATA=ALL
>FROMDATE=TODAY-1 FROMTIME=NOW every day at the same time.
>As soon as the audit finishes on the test server, create a snapshot and
>restore it on the production server and import all export volumes
>created.
>
>Am I missing anything or should this work?
>Import as well as export are single processes, so performance can be an
>issue here...
>
>Thank you very much for your replies in advance!
>Kind regards,
>Eric van Loon
>KLM Royal Dutch Airlines
></pre>********************************************************<br>For
>information, services and offers, please visit our web site:
>http://www.klm.com. This e-mail and any attachment may contain
>confidential and privileged material intended for the addressee only.
If
>you are not the addressee, you are notified that no part of the e-mail
>or any attachment may be disclosed, copied or distributed, and that any
>other action related to this e-mail or attachment is strictly
>prohibited, and may be unlawful. If you have received this e-mail by
>error, please notify the sender immediately by return e-mail, and
delete
>this message.<br><br>Koninklijke Luchtvaart Maatschappij NV (KLM), its
>subsidiaries and/or its employees shall not be liable for the incorrect
>or incomplete transmission of this e-mail or any attachments, nor
>responsible for any delay in receipt.<br>Koninklijke Luchtvaart
>Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is
registered
>in Amstelveen, The Netherlands, with registered number  3014286
><br>********************************************************<pre>
>
>IMPORTANT:  E-mail sent through the Internet is not secure. Legg Mason
>therefore recommends that you do not send any confidential or sensitive
>information to us via electronic mail, including social security
>numbers, account numbers, or personal identification numbers. Delivery,
>and or timely delivery of Internet mail is not guaranteed. Legg Mason
>therefore recommends that you do not send time sensitive
>or action-oriented messages to us via electronic mail.
>
>This message is intended for the addressee only and may contain
>privileged or confidential information. Unless you are the intended
>recipient, you may not use, copy or disclose to anyone any information
>contained in this message. If you have received this message in error,
>please notify the author by replying to this message and then kindly
>delete the message. Thank you.
>********************************************************
>For information, services and offers, please visit our web site:
http://www.klm.com. This e-mail and any attachment may contain
confidential and privileged material intended for the addressee only. If
you are not the addressee, you are notified that no part of the e-mail
or any attachment may be disclosed, copied or distributed, and that any
other action related to this e-mail or attachment is strictly
prohibited, and may be unlawful. If you have received this e-mail by
error, please notify the sender immediately by return e-mail, and delete
this message.
>
>Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
its employees shall not be liable for the incorrect or incomplete
transmission of this e-mail or any attachments, nor responsible for any
delay in receipt.
>Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch
Airlines) is registered in Amstelveen, The Netherlands, with registered
number 33014286
>********************************************************


--
Paul Zarnowski                            Ph: 607-255-4757
Manager, Storage Services                 Fx: 607-255-8521
719 Rhodes Hall, Ithaca, NY 14853-3801    Em: psz1 AT cornell DOT edu
********************************************************
For information, services and offers, please visit our web site: 
http://www.klm.com. This e-mail and any attachment may contain confidential and 
privileged material intended for the addressee only. If you are not the 
addressee, you are notified that no part of the e-mail or any attachment may be 
disclosed, copied or distributed, and that any other action related to this 
e-mail or attachment is strictly prohibited, and may be unlawful. If you have 
received this e-mail by error, please notify the sender immediately by return 
e-mail, and delete this message. 

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its 
employees shall not be liable for the incorrect or incomplete transmission of 
this e-mail or any attachments, nor responsible for any delay in receipt. 
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch 
Airlines) is registered in Amstelveen, The Netherlands, with registered number 
33014286
********************************************************

<Prev in Thread] Current Thread [Next in Thread>