Backup of domino fails restarting domino server

[ohwell]

Hi all,

I have problem backing up Domino ending with Domino server restart.
Backup fails with last message:
"Waiting for Domino server.................................."


This is from backup task log:

IBM Tivoli Storage Manager for Mail:
Data Protection for Lotus Domino
Version 5, Release 5, Level 2.03
(C) Copyright IBM Corporation 1999, 2010. All rights reserved.

ACD5221I The C:\Program Files\Tivoli\TSM\domino\domdsm.log log file has been pruned successfully.

Starting Domino database backup...
Initializing Domino connection...
Querying Domino for a list of databases, please wait...

Backing up database mailbox.nsf, 1 of 1.
Full: 6 Read: 25165824 Written: 0 Rate: 0.00 Kb/Sec
Full: 6 Read: 25165824 Written: 0 Rate: 0.00 Kb/Sec
Full: 6 Read: 29360128 Written: 4194304 Rate: 1,091.68 Kb/Sec
Full: 6 Read: 33554432 Written: 8388608 Rate: 1,637.74 Kb/Sec
Full: 6 Read: 37748736 Written: 12582912 Rate: 1,965.45 Kb/Sec
[...]
Full: 4 Read: 2160066560 Written: 2139095040 Rate: 1,564.78 Kb/Sec
Full: 4 Read: 2160066560 Written: 2139095040 Rate: 1,563.31 Kb/Sec
Full: 3 Read: 2160066560 Written: 2143289344 Rate: 1,564.91 Kb/Sec

Waiting for Domino server..........................................




And that's it, no sending results to TSM server (which usually happens) or anything.


In dsmsched.log there is:

08/07/2010 18:00:18 Node Name: Lotus
08/07/2010 18:00:18 Session established with server TSMserver: Windows
08/07/2010 18:00:18 Server Version 5, Release 5, Level 4.3
08/07/2010 18:00:18 Server date/time: 08/07/2010 18:00:18 Last access: 08/07/2010 12:13:17

08/07/2010 18:00:18
Executing Operating System command or script:
%dir%\lotus_selective.cmd
08/07/2010 18:24:16 Finished command. Return code is: 5
08/07/2010 18:24:16 ANS1909E The scheduled command failed.
08/07/2010 18:24:16 ANS1512E Scheduled event 'LOTUS_SELECTIVE' failed. Return code = 5.
08/07/2010 18:24:16 Sending results for scheduled event 'LOTUS_SELECTIVE'.
08/07/2010 18:24:16 Results sent to server for scheduled event 'LOTUS_SELECTIVE'.

In above case it always happens on the same mailbox and it fails in about 80% of tries. However recently it also happend on two completely different servers, fortunatelly one-time only 'till now, so here is no rule.
When it occurs Domino server restarts, but I can't tell what happens first as the information about backup result and sending start signal to domino comes from the same time (info from windows event log) and don't know why is it happen.
It doesn't matter if it is a selective or incremental backup.

Has anyone seen something similar ?
Any help would be appreciated.




software levels:
in first case
TDP for domino version 5.5.2.3
TSM server version 5.5.4.3
Domino server version: 8.5.1.3

two other cases
TDP for domino version 5.5.2.3
TSM server version 6.1.3.1
Domino server version: 8.5.1.4

Platform: Windows 2003


Thank you in advance.

 

[TonyB]

First things first: the mailbox.nsf is a transient holder for mail data while its being routed to a user mailbox. Frankly I see little point in backing it up.

Is the Domino instance being restarted via some external task, or is the backup task killing it? Typically when Domino crashes it generates a diagnostic dump - that'd be worth looking for.

It may be that the nsf is corrupt...there are various fixup type tools that run against databases at a Domino level. Might be worth trying.

 

[ohwell]

Thank you for your reply TonyB.

There is one thing I forgot to add to my post, certain names such as the 'mailbox.nsf', schedule name 'Lotus_selective', node_name or TSM server name has been changed by me not to disclose the real names.

The 'mailbox.nsf' is a normal user mailbox, it has around 7GB and lotus domino administrator claims it's fine and often used by several users, also TSM never report any possible data corruption during this mailbox backup, so i believe that mailbox corruption maybe rule out.

First things first: the mailbox.nsf is a transient holder for mail data while its being routed to a user mailbox. Frankly I see little point in backing it up.
[...]
It may be that the nsf is corrupt...there are various fixup type tools that run against databases at a Domino level. Might be worth trying.

I can not tell what is the reason of this situation because i do not know what happens first, is backup failling because domino server crashes or does domino crashes because of backup somehow? In windows event log i see only that the exact second backup ends, there is also sent a start signal to domino service (by ntauthority\system).

I've looked through that domino dump file, but don't see anything what could crash domino - however i'm not a domino server administrator, so any suggestion for what should i look for in there would also be a great help to me.

The facts are that there is strong time correlation between this 2 events, it would be obvious that if domino server crash, backup will end, but does backup could be the cause of this crash ? It is the main question i would like to find answer for... .

Is the Domino instance being restarted via some external task, or is the backup task killing it? Typically when Domino crashes it generates a diagnostic dump - that'd be worth looking for.

 

[TonyB]

One useful reference is in the Domino Infocenter: http://publib.boulder.ibm.com/infoc...c/H_ABOUT_TROUBLESHOOTING_SERVER_CRASHES.html

Is there a correlation between the size of the NSF being backed up and the crashes? Perhaps combined with the existence of some other Domino admin task that uses a fair whack of memory?

 

[ohwell]

Thank you for answer TonyB and btw sorry for not answering sooner but i was on vacation.

As for size of mailboxes being backed up when server restarts it's about 7GB in the first case, and about 9GB in second case (one time accident), also there were no maintenance task running on the domino server side that I would know of - usually i'm being informed about such.

 

[walkman]

did you have do any changes at domino database or reset the password at domino console? i have experience with lotus domino backup failed because we have reset the domino console password.

 

[ohwell]

Domino server administrator claims that there were no such changes recently.