TDP Exchange epic failure of backup...

[spiffy]

So i got to playing around with my exchange 2003 backup, and decided i wanted to try having exchange backup directly to tape, bypassing the diskpool completely because i am having capacity issues with diskpool, and straight to tape may be slower, but would help alleviate capacity issues

I created a new mgmt class, and backup copygroup, pointing it to the tapepool(primary stgpool), validated and activated it.
I modified the exchange node dsm.opt file with specific include statements to point to the new mgmt class, and also modified the node on the TSM server to use only 1 mount point, and to use client side compression.
I did a quick test, all drives where in use, so i just ran the excfull.cmd file to see if it would start backing up to disk or tape, it tried to grab a tape, which is what i wanted. i cancelled the backup and cleared all tsm logs on the node.

The scheduled backup started ok last night, it loaded a tape and started backing up to it.
About 1.5 hours into the backup, I noticed it failed and I looked at the excfull.log file and found the following errors

Backup of storage group PROD1 Storage Group failed.
ACN5798E MS Exchange API HRESEBACKUPREADFILE() failed with HRESULT: 0xc7ff1004 - Error returned from an ESE function call (-801).

Beginning full backup of PROD2 Storage Group, 2 of 2.
Backup of storage group PROD2 Storage Group failed.
ACN5798E MS Exchange API HRESEBACKUPSETUP() failed with HRESULT: 0x3ed - The volume does not contain a recognized file system.
Please make sure that all required file system drivers are loaded and that the volume is not corrupted.

Retrying failed backups...

It does this 4 times before saying enough and fails the backup.

in the tsm activity log, the error is that the storage groups failed with an RC 402 and 425

On the exchange server, in the application log, i see the following errors



Event ID: 904
Source: ESE BACKUP
Information Store (2576) Callback function call ErrESECBPrepareInstanceForBackup ended with error 0x000003ED The volume does not contain a recognized file system.
Please make sure that all required file system drivers are loaded and that the volume is not corrupted.

So i did some searching, but am not finding exactly what i am looking for for a cause and resolution, alot of what i am finding doesnt apply to those specific errors so I am wondering if anyone has any idea on what may be going on here.

oh one thing to note, the exchange storage groups all reside on SAN storage and I dont have LAN free agents, so the exchange backup is going though the LAN to the TSM server then out through fiber to the library.

Thanks

 

[Eldoraan]

I've seen the 425 errors in the past when a backup failed for some reason (you cancelling that manual may count here). No scheduled backups would run until our Exchange admins ran the full manually. Not sure what root cause was, they handled all of that side.

 

[moon-buddy]

For me, it looks like a mailstore group corruption (ESE function call appears on the log). Take Group 2 offline, wait 30 seconds and bring it back online. This action repairs any inconsistencies.

 

[spiffy]

just last week, we stopped all exchange services for maintenance... should this have cleared any inconsistencies? , or maybe i need to talk to the exchange admin to see if there is something else going on..

thanks

 

[spiffy]

Here is something funny, we did not reset exchange at all today(unmounting the store or anything), i did however set everything on TSM back to the way it was so that exchange will backup to disk again, and it works fine....

strange...

 

[moon-buddy]

The maintenance would have fixed any inconsistencies that is why the backup ran.

 

[spiffy]

We didnt do any maintenance on the exchange server yesterday, which is when the backup direct to tape failed.
I backed out of the changes i made (exchange direct to tape) after it failed, and exchange backed up successfully to diskpool....

 

[Jeff_Jeske]

I'm curious why you don't just set the Maximum Size Threshold on your disk pool to something smaller than the exchange DB size.

Doesn't your tape drive use hardware compression? Why attempt to compress at the client level for physical tape? I would expect to see better throughput when streaming an exchange TDP to tape.

 

[spiffy]

I never played with the max size threshold setting before to be honest, that is why i havent used it.

Tape drives are not using hardware compression, i inherited it this way and really havent found the time to change it.

as for client side compression, i was more or less just testing/playing to see what happens.

thanks

 

[moon-buddy]

We didnt do any maintenance on the exchange server yesterday, which is when the backup direct to tape failed.
I backed out of the changes i made (exchange direct to tape) after it failed, and exchange backed up successfully to diskpool....

By stopping all Exchange services and restarting it again automatically fixes any database inconsistencies. This in a way "doing maintenance" even if your are not aware that you are doing so.

 

[spiffy]

yes i know that cycling the exchange services do this, but what i am saying is that we did not cycle the services or anything on the exchange server this week, i made the change to backup exchange direct to tape this week, it failed with the errors seen in my original post, so i reverted the change and exchange now backups to disk, which runs ok,, all this was done without any service restarts of exchange.

I guess i just dont understand why the backup runs fine when backing up to diskpool, but failed when backing up direct to tape.

The original configuration as of Monday - Backup to Diskpool - successsful backup that night (all previous backups to disk, successful too)

Changed the configuration Tuesday - Backup direct to tape - backup failed with the errors below

Changed the configuration back to original on Wednesday - backup to diskpool - backups have been successful

I must be missing something here in configuring TDP for Exchange to backup to tape, some tuning or something....

 

[moon-buddy]

yes i know that cycling the exchange services do this, but what i am saying is that we did not cycle the services or anything on the exchange server this week, i made the change to backup exchange direct to tape this week, it failed with the errors seen in my original post, so i reverted the change and exchange now backups to disk, which runs ok,, all this was done without any service restarts of exchange.

I guess i just dont understand why the backup runs fine when backing up to diskpool, but failed when backing up direct to tape.

The original configuration as of Monday - Backup to Diskpool - successsful backup that night (all previous backups to disk, successful too)

Changed the configuration Tuesday - Backup direct to tape - backup failed with the errors below

Changed the configuration back to original on Wednesday - backup to diskpool - backups have been successful

I must be missing something here in configuring TDP for Exchange to backup to tape, some tuning or something....

There could be a lot of reasons why direct tape backup fails. One possible explanation is that the tape "shoe shines" a lot while waiting for data causing it to stop after a preset cycle is reached. This is to protect the read/write head from premature damage. If you want to transfer to tape directly, I suggest making sure that you can sustain a fast data stream for the tape to write continuously.

I have this happen before and the only solution was to store to disk first.

 

[Mita201]

Maybe you can try to let the node to have more than one mount point allowed. I just wonder what happens if your backup reach end of tape in middle of backup, when there is only one mount point allowed.
It is just a thought.

 

[JohanW]

It mounts a new tape and starts writing to that one. It's not a problem.

Allowing two mount points and writing to primary and copy stg pools in parallel probably makes sense in this situation, BTW.

This works for me. Sorry I have no specific help on the ESE error, but it doesn't look like a TSM error, rather an Exchange one.