Len,

 

Thanks for your interest in our issue…

 

-          When you say that all your backup data resides on 2 Data Domain storage devices, I assume you mean just the backup media , tape or disk pool, and not the disk unit holding the restored data?

 

All backup data created by netbackup is sent straight to one of 2 DataDomain restorers in our home data center.  That data is then replicated to identical DataDomain appliances in our DR hosting site in Philadelphia.  All of the images that we restore during either a DR exercise or a regular recovery of some kind come from the DataDomain appliances.  The DataDomain appliances are not used for primary storage or as target for any restored data.  All data was being restored to SAN attached disk on each host.

 

-          Did Netbackup report a failure to restore files? If so what reason was it reporting.

 

NetBackup did not report any errors during the restores.  Our signal that files / data was missing was that after restoring what we believed to be entire mountpoints, we would be short when compared to the amount of data in our production environments, sometimes by up to 10GB.

 

-          You talk about finding that missing files were found after the servers were turned over to your DBA’s. Do you mean missing files showed up without doing a restore? If so what file systems were you using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?

 

I didn’t word this very well.  While it was obvious data was missing if we were GB’s short, some files or binaries needed by our DBA’s for RMAN recoveries etc.. were not seen to be missing until they were unable to do what they needed to.  The files didn’t show up on their own, their absence was identified after the data restores had apparently completed successfully.  This was almost exclusively an issue with data being restored to unix boxes into both VxFS and ZFS file systems on SAN attached disk

 

-          So are you talking about a failure of Netbackup or server/filesystems or both?

 

I don’t think we’re talking of a failure of the filesystems and NetBackup didn’t fail completely.  Once a file or directory was identified as missing after a restore, we would always drill down in Netbackup to make sure that the file was actually there to be restored.  They always were and were which means that NetBackup is working from a backup point of view.  We were then able to successfully restore these specific files or directories if we drilled down to the specific file or sub directory.  In that respect, NBU didn’t fail from a restore point of view either because we were able to restore everything we wanted to, just not always the first time around !! 

 

It is indeed an interesting problem and thankfully was one that we were able to work around.  I haven’t raised this question with veritas yet but plan to do so if some more testing in our home data center generates similar results.

 

 

 

Hello Mark,

 

I have a few questions about your report.

 

-          When you say that all your backup data resides on 2 Data Domain storage devices, I assume you mean just the backup media , tape or disk pool, and not the disk unit holding the restored data?

-          Did Netbackup report a failure to restore files? If so what reason was it reporting.

-          You talk about finding that missing files were found after the servers were turned over to your DBA’s. Do you mean missing files showed up without doing a restore? If so what file systems were you using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?

-          So are you talking about a failure of Netbackup or server/filesystems or both?

 

A very interesting problem.

 

Thanks len

 

 

 

 

I’ve just got back to the office after a 24 DR test and wanted to get this question out there before I forget.

 

We found during our testing that even if an entire directory tree was selected to be restored, after the restore finished we would be missing random files and directories.  Re-running the restore with the same parameters would almost always lay down additional data which should have been restored the first time around. 

 

Once we turned these servers over to our DBA’s to run their RMAN restores, we spent several hours completing additional requests for individual files to be restored which had been missed earlier on.  When we would drill down to check that these files existed we always found that they were there.

 

For reference, all of our backup data resides on 2 Data Domain storage devices so it isn’t an issue with missing media.

 

Has anyone else seen anything like this ?

 

Thanks,

 

Mark Glazerman

Enterprise Storage Administrator

Spartech Corporation

Desk: 314-889-8282

Fax: 314-854-8282

Cell: 618-520-3401

mark.glazerman AT spartech DOT com

http://www.spartech.com

P please don't print this e-mail unless you really need to

This e-mail and any files transmitted with it are confidential, are intended solely for the use of the addressee, and may be legally privileged. If you have received this email in error please notify the sender immediately, and do not copy or forward it.