Veritas-bu

Re: [Veritas-bu] NBU 6.5.1 restores miss files

2009-01-20 09:17:02
Subject: Re: [Veritas-bu] NBU 6.5.1 restores miss files
From: Travis Kelley <travis.kelley AT etrade DOT com>
To: Mark Glazerman <Mark.Glazerman AT spartech DOT com>
Date: Tue, 20 Jan 2009 09:03:47 -0500
You didn't mention how your performed the restore.  Bob already
suggested using the command line.  In case you are using the java admin
gui I'll point you over to a technote showing a problem we had where
files weren't restored.  I believe this only applies if you are using
the java admin gui.

http://seer.entsupport.symantec.com/docs/313460.htm


Mark Glazerman wrote:
> Len,
> 
>  
> 
> Thanks for your interest in our issue…
> 
>  
> 
> -          When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
>  
> 
> All backup data created by netbackup is sent straight to one of 2
> DataDomain restorers in our home data center.  That data is then
> replicated to identical DataDomain appliances in our DR hosting site in
> Philadelphia.  All of the images that we restore during either a DR
> exercise or a regular recovery of some kind come from the DataDomain
> appliances.  The DataDomain appliances are not used for primary storage
> or as target for any restored data.  All data was being restored to SAN
> attached disk on each host.
> 
>  
> 
> -          Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
>  
> 
> NetBackup did not report any errors during the restores.  Our signal
> that files / data was missing was that after restoring what we believed
> to be entire mountpoints, we would be short when compared to the amount
> of data in our production environments, sometimes by up to 10GB.
> 
>  
> 
> -          You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
>  
> 
> I didn’t word this very well.  While it was obvious data was missing if
> we were GB’s short, some files or binaries needed by our DBA’s for RMAN
> recoveries etc.. were not seen to be missing until they were unable to
> do what they needed to.  The files didn’t show up on their own, their
> absence was identified after the data restores had apparently completed
> successfully.  This was almost exclusively an issue with data being
> restored to unix boxes into both VxFS and ZFS file systems on SAN
> attached disk
> 
>  
> 
> -          So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>  
> 
> I don’t think we’re talking of a failure of the filesystems and
> NetBackup didn’t fail completely.  Once a file or directory was
> identified as missing after a restore, we would always drill down in
> Netbackup to make sure that the file was actually there to be restored. 
> They always were and were which means that NetBackup is working from a
> backup point of view.  We were then able to successfully restore these
> specific files or directories if we drilled down to the specific file or
> sub directory.  In that respect, NBU didn’t fail from a restore point of
> view either because we were able to restore everything we wanted to,
> just not always the first time around !! 
> 
>  
> 
> It is indeed an interesting problem and thankfully was one that we were
> able to work around.  I haven’t raised this question with veritas yet
> but plan to do so if some more testing in our home data center generates
> similar results.
> 
>  
> 
> *Mark Glazerman*
> 
> Desk: 314-889-8282
> 
> Cell: 618-520-3401
> 
> P please don't print this e-mail unless you really need to
> 
>  
> 
> *From:* Len Boyle [mailto:Len.Boyle AT sas DOT com]
> *Sent:* Monday, January 19, 2009 2:47 PM
> *To:* Mark Glazerman; Veritas-bu AT mailman.eng.auburn DOT edu
> *Subject:* RE: NBU 6.5.1 restores miss files
> 
>  
> 
> Hello Mark,
> 
>  
> 
> I have a few questions about your report.
> 
>  
> 
> -          When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
> -          Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
> -          You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
> -          So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>  
> 
> A very interesting problem.
> 
>  
> 
> Thanks len
> 
>  
> 
>  
> 
>  
> 
> *From:* veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] *On Behalf Of *Mark
> Glazerman
> *Sent:* Monday, January 19, 2009 3:03 PM
> *To:* Veritas-bu AT mailman.eng.auburn DOT edu
> *Subject:* [Veritas-bu] NBU 6.5.1 restores miss files
> 
>  
> 
> I’ve just got back to the office after a 24 DR test and wanted to get
> this question out there before I forget.
> 
>  
> 
> We found during our testing that even if an entire directory tree was
> selected to be restored, after the restore finished we would be missing
> random files and directories.  Re-running the restore with the same
> parameters would almost always lay down additional data which should
> have been restored the first time around. 
> 
>  
> 
> Once we turned these servers over to our DBA’s to run their RMAN
> restores, we spent several hours completing additional requests for
> individual files to be restored which had been missed earlier on.  When
> we would drill down to check that these files existed we always found
> that they were there.
> 
>  
> 
> For reference, all of our backup data resides on 2 Data Domain storage
> devices so it isn’t an issue with missing media.
> 
>  
> 
> Has anyone else seen anything like this ?
> 
>  
> 
> Thanks,
> 
>  
> 
> *Mark Glazerman*
> 
> *Enterprise Storage Administrator*
> 
> *Spartech Corporation*
> 
> Desk: 314-889-8282
> 
> Fax: 314-854-8282
> 
> Cell: 618-520-3401
> 
> *mark.glazerman AT spartech DOT com* <mailto:mark.glazerman AT spartech DOT 
> com>
> 
> *http://www.spartech.com* <http://www.spartech.com/>
> 
> P please don't print this e-mail unless you really need to
> 
> This e-mail and any files transmitted with it are confidential, are
> intended solely for the use of the addressee, and may be legally
> privileged. If you have received this email in error please notify the
> sender immediately, and do not copy or forward it.
> 
>  
> 

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu