Veritas-bu

Re: [Veritas-bu] NBU 6.5.1 restores miss files

2009-01-20 09:32:20
Subject: Re: [Veritas-bu] NBU 6.5.1 restores miss files
From: "Mark Glazerman" <Mark.Glazerman AT spartech DOT com>
To: "Travis Kelley" <travis.kelley AT etrade DOT com>
Date: Tue, 20 Jan 2009 08:15:04 -0600
Travis,

Sorry for missing out that detail.

We initiated all of our restores through the GUI.  I just took a look at the 
tech note you mentioned and was certainly interested at what it talked about.  
This would certainly explain our issue.  

However, I followed the steps to see if there was a similar mismatch in our 
production environment.  I chose a directory with 8 files in it and initiated a 
restore of those files to a different "test" directory on the same server.  All 
8 files were restored.

I don't know if the problem mentioned in the tech note happens with EVERY 
restore or just randomly but this very limited test seems to imply that we 
don't have that problem here.

Thanks,

Mark Glazerman
Desk: 314-889-8282
Cell: 618-520-3401
 please don't print this e-mail unless you really need to


-----Original Message-----
From: Travis Kelley [mailto:travis.kelley AT etrade DOT com] 
Sent: Tuesday, January 20, 2009 8:04 AM
To: Mark Glazerman
Cc: Veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] NBU 6.5.1 restores miss files

You didn't mention how your performed the restore.  Bob already
suggested using the command line.  In case you are using the java admin
gui I'll point you over to a technote showing a problem we had where
files weren't restored.  I believe this only applies if you are using
the java admin gui.

http://seer.entsupport.symantec.com/docs/313460.htm


Mark Glazerman wrote:
> Len,
> 
>  
> 
> Thanks for your interest in our issue…
> 
>  
> 
> -          When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
>  
> 
> All backup data created by netbackup is sent straight to one of 2
> DataDomain restorers in our home data center.  That data is then
> replicated to identical DataDomain appliances in our DR hosting site in
> Philadelphia.  All of the images that we restore during either a DR
> exercise or a regular recovery of some kind come from the DataDomain
> appliances.  The DataDomain appliances are not used for primary storage
> or as target for any restored data.  All data was being restored to SAN
> attached disk on each host.
> 
>  
> 
> -          Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
>  
> 
> NetBackup did not report any errors during the restores.  Our signal
> that files / data was missing was that after restoring what we believed
> to be entire mountpoints, we would be short when compared to the amount
> of data in our production environments, sometimes by up to 10GB.
> 
>  
> 
> -          You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
>  
> 
> I didn’t word this very well.  While it was obvious data was missing if
> we were GB’s short, some files or binaries needed by our DBA’s for RMAN
> recoveries etc.. were not seen to be missing until they were unable to
> do what they needed to.  The files didn’t show up on their own, their
> absence was identified after the data restores had apparently completed
> successfully.  This was almost exclusively an issue with data being
> restored to unix boxes into both VxFS and ZFS file systems on SAN
> attached disk
> 
>  
> 
> -          So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>  
> 
> I don’t think we’re talking of a failure of the filesystems and
> NetBackup didn’t fail completely.  Once a file or directory was
> identified as missing after a restore, we would always drill down in
> Netbackup to make sure that the file was actually there to be restored. 
> They always were and were which means that NetBackup is working from a
> backup point of view.  We were then able to successfully restore these
> specific files or directories if we drilled down to the specific file or
> sub directory.  In that respect, NBU didn’t fail from a restore point of
> view either because we were able to restore everything we wanted to,
> just not always the first time around !! 
> 
>  
> 
> It is indeed an interesting problem and thankfully was one that we were
> able to work around.  I haven’t raised this question with veritas yet
> but plan to do so if some more testing in our home data center generates
> similar results.
> 
>  
> 
> *Mark Glazerman*
> 
> Desk: 314-889-8282
> 
> Cell: 618-520-3401
> 
> P please don't print this e-mail unless you really need to
> 
>  
> 
> *From:* Len Boyle [mailto:Len.Boyle AT sas DOT com]
> *Sent:* Monday, January 19, 2009 2:47 PM
> *To:* Mark Glazerman; Veritas-bu AT mailman.eng.auburn DOT edu
> *Subject:* RE: NBU 6.5.1 restores miss files
> 
>  
> 
> Hello Mark,
> 
>  
> 
> I have a few questions about your report.
> 
>  
> 
> -          When you say that all your backup data resides on 2 Data
> Domain storage devices, I assume you mean just the backup media , tape
> or disk pool, and not the disk unit holding the restored data?
> 
> -          Did Netbackup report a failure to restore files? If so what
> reason was it reporting.
> 
> -          You talk about finding that missing files were found after
> the servers were turned over to your DBA’s. Do you mean missing files
> showed up without doing a restore? If so what file systems were you
> using? Windows 2003 ntfs, solaris zfs, Linux ext3? Local disk or remote?
> 
> -          So are you talking about a failure of Netbackup or
> server/filesystems or both?
> 
>  
> 
> A very interesting problem.
> 
>  
> 
> Thanks len
> 
>  
> 
>  
> 
>  
> 
> *From:* veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] *On Behalf Of *Mark
> Glazerman
> *Sent:* Monday, January 19, 2009 3:03 PM
> *To:* Veritas-bu AT mailman.eng.auburn DOT edu
> *Subject:* [Veritas-bu] NBU 6.5.1 restores miss files
> 
>  
> 
> I’ve just got back to the office after a 24 DR test and wanted to get
> this question out there before I forget.
> 
>  
> 
> We found during our testing that even if an entire directory tree was
> selected to be restored, after the restore finished we would be missing
> random files and directories.  Re-running the restore with the same
> parameters would almost always lay down additional data which should
> have been restored the first time around. 
> 
>  
> 
> Once we turned these servers over to our DBA’s to run their RMAN
> restores, we spent several hours completing additional requests for
> individual files to be restored which had been missed earlier on.  When
> we would drill down to check that these files existed we always found
> that they were there.
> 
>  
> 
> For reference, all of our backup data resides on 2 Data Domain storage
> devices so it isn’t an issue with missing media.
> 
>  
> 
> Has anyone else seen anything like this ?
> 
>  
> 
> Thanks,
> 
>  
> 
> *Mark Glazerman*
> 
> *Enterprise Storage Administrator*
> 
> *Spartech Corporation*
> 
> Desk: 314-889-8282
> 
> Fax: 314-854-8282
> 
> Cell: 618-520-3401
> 
> *mark.glazerman AT spartech DOT com* <mailto:mark.glazerman AT spartech DOT 
> com>
> 
> *http://www.spartech.com* <http://www.spartech.com/>
> 
> P please don't print this e-mail unless you really need to
> 
> This e-mail and any files transmitted with it are confidential, are
> intended solely for the use of the addressee, and may be legally
> privileged. If you have received this email in error please notify the
> sender immediately, and do not copy or forward it.
> 
>  
> 

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu