Re: [Veritas-bu] "Successful" NDMP Backups Missing Data

Thanks everyone for the comments. Here's some more information:

Client: Sun StorageTek 5370 (nbnfs)

Master/Media Server: Windows 2003 STD SP2 running Netbackup Enterprise 6.0 MP5 (nbbk01)

nbnfs 12/14/2007 8:00:00 PM PR_NDMP 46865896 772513720 Full-Weekly Full 21:49:02 No
nbnfs 12/21/2007 8:00:00 PM PR_NDMP 47584824 796582733 Full-Weekly Full 21:55:57 No
nbnfs 12/28/2007 8:00:00 PM PR_NDMP 47683917 805472963 Full-Monthly Full 22:59:43 No
nbnfs 1/8/2008 9:06:28 AM PR_NDMP 33581624 536189959 Full-Weekly Full 13:56:53 No
nbnfs 1/11/2008 8:00:00 PM PR_NDMP 48033174 810362561 Full-Weekly Full 22:28:07 No
nbnfs 1/18/2008 8:00:00 PM PR_NDMP 37250565 611787136 Full-Weekly Full 17:26:27 No
nbnfs 1/25/2008 8:00:00 PM PR_NDMP 48885260 833748549 Full-Monthly Full 23:03:54 No
nbnfs 2/1/2008 8:00:00 PM PR_NDMP 49524885 838532189 Full-Weekly Full 23:47:23 No
nbnfs 2/8/2008 8:00:00 PM PR_NDMP 49908417 848834139 Full-Weekly Full 23:07:55 No
nbnfs 2/15/2008 8:00:00 PM PR_NDMP 50243186 855850214 Full-Weekly Full 1 00:01:05 No
nbnfs 2/22/2008 8:00:00 PM PR_NDMP 50511980 863140542 Full-Weekly Full 1 00:46:31 No
nbnfs 3/3/2008 12:10:12 PM PR_NDMP 19450344 316300098 Full-Monthly Full 07:01:54 No

nbnfs 3/7/2008 8:00:00 PM PR_NDMP 47423669 775305192 Full-Weekly Full 20:16:58 No

If you notice, all of the red entries are less than they should be. The full backup qty gets incrementally bigger each week with the exception of the "red" failures. The 316GB backup was a failed robotic controller so I expected that but the other three backups were status 0 in Netbackup. I've got three sites all simmilarly configured and two sites (this one and another) are showing this same issue. One intresting point about this particular host is that the same directories are missing from all three "successful" failed backups. We've got a ticket open with Sun, as Netbackup seems to think its a filer issue. (We agree.) We've had issues in the past with super long directory paths on this NAS device, maybe this is simmilar.

Personally, I'm appauled that there seems to be no better communication between the filer and Netbackup about the backup status and that NBU just assumes everything's 100% hunky dory. We use the activty monitor and BPDBJOBS in a NUMBER of scripts to monitor completion. I think its unreasonable to assume the storage guys need to check their filer logs every week to make sure their backups are complete. This may be an excuse for me to write the script I've been planning that measures job success on how much data was backed up (compared to the trend.) For this weekend we're going to backup the data via NFS mount on another server. Stoopid NAS.

I love this job... /sigh

-Jonathan

PS: Should have bought a NetApp =P

From: Jeff Lightner [mailto:jlightner AT water DOT com]
Sent: Thursday, March 13, 2008 5:10 PM
To: Martin, Jonathan; veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] "Successful" NDMP Backups Missing Data

You might want to verify your policy “selections” included everything and also that you don’t have an exclude list specification on the client that would have made it skip certain things.

It would be very bad indeed if NBU gave a status 0 for backups that didn’t backup exactly what they were told to backup. My first suspicion would be that this didn’t occur – that either its not in the selections for the policy or it was excluded due an exclude list.

Of course there are other things that could cause this – e.g. do not cross mount points, don’t backup NFS etc… that might have been set on the policy by default.

From: veritas-bu-bounces AT mailman.eng.auburn DOT edu [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Martin, Jonathan
Sent: Thursday, March 13, 2008 4:11 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: [Veritas-bu] "Successful" NDMP Backups Missing Data

I'm looking at an Full NDMP backup from this past weekend. According to the activity monitor it was Status - 0 Successful. However, when drilling down into the data via Backup and Recover I notice that certain key folders are missing. Upon further inspection, I notice the past 4 weeks backups look something like this - 833GB, 833.2GB, 833.7GB, 780GB. Although Netbackup gives me the "0" I'm obviously missing data. I just opened a case with Symantec (they will call me tomorrow), and I'm pretty sure the data is gone but can anyone give me any hints on where to start troubleshooting this? The good news is that it looks like MOST of the data made it to tape. The VERY BAD news is that 100% of the data was not written to tape and that Netbackup did not give me an error.

-Jonathan

----------------------------------
CONFIDENTIALITY NOTICE: This e-mail
may contain privileged or confidential information and is for the sole use of
the intended recipient(s). If you are not the intended recipient, any
disclosure, copying, distribution, or use of the contents of this information is
prohibited and may be unlawful. If you have received this electronic
transmission in error, please reply immediately to the sender that you have
received the message in error, and delete it. Thank
you.
----------------------------------

_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu