Veritas-bu

Re: [Veritas-bu] Catalog Backups not running

2011-01-17 17:39:08
Subject: Re: [Veritas-bu] Catalog Backups not running
From: "Mark Glazerman" <Mark.Glazerman AT spartech DOT com>
To: "Nate Sanders" <sandersn AT dmotorworks DOT com>
Date: Mon, 17 Jan 2011 16:39:01 -0600
Nate,

I had the exact same problem and support identified a knowledge base article 
that points to nbpem getting into a looped state which causes policies to be 
marked as corrupt.  In your problems report do you see anything like this...

5/16/2008            9:00:07 PM          unknown             
nbumaster01.sun.spartech.com               Error      1571       Backup backup 
of client nbumaster01.sun.spartech.com exited with status 2 (none of the 
requested files were backed up)
5/16/2008            9:33:43 PM          unknown             
nbumaster01.sun.spartech.com               Error      1617       Backup backup 
of client nbumaster01.sun.spartech.com exited with status 150 (termination 
requested by administrator)
5/16/2008            9:33:43 PM          unknown             
nbumaster01.sun.spartech.com               Error      1617       Backup [1] 
suspending further backup attempts for client nbumaster01.sun.spartech.com, 
policy NBU-Catalog-Hot, schedule Cumulative_Day: exceeded configured number of 
tries
5/17/2008            8:27:38 AM         unknown             
nbumaster01.sun.spartech.com               Error      2060       Backup backup 
of client nbumaster01.sun.spartech.com exited with status 227 (no entity was 
found)
5/17/2008            8:29:30 AM         unknown             
nbumaster01.sun.spartech.com               Error      2063       Backup backup 
of client nbumaster01.sun.spartech.com exited with status 227 (no entity was 
found)
5/17/2008            8:29:30 AM         unknown             
nbumaster01.sun.spartech.com               Error      2063       Backup [1] 
suspending further backup attempts for client nbumaster01.sun.spartech.com, 
policy NBU_CATALOG_HOT1, schedule Cumulative-Night: exceeded configured number 
of tries

Supports workaround for me was to completely delete the current failing catalog 
policy, restart the NBU services and recreate a new policy with a different 
name.  This fixed the issue in 6.5.1 and we then moved to 6.5.4 and didn't see 
the problem again.  We're now at 7.0 and haven't seen anything similar in this 
release.

The errors that I saw during a failing catalog backup can be seen below where 
the db verify failed caused the whole catalog backup to fail.

5/16/2008 9:00:48 PM - Info bpdbm(pid=29892) staging relational database files 
for catalog backup      
5/16/2008 9:00:48 PM - Info bpdbm(pid=29892) staging NBDB backup to 
/usr/openv/db/staging        
5/16/2008 9:00:52 PM - Info bpdbm(pid=29892) done staging NBDB backup to 
/usr/openv/db/staging       
5/16/2008 9:00:52 PM - Info bpdbm(pid=29892) validating NBDB backup in 
/usr/openv/db/staging        
5/16/2008 9:00:53 PM - Error bpdbm(pid=29892) error validating NBDB backup in 
/usr/openv/db/staging       
none of the requested files were backed up(2)

Hope this helps,

Mark Glazerman
Desk: 314-889-8282
Cell: 618-520-3401
 please don't print this e-mail unless you really need to

-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu 
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Nate 
Sanders
Sent: Monday, January 17, 2011 3:24 PM
Cc: VERITAS-BU AT MAILMAN.ENG.AUBURN DOT EDU
Subject: Re: [Veritas-bu] Catalog Backups not running

Policy settings:

Type: NBU-Catalog
Destination:
Policy storage: Any_available
Policy volume pool: CatalogBackup (there are two tapes in this pool)

Schedule (name: Differential-Inc)
Type: Differential Incremental Backup
Schedule Type: (*) After each backup session
Retention: 2 weeks

It used to be I'd see about 100 diff-inc backup jobs a day.


On 01/17/2011 03:19 PM, Nate Sanders wrote:
> I ran -updatepolicies and then checked upcoming schedules
>
>> root@backup1:~$ nbpemreq -predict -date 01/17/2011 20:00:00
>> Predicted work assuming no job run between now and Mon 17 Jan 2011
>> 08:00:00 PM CST
>> rl client               policy               schedule             type
>> -- -------------------- -------------------- --------------------
>> ----------
>> 03 stage-imagedb01      RMAN_stage_simg01    Archive_Logs        
>> 1         
>> 03 netapp02             NDMP_autoload_subaru_ca Diff_Incr           
>> 1         
>> 03 stage-cleandb01      RMAN_stage_scln01    Archive_Logs        
>> 1         
>> 03 watson               RMAN_prod_dbaweb     Archive_Logs        
>> 1         
>> 03 bzsearch02           RMAN_prod_pbz02      Archive_Logs        
>> 1         
>> 04 netapp01             NDMP_netapp1vmvol1   Full_Monthly        
>> 0         
>> 03 gmcert01             RMAN_prod_pgmc01     Archive_Logs        
>> 1         
>> 03 vplayer01            RMAN_prod_pvid01     Archive_Logs        
>> 1         
>> 03 gmcert02             RMAN_prod_pgmc02     Archive_Logs        
>> 1         
>> 03 vplayer02            RMAN_prod_pvid02     Archive_Logs        
>> 1         
>> 03 stage-bzsearch01     RMAN_stage_sbz01     Archive_Logs        
>> 1         
>> 03 stage-imagedb02      RMAN_stage_simg02    Archive_Logs        
>> 1         
>> 03 stage-vplayer01      RMAN_stage_svid01    Archive_Logs        
>> 1         
>> 03 bzsearch01           RMAN_prod_pbz01      Archive_Logs        
>> 1         
> Setting it out past 7 days does show the Full is scheduled to run (which
> is once a week), but should the Incrementals which are scheduled for
> "after each session" show up?
>
>
> On 01/17/2011 03:07 PM, Martin, Jonathan wrote:
>> Have you tried "nbpemreq -updatepolicies"?
>>
>> What happens if you run "nbpemreq -predict -date <mm/dd/yyyy HH:MM:SS>"
>> using the time of the next full or incr to run?
>>
>> -Jonathan
>>
>> -----Original Message-----
>> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
>> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Nate
>> Sanders
>> Sent: Monday, January 17, 2011 3:55 PM
>> To: 'VERITAS-BU AT MAILMAN.ENG.AUBURN DOT EDU'
>> Subject: [Veritas-bu] Catalog Backups not running
>>
>> We just migrated from old hardware to new hardware, maintaining version
>> 6.5.6 on RHEL (moved from RHEL 4.x to 5.x). We did a DR recovery to the
>> new host and had no problems. But now on the new hardware my hot Catalog
>> policy is no longer running incrementals after each session. The policy
>> hasn't changed and I verified everything is still configured
>> appropriately. A manual Incr or Full works fine, but the automated ones
>> aren't running.
>>
>> Thoughts?
>>

-- 
Nate Sanders            Digital Motorworks
System Administrator      (512) 692 - 1038




This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please notify us immediately by e-mail and delete the message and any 
attachments from your system.
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu