SAP backup failing

adminprem

Active Newcomer
Joined
Aug 13, 2015
Messages
9
Reaction score
0
Points
0
PREDATAR Control23

Hi Experts,

I am looking for a help on below issue where my all SAP db backup started failing all of a sudden and taking more time to even completing and throwing error. below is the settings i am using.

RMAN Channel -4

Error snapshot
Recovery Manager complete.
BR0280I BRBACKUP time stamp: 2019-01-22 12.32.09
BR0279E Return code from 'SHELL=/bin/sh /oracle/PB1/112_64/bin/rman nocatalog': 1
BR0522I 178 of 178 files / save sets processed by RMAN
BR0536E RMAN call for database instance PB1 failed
BR0200I BR_TRACE: location BrRmanCall-56, commands for RMAN in: /oracle/PB1/sapbackup/.bfahsjqi.cmd
@/oracle/PB1/sapbackup/..bfahsjqi..cmd
host '/usr/sap/PB1/SYS/exe/run/brtools -f delete /oracle/PB1/sapbackup/..bfahsjqi..cmd';
run { allocate channel sbt_1 device type 'SBT_TAPE'
parms 'SBT_LIBRARY=/oracle/PB1/112_64/lib/libobk.sl,ENV=(XINT_PROFILE=/oracle/PB1/112_64/dbs/initPB1.utl,PROLE_PORT=57323,BR_CALLER=BRBACKUP,BR_BACKUP=FULL,BR_REQUEST=NEW,BR_RUN=bfahsjqi.fnr)';
allocate channel sbt_2 device type 'SBT_TAPE'
parms 'SBT_LIBRARY=/oracle/PB1/112_64/lib/libobk.sl,ENV=(XINT_PROFILE=/oracle/PB1/112_64/dbs/initPB1.utl,PROLE_PORT=57323,BR_CALLER=BRBACKUP,BR_BACKUP=FULL,BR_REQUEST=NEW,BR_RUN=bfahsjqi.fnr)';
backup incremental level 0 tag bfahsjqi format 'PB1_bfahsjqi.%s_%p' filesperset 1 check logical
database;

surprisingly when i am changing settings on sid.sap file from backup_dev_type = rman_util to util_file all the backup is completing successfully, i have no clue when i am changing setting to use backup_dev_type = rman_util it's failing.

Summary- Less than 6 hours taking to complete the backup through backup_dev_type = util_file.
Summary- more than 18 hours taking to even throwing error and backup is not getting completed and throwing error which i reported above please advise what can be done to solve this issue.

TSM Version. 6.3.4.
SAP-11.2

Thanks
Prem
 
PREDATAR Control23

Hi,

Eventlogs, errorlogs and config files may give us a clue to figure out where the error may be.

Recommend that you open a ticket with SAP/Oracle/IBM to resolve the issue.



-= Trident =-
 
PREDATAR Control23

Hi,

Eventlogs, errorlogs and config files may give us a clue to figure out where the error may be.

Recommend that you open a ticket with SAP/Oracle/IBM to resolve the issue.



-= Trident =-
Hi ,

I am pretty sure this is a configuration issue only, why because server is same, database is same, media is same, when I am using through util file all is normal, database and archive log backup functioning, at the other side when I use rman_util in sid.sap file it is started failing.
 
PREDATAR Control23

Hey Prem,

When you use backup_dev_type = rman_util you will also need to configure some RMAN parameters.

go to rman and show all

bg0adm> rman target /
Recovery Manager: Release 12.2.0.1.0 - Production on Thu Feb 14 17:19:11 2019
Copyright (c) 1982, 2017, Oracle and/or its affiliates. All rights reserved.
connected to target database: BG0 (DBID=823113272)
RMAN> show all;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name BG0 are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP ON; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/api/bin64/dsm_BG0.opt)';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/oracle/BG0/122/dbs/snapcf_BG0.f'; # default
RMAN> exit


You probably need to configure some SBT_TAPE parameters. Give that a try and run again using rman_util.

I have also found that when you configure CHANNEL DEVICE TYPE the other SBT_TAPE parameters automatically get added. Let us know.

Regards
 
PREDATAR Control23

Hi,

Thanks for the detailed information I am sharing 2 SID details wherein from one SID HP1 my backup are functioning perfectly from RMAN and other one is for PB1 where backups are failing but I could not see any major change in both in their settings.

rman target /
Recovery Manager: Release 12.1.0.2.0 - Production on Fri Feb 15 20:24:49 2019
Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.
connected to target database: HP1 (DBID=3621477099)
RMAN> show all ;
using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name HP1 are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOA D TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/oracle/HP1/12102/dbs/snapcf_HP1.f'; # default

rman target /

Recovery Manager: Release 11.2.0.4.0 - Production on Fri Feb 15 19:18:53 2019

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: PB1 (DBID=3888320353)

RMAN> show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name PB1 are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/oracle/PB1/112_64/dbs/snapcf_PB1.f'; # default

Please advise, Thanks

Prem
 
PREDATAR Control23

Prem,

Set these parms in your utilPB1.sap file:

> /oracle/PB1/112_64/dbs/utilPB1.sap

backup_dev_type = rman_util
tape_copy_cmd = rman

rman_parms="ENV=(XINT_PROFILE=/oracle/PB1/112_64/dbs/initPB1.utl,PROLE_PORT=57323,&BR_INFO)"

Double check your rman parameters. I don't see any SBT_TAPE configuration parameters in your above "show all;" lists. You should have these set in RMAN in addition to the normal ones for DISK:

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE DEVICE TYPE 'SBT_TAPE' PARALLELISM 1 BACKUP TYPE TO BACKUPSET;
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default

CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default

The below parameter is specific to the IBM Tivoli tape system:
CONFIGURE CHANNEL DEVICE TYPE 'SBT_TAPE' PARMS 'ENV=(TDPO_OPTFILE=/opt/tivoli/tsm/client/api/bin64/dsm_PB1.opt)';

The dsm_PB1.opt does not need to exist. The system will create it. If you are not using Tivoli, then ignore this parameter.

I would also compare your /oracle/PB1/112_64/dbs/utilPB1.sap against your /oracle/HP1 /112_64/dbs/utilHP1.sap. Also compare utilPB1.utl with utilHP1.utl and again compare rman parms between the 2 systems.

Also check your permissions and ownerships of your BRTOOLS. /usr/sap/PB1/SYS/exe/run directory. Refer to note 776505 - ORA-01017/ORA-01031 in BR*Tools on Linux and Solaris 11. If your oracle binaries are owned by pb1adm your brtools executable's should look like this:
-rwsr-sr-- 1 orapb1 dba 9325374 2012-12-19 04:52 brarchive
-rwsr-sr-- 1 orapb1 dba 9443223 2012-12-19 04:52 brbackup
-rwsr-sr-- 1 orapb1 dba 11333756 2012-12-19 04:55 brconnect
-rwxr-xr-x 1 pb1adm sapsys 9893702 2012-12-19 04:56 brrecover
-rwxr-xr-x 1 pb1adm sapsys 6203096 2012-12-19 04:56 brrestore
-rwxr-xr-x 1 pb1adm sapsys 11766827 2012-12-19 04:58 brspace
-rwxr-xr-x 1 pb1adm sapsys 6797524 2012-12-19 04:58 brtools


su - root
cd /usr/sap/PB1/SYS/exe/run
chown orapb1:dba brarchive brbackup brconnect
chmod 6754 brarchive brbackup brconnect


This is a lot of changes. You could cause other issues with all of these changes. Take a couple at a time and see what happens. There are lots of pieces to getting brtools to work properly.

Regards
 
PREDATAR Control23

Hi,

Thanks for the detailed analysis I am sorry for replying late caught up in other activities,

It is 2 node cluster now we have moved into primary node now at the last it reported different error this time.

BR0278E Command output of '/usr/sbin/cmviewcl':
Permission denied to 127.0.0.1
cmviewcl: Cannot view the cluster configuration: Permission denied.
This user doesn't have access to view the cluster configuration.
BR0280I BRBACKUP time stamp: 2019-02-23 02.37.06
BR0279E Return code from '/usr/sbin/cmviewcl': 1
BR0299E Split/resynchronization of mirror disks failed
 
PREDATAR Control23

Prem,

Again I would double check SAP note
776505 for the proper permissions on the brtools executable's. There are some specific parameters related to RAC systems in init<SID>.sap file located in /oracle/SID/112_64/dbs. Also check your environment variables for <SID>adm user to make sure your $ORACLE_HOME, $ORACLE_BASE, $ORACLE_SID, $SAPDATA_HOME, etc... are all set correctly. You are running your brtools while logged in as <SID>adm user?

Regards
 
PREDATAR Control23

I am running brtools as a orasid, frankly speaking I will not lie I am not convinced with permission issue/theory if that applies then through util files also it should report same error that's my opinion and what is one difference I notice backup package is active on secondary node I don't know why And we are running backup from primary node, I am not sure that could be one of case for that error.
 
Top