TDPO failed with I/O Error

zaturday

ADSM.ORG Member
Joined
Jun 5, 2014
Messages
18
Reaction score
0
Points
0
PREDATAR Control23

please help for investigate error
I has still problem from backup RMAN online to tape. when error i found about "ANR8302E I/O error on drive XXXX" so the problem not happen every times

from msg error on RMAN
Code:
ORA-19502: write error on file "backup_database_626_gcpprd_inc1_daily_14_20150516_7373_1.bck", block number 2662721 (block size=8192)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
   ANS1315W (RC15)   Unexpected retry request. The server found an error while writing the data.

Log volume and actlog for this session failed
Code:
tsm: RATTSM01>q vol stg=dd_gcp*
GCP001,DD_GCP_ORA,DD_GCP_DEV,819.2 G,1.3,Filling
GCP002,DD_GCP_ORA,DD_GCP_DEV,819.2 G,0.2,Filling

tsm: RATTSM01>q content gcp001
RPRDAGCPDB01_ORA,Bkup,/rprdagcpdb01_ora,1,//archivelog_626_gcpprd_inc1_daily_14_20150515_7366_1.arc
RPRDAGCPDB01_ORA,Bkup,/rprdagcpdb01_ora,1,//controlfile_626_gcpprd_inc1_daily_14_20150515_7367_1.ctl
RPRDAGCPDB01_ORA,Bkup,/rprdagcpdb01_ora,1,//spfile_626_gcpprd_inc1_daily_14_GCPPRD_7368_20150515.bck
RPRDAGCPDB01_ORA,Bkup,/rprdagcpdb01_ora,1,//backup_database_626_gcpprd_inc1_daily_14_20150515_7369_1.bck

tsm: RATTSM01>q content gcp002
RPRDAGCPDB01_ORA,Bkup,/rprdagcpdb01_ora,1,//backup_database_626_gcpprd_inc1_daily_14_20150516_7372_1.bck

tsm: RATTSM01>q actlog search="SESSION: 864163" begind=-1                                          
05/16/15   02:55:07,ANR0406I Session 864163 started for node RPRDAGCPDB01_ORA (TDP Oracle AIX) (Tcp/Ip 192.168.238.86(47910)). (SESSION: 864163)
05/16/15   02:55:10,ANR8337I LTO volume GCP002 mounted in drive GCP_VTL102 (/dev/mt102). (SESSION: 864163)
05/16/15   02:55:10,ANR0511I Session 864163 opened output volume GCP002. (SESSION: 864163)
05/16/15   04:23:36,"ANE4991I (ANE4985I Session: 864163, ANE4986I Node: RPRDAGCPDB01_ORA)  TDP Oracle AIX ANU0599 TDP for Oracle: (2887868): =>() ANU2535I File /rprdagcpdb01_ora//backup_database_626_gcpprd_inc1_daily_14_20150516_7372_1.bck = 1936982016 bytes sent (SESSION: 864163)"
05/16/15   04:23:36,"ANE4991I (ANE4985I Session: 864163, ANE4986I Node: RPRDAGCPDB01_ORA)  TDP Oracle AIX ANU0599 TDP for Oracle: (2887868): =>() ANU2526I Backup details for backup piece /rprdagcpdb01_ora//backup_database_626_gcpprd_inc1_daily_14_20150516_7372_1.bck (database "gcpprd"). Total bytes processed: 1936982016. Deduplicated: No. Bytes after deduplication: 1936982016. Deduplication reduction: 0.00%. Compressed: No. Bytes after compression: 1936982016. Compressed by: 0.00%. Encryption: None. LAN-Free: No. Total bytes sent: 1936982016. Total data reduction: 0.00%. Total processing time: 01:28:29. Throughput rate: 356.30 Kb/Sec. (SESSION: 864163)"
05/16/15   04:23:36,ANR0514I Session 864163 closed volume GCP002. (SESSION: 864163)
05/16/15   04:23:36,ANR0403I Session 864163 ended for node RPRDAGCPDB01_ORA (TDP Oracle AIX). (SESSION: 864163)
05/16/15   05:09:32,"ANR8302E I/O error on drive GCP_VTL102 (/dev/mt102) with volume GCP002 (OP=WRITE, Error Number=5, CC=206, rc = 1, KEY= FF, ASC=FF, ASCQ= FF, SENSE=**NONE**, Description=General SCSI failure). Refer to the &STGM; documentation on I/O error code descriptions.   (SESSION: 864163)"
05/16/15   05:09:32,ANR8468I LTO volume GCP002 dismounted from drive GCP_VTL102 (/dev/mt102) in library DD_GCP_DEV. (SESSION: 864163)
05/16/15   07:54:34,ANR2017I Administrator MF01573 issued command: QUERY ACTLOG search='SESSION: 864163' begind=-1  (SESSION: 869979)

full log RMAN
Code:
Recovery Manager: Release 11.2.0.1.0 - Production on Sat May 16 02:59:11 2015

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

connected to target database: GCPPRD (DBID=688785118)

RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 
using target database control file instead of recovery catalog
allocated channel: ch1
channel ch1: SID=3965 instance=gcpprd1 device type=SBT_TAPE
channel ch1: Data Protection for Oracle: version 6.3.0.0

Starting backup at 16-MAY-2015 02:59:20
channel ch1: starting incremental level 1 datafile backup set
channel ch1: specifying datafile(s) in backup set
input datafile file number=00014 name=/oradata/gcpprd/GCPPS_USRDEF_FT1.dbf
input datafile file number=00031 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_03.dbf
input datafile file number=00015 name=/oradata/gcpprd/GCPPS_TB_FT1.dbf
input datafile file number=00027 name=/oradata/gcpprd/GCPPS_TB_FT1_02.dbf
input datafile file number=00029 name=/oradata/gcpprd/GCPPS_TB_FT1_04.dbf
input datafile file number=00039 name=/oradata/gcpprd/GCPPS_TB_FT1_05.dbf
input datafile file number=00041 name=/oradata/gcpprd/GCPPS_IND_FT1_05.dbf
input datafile file number=00047 name=/oradata/gcpprd/GCPPS_TB_FT1_08.dbf
input datafile file number=00051 name=/oradata/gcpprd/GCPPS_TB_FT1_10.dbf
input datafile file number=00054 name=/oradata/gcpprd/GCPPS_TB_FT1_13.dbf
input datafile file number=00067 name=/oradata/gcpprd/GCPPS_TB_FT1_17.dbf
input datafile file number=00078 name=/oradata/gcpprd/GCPPS_TB_FT1_19.dbf
input datafile file number=00080 name=/oradata/gcpprd/GCPPS_TB_FT1_21.dbf
input datafile file number=00088 name=/oradata/gcpprd/GCPPS_TB_FT1_23.dbf
input datafile file number=00042 name=/oradata/gcpprd/GCPPS_IND_FT1_06.dbf
input datafile file number=00060 name=/oradata/gcpprd/GCPPS_TB_FT1_15.dbf
input datafile file number=00038 name=/oradata/gcpprd/GCPPS_IND_FT1_04.dbf
input datafile file number=00056 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_06.dbf
input datafile file number=00001 name=/oradata/gcpprd/system01.dbf
input datafile file number=00004 name=/oradata/gcpprd/system02.dbf
input datafile file number=00045 name=/oradata/gcpprd/GCPPS_IND_FT1_09.dbf
input datafile file number=00089 name=/oradata/gcpprd/GCPPS_TB_FT1_24.dbf
input datafile file number=00086 name=/oradata/gcpprd/undotbs03.dbf
input datafile file number=00020 name=/oradata/gcpprd/GCPPS_BALI1.dbf
input datafile file number=00062 name=/oradata/gcpprd/GCPPS_AFL_TBS.dbf
input datafile file number=00092 name=/oradata/gcpprd/BILL_PAYMENT_ACTIVITY_WEEK_RA02.dbf
input datafile file number=00061 name=/oradata/gcpprd/GCPPS_IND_FT1_12.dbf
input datafile file number=00069 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_09.dbf
input datafile file number=00071 name=/oradata/gcpprd/GCPPS_IND_FT1_13.dbf
input datafile file number=00075 name=/oradata/gcpprd/GCPPS_IND_FT1_15.dbf
input datafile file number=00082 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_11.dbf
input datafile file number=00084 name=/oradata/gcpprd/GCPPS_IND_FT1_17.dbf
input datafile file number=00093 name=/oradata/gcpprd/BILL_PAYMENT_ACTIVITY_INDEX01.dbf
input datafile file number=00065 name=/oradata/gcpprd/GCPPS_TMPBLCL_FT1_03.dbf
input datafile file number=00073 name=/oradata/gcpprd/GCPPS_CASHVAR1_03.dbf
input datafile file number=00072 name=/oradata/gcpprd/GCPPS_BALI1_03.dbf
input datafile file number=00023 name=/oradata/gcpprd/undotbs2_02.dbf
input datafile file number=00037 name=/oradata/gcpprd/undotbs2_03.dbf
input datafile file number=00005 name=/oradata/gcpprd/undotbs02.dbf
input datafile file number=00043 name=/oradata/gcpprd/GCPPS_IND_FT1_07.dbf
input datafile file number=00102 name=/oradata/gcpprd/system04.dbf
input datafile file number=00006 name=/oradata/gcpprd/GCPPS_USRDEF_FT.dbf
input datafile file number=00017 name=/oradata/gcpprd/GCPPS_MASTER_FT1.dbf
input datafile file number=00026 name=/oradata/gcpprd/GCPPS_CASHVAR1_02.dbf
input datafile file number=00036 name=/oradata/gcpprd/dbiis01.dbf
input datafile file number=00007 name=/oradata/gcpprd/GCPPS_TB_FT.dbf
input datafile file number=00009 name=/oradata/gcpprd/GCPPS_MASTER_FT.dbf
input datafile file number=00011 name=/oradata/gcpprd/GCPPS_CASHVAR.dbf
input datafile file number=00021 name=/oradata/gcpprd/gcpdmuatKBRFAX.dbf
input datafile file number=00100 name=/oradata/gcpprd/GCPPS_TB_FT1_29.dbf
input datafile file number=00013 name=/oradata/gcpprd/xdb01.dbf
input datafile file number=00035 name=/oradata/gcpprd/monitor01.dbf
channel ch1: starting piece 1 at 16-MAY-2015 02:59:23
channel ch1: finished piece 1 at 16-MAY-2015 04:28:10
piece handle=backup_database_626_gcpprd_inc1_daily_14_20150516_7372_1.bck tag=GCPPRD_DAILY_20150516 comment=API Version 2.0,MMS Version 6.3.0.0
channel ch1: backup set complete, elapsed time: 01:28:47
channel ch1: starting incremental level 1 datafile backup set
channel ch1: specifying datafile(s) in backup set
input datafile file number=00030 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_02.dbf
input datafile file number=00032 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_04.dbf
input datafile file number=00016 name=/oradata/gcpprd/GCPPS_IND_FT1.dbf
input datafile file number=00028 name=/oradata/gcpprd/GCPPS_TB_FT1_03.dbf
input datafile file number=00034 name=/oradata/gcpprd/GCPPS_IND_FT1_03.dbf
input datafile file number=00040 name=/oradata/gcpprd/GCPPS_TB_FT1_06.dbf
input datafile file number=00046 name=/oradata/gcpprd/GCPPS_TB_FT1_07.dbf
input datafile file number=00049 name=/oradata/gcpprd/GCPPS_TB_FT1_09.dbf
input datafile file number=00053 name=/oradata/gcpprd/GCPPS_TB_FT1_12.dbf
input datafile file number=00066 name=/oradata/gcpprd/GCPPS_TB_FT1_16.dbf
input datafile file number=00077 name=/oradata/gcpprd/GCPPS_TB_FT1_18.dbf
input datafile file number=00079 name=/oradata/gcpprd/GCPPS_TB_FT1_20.dbf
input datafile file number=00087 name=/oradata/gcpprd/GCPPS_TB_FT1_22.dbf
input datafile file number=00090 name=/oradata/gcpprd/GCPPS_TB_FT1_25.dbf
input datafile file number=00033 name=/oradata/gcpprd/GCPPS_IND_FT1_02.dbf
input datafile file number=00055 name=/oradata/gcpprd/GCPPS_TB_FT1_14.dbf
input datafile file number=00050 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_05.dbf
input datafile file number=00057 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_07.dbf
input datafile file number=00052 name=/oradata/gcpprd/GCPPS_TB_FT1_11.dbf
input datafile file number=00044 name=/oradata/gcpprd/GCPPS_IND_FT1_08.dbf
input datafile file number=00063 name=/oradata/gcpprd/system03.dbf
input datafile file number=00058 name=/oradata/gcpprd/GCPPS_IND_FT1_10.dbf
input datafile file number=00059 name=/oradata/gcpprd/GCPPS_IND_FT1_11.dbf
input datafile file number=00098 name=/oradata/gcpprd/GCPPS_TB_FT1_27.dbf
input datafile file number=00048 name=/oradata/gcpprd/GCPPS_BALI1_02.dbf
input datafile file number=00091 name=/oradata/gcpprd/BILL_PAYMENT_ACTIVITY_WEEK_RA01.dbf
input datafile file number=00018 name=/oradata/gcpprd/GCPPS_TMPBLCL_FT1.dbf
input datafile file number=00068 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_08.dbf
input datafile file number=00070 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_10.dbf
input datafile file number=00074 name=/oradata/gcpprd/GCPPS_IND_FT1_14.dbf
input datafile file number=00076 name=/oradata/gcpprd/GCPPS_IND_FT1_16.dbf
input datafile file number=00083 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_12.dbf
input datafile file number=00085 name=/oradata/gcpprd/GCPPS_USRDEF_FT1_13.dbf
input datafile file number=00094 name=/oradata/gcpprd/BILL_PAYMENT_ACTIVITY_INDEX02.dbf
input datafile file number=00024 name=/oradata/gcpprd/GCPPS_TMPBLCL_FT1_02.dbf
input datafile file number=00095 name=/oradata/gcpprd/IDX_BTR_HISTORY_ACTIVITY_01.dbf
input datafile file number=00096 name=/oradata/gcpprd/IDX_BTR_HISTORY_ACTIVITY_02.dbf
input datafile file number=00022 name=/oradata/gcpprd/undotbs2_01.dbf
input datafile file number=00003 name=/oradata/gcpprd/undotbs01.dbf
input datafile file number=00019 name=/oradata/gcpprd/GCPPS_CASHVAR1.dbf
input datafile file number=00097 name=/oradata/gcpprd/GCPPS_TB_FT1_26.dbf
input datafile file number=00064 name=/oradata/gcpprd/GCPPS_AFL_TBS_02.dbf
input datafile file number=00025 name=/oradata/gcpprd/GCPPS_MASTER_FT1_02.dbf
input datafile file number=00081 name=/oradata/gcpprd/GCPPS_TMPBLCL_FT1_04.dbf
input datafile file number=00002 name=/oradata/gcpprd/sysaux01.dbf
input datafile file number=00008 name=/oradata/gcpprd/GCPPS_IND_FT.dbf
input datafile file number=00010 name=/oradata/gcpprd/GCPPS_TMPBLCL_FT.dbf
input datafile file number=00012 name=/oradata/gcpprd/GCPPS_BALI.dbf
input datafile file number=00099 name=/oradata/gcpprd/GCPPS_TB_FT1_28.dbf
input datafile file number=00101 name=/oradata/gcpprd/GCPPS_TB_FT1_30.dbf
channel ch1: starting piece 1 at 16-MAY-2015 04:28:12
released channel: ch1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on ch1 channel at 05/16/2015 05:14:38
ORA-19502: write error on file "backup_database_626_gcpprd_inc1_daily_14_20150516_7373_1.bck", block number 2662721 (block size=8192)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
   ANS1315W (RC15)   Unexpected retry request. The server found an error while writing the data.

RMAN> 

Recovery Manager complete.

Thank for your help and do you want any information please tell me.

Cheer Zaturday A.
 
PREDATAR Control23

tsm: RATTSM01>q actlog search="SESSION: 864163" begind=-1

FYI:
The "search" parameter is great if you know exactly what you are looking for.
The bad thing about the "search" parameter it will filter out any important message(s) other than what you are "searching" for.

05/16/15 05:09:32,"ANR8302E I/O error on drive GCP_VTL102 (/dev/mt102) with volume GCP002 (OP=WRITE, Error Number=5, CC=206, rc = 1, KEY= FF, ASC=FF, ASCQ= FF, SENSE=**NONE**, Description=General SCSI failure). Refer to the &STGM; documentation on I/O error code descriptions. (SESSION: 864163)"

Since the "search" parameter was used, not sure what other messages leading up to the
ANR8302E. Notice in the ANR8302E message that there are "KEY= FF, ASC=FF, ASCQ= FF".
These parameters should return a numeric value rather than an alpha character.
Historically, there is an issue with either the device driver and/or with the device it self.

Not sure what os that is hosting the TSM Server.
If its AIX, check the errpt to see if there are any errors.

If we are using an IBM tape device, upgrade the ATape.driver to the latest version.
If we are at the latest ATape.driver, need to contact IBM Hardware to have the device driver checked out.

If we are using the TSM device driver, may need to upgrade the TSM Server to the latest maintenance release. To upgrade the TSM device driver, the TSM Server it self also need to be upgraded. If we are already at the latest TSM Server maintenace release, need to contact TSM Support to have the driver checked out.

Good Luck,
Sias
 
Top