ADSM-L

Oracle backup problem on AIX - memory leak ?

1998-07-03 05:06:26
Subject: Oracle backup problem on AIX - memory leak ?
From: "Vilmundur Palmason (N}herji)" <villi2 AT ITN DOT IS>
Date: Fri, 3 Jul 1998 09:06:26 +0000
Hi,

I recently installed Oracle ADSMConnect Agent for AIX (try-buy) on an F50
running AIX 4.2.1 and Oracle 7.3.3 (EBU V 2.1.0.1.2)
My obackup currently runs about halfway through until encountering the
error: (from obackup trace file)

EBU-4003: Process pid 78454 exited; exit code = 5
  on 06/30/1998 20:46:55 [ 77424 : brio ]
Process terminated with SIGNAL 4

Signal 4 means illegal instruction which sounds like a bug.  As can be seen
below, the obackup processes seem to be experiencing a memory leak.

Nothing is logged in the dsierror.log file.

The total size of all datafiles is about 40GB and the backup failes after
about 20GB so the database size can hardly be a problem.

Any help would be greatly appreciated.

Vilmundur Palmason
Systems Engineer
Nyherji HF
Reykjavik
Iceland



----------------------------------------
The full saga:
The full saga:
I first ran into problems getting obackup to start getting the message:

EBU-0:
  on 30-JUN-98 18:19:46 [ 59914 : sbrmget ]
EBU-2230: Error receiving acknowledgment of INIT message from pid 44562
  on 30-JUN-98 18:19:46 [ 59914 : brcriini ]
Job 62 with pid=59914 completed on 30-JUN-98 18:22:00
EBU-2003: Job 62 failed due to an internal error
  on 30-JUN-98 18:22:00 [ 59914 : brccmd ]

BACKUP job FAILED

After a fix was applied from Oracle (which involved relinking the ebu
components), this problem was no longer seen and database backup started
However, now it looked like the obackup processes were increasing in size
during the backup.  The backup ran about halfway , then came the error:

EBU-522: Could not allocate 131072 bytes

I increased the 'Soft DATA segment' parameter for the oracle user (from the
default 131MB, which should be plenty).
Next time it went a little further but then got SIGNAL 4.

Following is the obackup log file for the latest run and information about
installed ADSM components.

----------------------------------------
Oracle7 Enterprise Backup Utility: Release 2.1.0.1.2 - Production on Tue
Oracle7 Enterprise Backup Utility: Release 2.1.0.1.2 - Production on Tue
Jun 30 18:52:39 1998

Copyright (c) Oracle Corporation 1979, 1996.  All rights reserved.


CORE Version 3.5.3.0.0 - Production
NLSRTL Version 3.2.3.0.0 - Production
Tape Management Software: Release 32.3.65.76
Tape API: Release 0.0
ORACLE_HOME = /u01/app/oracle/product/7.3.3
System name:   AIX
Node name:     sleipnir
Release:  2
Version:  4
Machine:  004200994C00

Command script "online.slp", follows:
      backup online database
      db_name = "SLEIPNIR"
      oracle_sid = "slp"
      log = "/u01/app/oracle/product/7.3.3/obackup/log/online.log"
      trace = "/u01/app/oracle/product/7.3.3/obackup/log/online.trc"
      parallel = 2

Verifying presence of Instance Manager process...
Found Instance Manager (brd) process running with pid = 75790

Started job 63 with pid=44650 on 30-JUN-98 18:52:42 for database:
     DB_NAME = SLEIPNIR
  ORACLE_SID = slp

Verifying registration information...
No registration update is needed, Target Database information is up to date

Using database information registered on 30-JUN-98 18:03:37

Starting job building, involved files are:
  Datafile "/u03/oradata/slp/system01.dbf"
  Datafile "/u03/oradata/slp/tools01.dbf"
  Datafile "/u03/oradata/slp/users01.dbf"
  Datafile "/u03/oradata/slp/users02.dbf"
  Datafile "/u03/oradata/slp/users03.dbf"
  Datafile "/u03/oradata/slp/users04.dbf"
  Datafile "/u03/oradata/slp/users05.dbf"
  Datafile "/u01/oradata/slp/users06.dbf"
  Datafile "/u03/oradata/slp/users06.dbf"
  Datafile "/u02/oradata/slp/users08.dbf"
  Datafile "/u02/oradata/slp/users09.dbf"
  Datafile "/u02/oradata/slp/users10.dbf"
  Datafile "/u02/oradata/slp/users11.dbf"
  Datafile "/u02/oradata/slp/rbs01.dbf"
  Datafile "/u02/oradata/slp/rbs02.dbf"
  Datafile "/u02/oradata/slp/rbs03.dbf"
  Datafile "/u04/oradata/slp/temp01.dbf"
  Datafile "/u05/oradata/slp/rb01.dbf"
  Datafile "/u05/oradata/slp/rb03.dbf"
  Datafile "/u05/oradata/slp/rb04.dbf"
  Datafile "/u05/oradata/slp/rb05.dbf"
  Datafile "/u05/oradata/slp/rb06.dbf"
  Datafile "/u05/oradata/slp/rb02.dbf"
  Datafile "/u04/oradata/slp/lidata01.dbf"
  Datafile "/u04/oradata/slp/lidata02.dbf"
  Datafile "/u04/oradata/slp/lidata03.dbf"
  Datafile "/u04/oradata/slp/lidata04.dbf"
  Datafile "/u05/oradata/slp/liindex01.dbf"
  Datafile "/u04/oradata/slp/rbsload01.dbf"
  Datafile "/u04/oradata/slp/rbsload02.dbf"
  Datafile "/u04/oradata/slp/rbsload03.dbf"
  Datafile "/u04/oradata/slp/rbsload04.dbf"
  Datafile "/u04/oradata/slp/rbsload05.dbf"
  Datafile "/u04/oradata/slp/rbsload06.dbf"
  Datafile "/u04/oradata/slp/rbsload07.dbf"
  Datafile "/u04/oradata/slp/rtemp01.dbf"
  Datafile "/u04/oradata/slp/rtemp02.dbf"
  Datafile "/u04/oradata/slp/rtemp03.dbf"
  Datafile "/u04/oradata/slp/rtemp04.dbf"
  Datafile "/u04/oradata/slp/rtemp05.dbf"
  Datafile "/u04/oradata/slp/rtemp06.dbf"
  Datafile "/u04/oradata/slp/rbindex01.dbf"
  Datafile "/u04/oradata/slp/rbindex02.dbf"
  Datafile "/u04/oradata/slp/rbindex03.dbf"
  Datafile "/u04/oradata/slp/liahdata01.dbf"
  Datafile "/u04/oradata/slp/liahdata02.dbf"
  Datafile "/u04/oradata/slp/liahdata03.dbf"
  Datafile "/u04/oradata/slp/liahdata04.dbf"
  Datafile "/u04/oradata/slp/liahdata05.dbf"
  Datafile "/u04/oradata/slp/liahdata06.dbf"
  Datafile "/u04/oradata/slp/liahdata07.dbf"
  Datafile "/u04/oradata/slp/liahdata08.dbf"
  Datafile "/u04/oradata/slp/liahdata09.dbf"
  Datafile "/u04/oradata/slp/liahdata10.dbf"
  Datafile "/u04/oradata/slp/liahdata11.dbf"
  Datafile "/u04/oradata/slp/liahdata12.dbf"
  Datafile "/u04/oradata/slp/liahdata13.dbf"
  Datafile "/u04/oradata/slp/liahdata14.dbf"
  Datafile "/u04/oradata/slp/liahdata15.dbf"
  Datafile "/u04/oradata/slp/liahdata16.dbf"
  Datafile "/u04/oradata/slp/liahdata17.dbf"
  Datafile "/u05/oradata/slp/liahindex01.dbf"
  Datafile "/u05/oradata/slp/liahindex02.dbf"
  Datafile "/u05/oradata/slp/liahindex03.dbf"
  Datafile "/u05/oradata/slp/liahindex04.dbf"
  Datafile "/u05/oradata/slp/liahindex05.dbf"
  Datafile "/u05/oradata/slp/liahindex06.dbf"
  Datafile "/u05/oradata/slp/liahindex07.dbf"
  Datafile "/u05/oradata/slp/liahindex08.dbf"
  Parameter file "/u01/app/oracle/admin/slp/pfile/initslp.ora"
  Saving Current Control file "/u01/oradata/slp/ctrl01.ctl"
    as Backup Control file "/u01/oradata/slp/ctrl01.ctl.00063"
  Archive log directory "/u04/oradata/slp/arch"
Ending job building

Number of parallel I/O streams: 2
        Disk input/output size: 65536 bytes
         Maximum tape I/O size: 131072 bytes
    Buffer size per I/O stream: 524288 bytes
      Total shared memory used: 1315280 bytes
                    Semaphores: 12 (in 1 sets)

Starting backing up to tape
  Issuing ALTER TABLESPACE SYSTEM BEGIN BACKUP statement
  Issuing ALTER TABLESPACE TOOLS BEGIN BACKUP statement
  Writing database file "/u03/oradata/slp/tools01.dbf"
    to Backup file "10408899232768" for 20M bytes
  Writing database file "/u03/oradata/slp/system01.dbf"
    to Backup file "10407899232768" for 120M bytes
  Finished writing database file "/u03/oradata/slp/system01.dbf" (   0,85
MB/s)
  Issuing ALTER TABLESPACE SYSTEM END BACKUP statement
  Finished writing database file "/u03/oradata/slp/tools01.dbf" (   0,00
MB/s)
  Issuing ALTER TABLESPACE USERS BEGIN BACKUP statement
  Issuing ALTER TABLESPACE TOOLS END BACKUP statement
  Writing database file "/u03/oradata/slp/users01.dbf"
    to Backup file "10409899232768" for 400M bytes
  Writing database file "/u03/oradata/slp/users02.dbf"
    to Backup file "10410899232768" for 800M bytes
  Finished writing database file "/u03/oradata/slp/users01.dbf" (   3,50
MB/s)
  Writing database file "/u03/oradata/slp/users03.dbf"
    to Backup file "10411899232768" for 950M bytes
  Finished writing database file "/u03/oradata/slp/users02.dbf" (   2,11
MB/s)
  Writing database file "/u03/oradata/slp/users04.dbf"
    to Backup file "10412899232768" for 800M bytes
  Finished writing database file "/u03/oradata/slp/users03.dbf" (   1,88
MB/s)
  Writing database file "/u03/oradata/slp/users05.dbf"
    to Backup file "10413899232768" for 300M bytes
  Finished writing database file "/u03/oradata/slp/users04.dbf" (   1,38
MB/s)
  Writing database file "/u01/oradata/slp/users06.dbf"
    to Backup file "10414899232768" for 800M bytes
  Finished writing database file "/u03/oradata/slp/users05.dbf" (   0,90
MB/s)
  Writing database file "/u03/oradata/slp/users06.dbf"
    to Backup file "10415899232768" for 400M bytes
  Finished writing database file "/u01/oradata/slp/users06.dbf" (   2,57
MB/s)
  Writing database file "/u02/oradata/slp/users08.dbf"
    to Backup file "10416899232768" for 600M bytes
  Finished writing database file "/u03/oradata/slp/users06.dbf" (   0,68
MB/s)
  Writing database file "/u02/oradata/slp/users09.dbf"
    to Backup file "10417899232768" for 800M bytes
  Finished writing database file "/u02/oradata/slp/users08.dbf" (   2,17
MB/s)
  Writing database file "/u02/oradata/slp/users10.dbf"
    to Backup file "10418899232768" for 800M bytes
  Finished writing database file "/u02/oradata/slp/users09.dbf" (   1,79
MB/s)
  Writing database file "/u02/oradata/slp/users11.dbf"
    to Backup file "10419899232768" for 600M bytes
  Finished writing database file "/u02/oradata/slp/users10.dbf" (   1,59
MB/s)
  Issuing ALTER TABLESPACE RBS BEGIN BACKUP statement
  Finished writing database file "/u02/oradata/slp/users11.dbf" (   1,09
MB/s)
  Issuing ALTER TABLESPACE USERS END BACKUP statement
  Writing database file "/u02/oradata/slp/rbs01.dbf"
    to Backup file "10420899232768" for 200M bytes
  Writing database file "/u02/oradata/slp/rbs02.dbf"
    to Backup file "10421899232768" for 200M bytes
  Finished writing database file "/u02/oradata/slp/rbs01.dbf" (   3,33
MB/s)
  Writing database file "/u02/oradata/slp/rbs03.dbf"
    to Backup file "10422899232768" for 200M bytes
  Finished writing database file "/u02/oradata/slp/rbs02.dbf" (   1,66
MB/s)
  Issuing ALTER TABLESPACE TEMP BEGIN BACKUP statement
  Finished writing database file "/u02/oradata/slp/rbs03.dbf" (   1,63
MB/s)
  Issuing ALTER TABLESPACE RBS END BACKUP statement
  Writing database file "/u04/oradata/slp/temp01.dbf"
    to Backup file "10423899232768" for 200M bytes
  Issuing ALTER TABLESPACE RB BEGIN BACKUP statement
  Finished writing database file "/u04/oradata/slp/temp01.dbf" (   2,14
MB/s)
  Issuing ALTER TABLESPACE TEMP END BACKUP statement
  Writing database file "/u05/oradata/slp/rb03.dbf"
    to Backup file "10425899232768" for 500M bytes
  Writing database file "/u05/oradata/slp/rb01.dbf"
    to Backup file "10424899232768" for 500M bytes
  Finished writing database file "/u05/oradata/slp/rb03.dbf" (   3,23 MB/s)
  Writing database file "/u05/oradata/slp/rb04.dbf"
    to Backup file "10426899232768" for 500M bytes
  Finished writing database file "/u05/oradata/slp/rb01.dbf" (   1,55 MB/s)
  Writing database file "/u05/oradata/slp/rb05.dbf"
    to Backup file "10427899232768" for 500M bytes
  Finished writing database file "/u05/oradata/slp/rb04.dbf" (   1,70 MB/s)
  Writing database file "/u05/oradata/slp/rb06.dbf"
    to Backup file "10428899232768" for 200M bytes
  Finished writing database file "/u05/oradata/slp/rb05.dbf" (   0,14 MB/s)
  Finished writing database file "/u05/oradata/slp/rb06.dbf" (   0,00 MB/s)
  Writing database file "/u05/oradata/slp/rb02.dbf"
    to Backup file "10429899232768" for 500M bytes
  Issuing ALTER TABLESPACE LIDATA BEGIN BACKUP statement
  Writing database file "/u04/oradata/slp/lidata01.dbf"
    to Backup file "10430899232768" for 500M bytes
  Finished writing database file "/u05/oradata/slp/rb02.dbf" (   3,09 MB/s)
  Issuing ALTER TABLESPACE RB END BACKUP statement
  Writing database file "/u04/oradata/slp/lidata02.dbf"
    to Backup file "10431899232768" for 490M bytes
  Finished writing database file "/u04/oradata/slp/lidata01.dbf" (   1,90
MB/s)
  Writing database file "/u04/oradata/slp/lidata03.dbf"
    to Backup file "10432899232768" for 200M bytes
  Finished writing database file "/u04/oradata/slp/lidata02.dbf" (   1,41
MB/s)
  Writing database file "/u04/oradata/slp/lidata04.dbf"
    to Backup file "10433899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/lidata03.dbf" (   0,86
MB/s)
  Issuing ALTER TABLESPACE LIINDEX BEGIN BACKUP statement
  Writing database file "/u05/oradata/slp/liindex01.dbf"
    to Backup file "10434899232768" for 200M bytes
  Finished writing database file "/u04/oradata/slp/lidata04.dbf" (   1,10
MB/s)
  Issuing ALTER TABLESPACE LIDATA END BACKUP statement
  Issuing ALTER TABLESPACE RBSLOAD BEGIN BACKUP statement
  Writing database file "/u04/oradata/slp/rbsload01.dbf"
    to Backup file "10435899232768" for 500M bytes
  Finished writing database file "/u05/oradata/slp/liindex01.dbf" (   1,71
MB/s)
  Issuing ALTER TABLESPACE LIINDEX END BACKUP statement
  Writing database file "/u04/oradata/slp/rbsload02.dbf"
    to Backup file "10436899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rbsload01.dbf" (   2,98
MB/s)
  Writing database file "/u04/oradata/slp/rbsload03.dbf"
    to Backup file "10437899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rbsload02.dbf" (   1,70
MB/s)
  Writing database file "/u04/oradata/slp/rbsload04.dbf"
    to Backup file "10438899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rbsload03.dbf" (   1,68
MB/s)
  Writing database file "/u04/oradata/slp/rbsload05.dbf"
    to Backup file "10439899232768" for 499M bytes
  Finished writing database file "/u04/oradata/slp/rbsload04.dbf" (   1,68
MB/s)
  Writing database file "/u04/oradata/slp/rbsload06.dbf"
    to Backup file "10440899232768" for 499M bytes
  Finished writing database file "/u04/oradata/slp/rbsload05.dbf" (   1,71
MB/s)
  Writing database file "/u04/oradata/slp/rbsload07.dbf"
    to Backup file "10441899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rbsload06.dbf" (   1,72
MB/s)
  Issuing ALTER TABLESPACE RTEMP BEGIN BACKUP statement
  Writing database file "/u04/oradata/slp/rtemp01.dbf"
    to Backup file "10442899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rbsload07.dbf" (   1,70
MB/s)
  Issuing ALTER TABLESPACE RBSLOAD END BACKUP statement
  Writing database file "/u04/oradata/slp/rtemp02.dbf"
    to Backup file "10443899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rtemp01.dbf" (   1,68
MB/s)
  Writing database file "/u04/oradata/slp/rtemp03.dbf"
    to Backup file "10444899232768" for 200M bytes
  Finished writing database file "/u04/oradata/slp/rtemp02.dbf" (   1,64
MB/s)
  Writing database file "/u04/oradata/slp/rtemp04.dbf"
    to Backup file "10445899232768" for 500M bytes
  Finished writing database file "/u04/oradata/slp/rtemp03.dbf" (   0,96
MB/s)
  Writing database file "/u04/oradata/slp/rtemp05.dbf"
    to Backup file "10446899232768" for 500M bytes
EBU-4003: Process pid 78454 exited; exit code = 5
  on 06/30/1998 20:46:55 [ 77424 : brio ]
  Issuing ALTER TABLESPACE RTEMP END BACKUP statement
BFS 10407899232768 of job 63 removed successfully from backup media
BFS 10408899232768 of job 63 removed successfully from backup media
BFS 10409899232768 of job 63 removed successfully from backup media
BFS 10410899232768 of job 63 removed successfully from backup media
BFS 10411899232768 of job 63 removed successfully from backup media
BFS 10412899232768 of job 63 removed successfully from backup media
BFS 10413899232768 of job 63 removed successfully from backup media
BFS 10414899232768 of job 63 removed successfully from backup media
BFS 10415899232768 of job 63 removed successfully from backup media
BFS 10416899232768 of job 63 removed successfully from backup media
BFS 10417899232768 of job 63 removed successfully from backup media
BFS 10418899232768 of job 63 removed successfully from backup media
BFS 10419899232768 of job 63 removed successfully from backup media
BFS 10420899232768 of job 63 removed successfully from backup media
BFS 10421899232768 of job 63 removed successfully from backup media
BFS 10422899232768 of job 63 removed successfully from backup media
BFS 10423899232768 of job 63 removed successfully from backup media
BFS 10424899232768 of job 63 removed successfully from backup media
BFS 10425899232768 of job 63 removed successfully from backup media
BFS 10426899232768 of job 63 removed successfully from backup media
BFS 10427899232768 of job 63 removed successfully from backup media
BFS 10428899232768 of job 63 removed successfully from backup media
BFS 10429899232768 of job 63 removed successfully from backup media
BFS 10430899232768 of job 63 removed successfully from backup media
BFS 10431899232768 of job 63 removed successfully from backup media
BFS 10432899232768 of job 63 removed successfully from backup media
BFS 10433899232768 of job 63 removed successfully from backup media
BFS 10434899232768 of job 63 removed successfully from backup media
BFS 10435899232768 of job 63 removed successfully from backup media
BFS 10436899232768 of job 63 removed successfully from backup media
BFS 10437899232768 of job 63 removed successfully from backup media
BFS 10438899232768 of job 63 removed successfully from backup media
BFS 10439899232768 of job 63 removed successfully from backup media
BFS 10440899232768 of job 63 removed successfully from backup media
BFS 10441899232768 of job 63 removed successfully from backup media
BFS 10442899232768 of job 63 removed successfully from backup media
BFS 10443899232768 of job 63 removed successfully from backup media
BFS 10444899232768 of job 63 removed successfully from backup media
Job 63 with pid=44650 completed on 30-JUN-98 20:48:23
EBU-2003: Job 63 failed due to an internal error
  on 30-JUN-98 20:48:24 [ 44650 : brccmd ]

BACKUP job FAILED

----------------------------------------
ADSM Components installed:
ADSM Components installed:

  adsm.api.client.aix42     3.1.20.3    C    ADSM Client - Application
                                             Programming Interface
  adsm.client.admin         2.1.20.6    C    ADSM Client - Administrative
GUI
  adsm.client.aix42.admin   3.1.20.0    C    ADSM Client - Administrative
                                             Command Line
  adsm.client.aix42.api     3.1.20.0    O    ADSM Client - Application
                                             Programming Interface
  adsm.client.aix42.base    3.1.20.0    C    ADSM Client - Backup/Archive
  adsm.client.aix42.common  3.1.20.0    C    ADSM Client - Common Files
  adsm.client.aob            2.1.0.7    C    Oracle ADSMConnect Agent for
AIX
                                             4.2.1 - 4.3
  adsm.client.base          2.1.20.6    O    ADSM Client - Backup/Archive
  adsm.client.common        2.1.20.6    C    ADSM Client - Common Files
  adsm.devices.rte           3.1.0.0    C    ADSM Device Support runtime
  adsm.license.rte           3.1.0.0    C    ADSM Server License
Registration
  adsm.server.gif            3.1.0.0    C    ADSM Server Web Administrator
                                             Icons
  adsm.server.rte            3.1.0.0    C    ADSM Server Runtime
  adsm.web.client.aix42     3.1.20.0    C    ADSM Client - WebClient
<Prev in Thread] Current Thread [Next in Thread>