ADSM-L

ANR9999D tb.c(3354)

2006-10-24 07:33:53
Subject: ANR9999D tb.c(3354)
From: Dierk Harbort <Dierk.Harbort AT BUERGEL DOT DE>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 24 Oct 2006 13:30:50 +0200
Hello TSMers,

the error wich occurs in actlog today is the following:

Date/Time                Message
--------------------
----------------------------------------------------------
10/24/2006 10:53:33      ANR9999D tb.c(3354): ThreadId<78> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2193)
10/24/2006 10:53:46      ANR9999D tb.c(3354): ThreadId<78> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2194)
10/24/2006 10:53:59      ANR9999D tb.c(3354): ThreadId<78> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2195)
10/24/2006 10:57:23      ANR9999D tb.c(3354): ThreadId<78> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2196)
10/24/2006 12:16:16      ANR9999D tb.c(3354): ThreadId<62> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2350)
10/24/2006 12:17:39      ANR9999D tb.c(3354): ThreadId<62> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2351)
10/24/2006 12:21:29      ANR2017I Administrator BWIHAB issued command:
QUERY ACTLOG
                          begint=17:00 begind=-1 search=9999 (SESSION:
2066)


Where does it come from? From this select command (from IBM operational
reporting tool):

10/24/2006 12:17:24      ANR0407I Session 2351 started for administrator
HABREP
                          (WinNT) (Tcp/Ip 192.168.1.130(4736)).(SESSION:
2351)
10/24/2006 12:17:24      ANR2017I Administrator HABREP issued command:
select
                          substr(char(date_time), 1, 16) as
date_time,message from
                          actlog where
cast((current_timestamp-date_time)minutes as
                          decimal)<960 and message not like
'ANR2034E%'(SESSION:
                          2351)
10/24/2006 12:17:39      ANR9999D tb.c(3354): ThreadId<62> >>ERROR Database
Page
                          Format: Invalid sibling for page 766269, left
sibling =
                          0.(SESSION: 2351)
10/24/2006 12:17:39      ANR2034E SELECT: No match found using this
                          criteria.(SESSION: 2351)


What happened before / yesterday?

Yesterday 14:20 the operation "define dbvolume
/tsm/brz1srvbk1/db1/brz1srvbk1_db110.dsm formatsize=2049"
failed, due to an error in the filesystem: After beginning to define the
new dbvol, the filesystem
(SLES9 ext3) went into read-only mode, caused by a Linux error.
TSM set all dbvols on this first path offline, the mirrors stayed online
and everything worked on well;
and the situation was shown in the actlog.
To provide a big crash (with dataloss) we decided to stop the server, and
sysadmins repaired the filesystem.
The Server started to work again at 16:58, no errors where seen. So we
thought everything was fine.

What happened today?
The first known error ist 10:53 today, as shown above: IBMs operational
reporting can't show entrys in the activvity log report.
So now while checking the system in the actlog we don't see anything newer
then yesterday 14:16 - the entrys from
14:20 are gone?? What happened? But q actlog in the following way other
information occurs:

tsm: BRZ1SRVBK1>q actlog begint=14:18 begind=-1
ANR2034E QUERY ACTLOG: No match found using this criteria.
ANS8001I Return code 11.

tsm: BRZ1SRVBK1>q actlog begint=14:19 begind=-1
ANR2034E QUERY ACTLOG: No match found using this criteria.
ANS8001I Return code 11.

tsm: BRZ1SRVBK1>q actlog begint=14:20 begind=-1

Date/Time                Message
--------------------
----------------------------------------------------------
10/23/2006 14:20:01      ANR0403I Session 18203 ended for node BRZ1SRVBK1
                          (Linux86).(SESSION: 18203)
10/23/2006 14:20:01      ANR0403I Session 18202 ended for node BRZ1SRVBK1
                          (Linux86).(SESSION: 18202)
10/23/2006 14:20:17      ANR0406I Session 18204 started for node BRZ1SRVBK1
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43007)).(SESSION:
                          18204)

tsm: BRZ1SRVBK1>

...and:

tsm: BRZ1SRVBK1>q actlog begint=14:23 begind=-1
ANR2034E QUERY ACTLOG: No match found using this criteria.
ANS8001I Return code 11.

tsm: BRZ1SRVBK1>q actlog begint=14:24 begind=-1

Date/Time                Message
--------------------
----------------------------------------------------------
10/23/2006 14:24:34      ANR0482W Session 17542 for node BWIHAB (Linux86)
                          terminated - idle for more than 240
minutes.(SESSION:
                          17542)
10/23/2006 14:24:45      ANR0407I Session 18220 started for administrator
OPER
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43066)).(SESSION:
                          18220)
10/23/2006 14:24:45      ANR2017I Administrator OPER issued command: QUERY
DRIVE
                          f=d (SESSION: 18220)
10/23/2006 14:24:45      ANR0405I Session 18220 ended for administrator
OPER
                          (Linux86).(SESSION: 18220)
10/23/2006 14:24:45      ANR0407I Session 18221 started for administrator
OPER
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43067)).(SESSION:
                          18221)
10/23/2006 14:24:45      ANR2017I Administrator OPER issued command: QUERY
DRIVE
                          f=d (SESSION: 18221)
10/23/2006 14:24:45      ANR0405I Session 18221 ended for administrator
OPER
                          (Linux86).(SESSION: 18221)
10/23/2006 14:24:49      ANR0407I Session 18222 started for administrator
OPER
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43070)).(SESSION:
                          18222)
10/23/2006 14:24:49      ANR2017I Administrator OPER issued command: QUERY
MOUNT
                          (SESSION: 18222)
10/23/2006 14:24:49      ANR2034E QUERY MOUNT: No match found using this
                          criteria.(SESSION: 18222)
10/23/2006 14:24:49      ANR0405I Session 18222 ended for administrator
OPER
                          (Linux86).(SESSION: 18222)
10/23/2006 14:24:49      ANR0407I Session 18223 started for administrator
OPER
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43071)).(SESSION:
                          18223)
10/23/2006 14:24:49      ANR2017I Administrator OPER issued command: QUERY
MOUNT
                          (SESSION: 18223)
10/23/2006 14:24:49      ANR2034E QUERY MOUNT: No match found using this
                          criteria.(SESSION: 18223)
10/23/2006 14:24:49      ANR0405I Session 18223 ended for administrator
OPER
                          (Linux86).(SESSION: 18223)
10/23/2006 14:25:41      ANR0407I Session 18224 started for administrator
BWIHAB
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43076)).(SESSION:
                          18224)
10/23/2006 14:25:41      ANR2017I Administrator BWIHAB issued command:
DEFINE
                          DBVOLUME /tsm/brz1srvbk1/db1/brz1srvbk1_db110.dsm
                          formatsize=2049 (SESSION: 18224)

tsm: BRZ1SRVBK1>

...and:

tsm: BRZ1SRVBK1>q actlog begint=14:25 begind=-1

Date/Time                Message
--------------------
----------------------------------------------------------
10/23/2006 14:25:41      ANR0407I Session 18224 started for administrator
BWIHAB
                          (Linux86) (Tcp/Ip
BRZ1SRVBK1.buergel.de(43076)).(SESSION:
                          18224)
10/23/2006 14:25:41      ANR2017I Administrator BWIHAB issued command:
DEFINE
                          DBVOLUME /tsm/brz1srvbk1/db1/brz1srvbk1_db110.dsm
                          formatsize=2049 (SESSION: 18224)

tsm: BRZ1SRVBK1>

...and:

tsm: BRZ1SRVBK1>q actlog begint=14:27 begind=-1 >
/tsm/brz1srvbk1/data1/log/after1427.txt
Output of command redirected to file
'/tsm/brz1srvbk1/data1/log/after1427.txt'

..what means: The actlog doesn' scroll on, you have to ask in parts. Seems
to me that there is something wrong in the database?!

But: Everything is fine until now, nomore problems. So what happened to the
actlog? Is a "auditdb" neccessary?
Or something else? How can we be sure?

Any help is appreciated, many thanks IA!


Other regular things about our tsm today:
All servertasks (backups, migrations, copy tapes, expiration, backup db,
space recl) worked properly today.

So now the server is dissabled for sessions, and an additional backup db
was done.


With kind regards
Mit freundlichem Gruß
Dierk Harbort
EDV Produktionsplanung und -betreuung

Hamburg

<Prev in Thread] Current Thread [Next in Thread>
  • ANR9999D tb.c(3354), Dierk Harbort <=