Hi all
We have a problem with an AIX5.2 Veritas NB4.5 mediaserver, attached to
a robot SAN, connected to 20 STK drives(intermix of 9940A, 9940B and
9840). We have about 5 different O/S mediaservers(TRU64 for most), SUN
master and 1 AIX mediaserver (the one in question). We had all 20 drives
available to SSO
on the AIX mediaserver at startup time, but it looks like after a
while, we see more(phantom) drives, of which several are undefined.
Now...if you know SSO, you'll understand that you cannot just UP the
drives when defined. SSO gets confused specifically on the AIX
mediaserver as there are tapes on undefined drives, drives DOWNED and a
whole mess around drive availability to SSO on this AIX mediaserver. We
think this to be an OS problem/Veritas communication problem as we do
not get the same on TRU64? Any opinion on this - as we're clutching
straws?
We want SSO to see all drives on this AIX mediaserver, and the 20 drives
should be seen by the OS...not more "phantom" drives. Also - the
"defined" status is worrisome? NOTE : There should be 1 internal tape
drive(RMT0), and only 20 STK drives - see phantoms :
$ lsdev -Cc tape
rmt0 Defined 3A-08-00-6,0 LVD SCSI Tape Drive
rmt1 Available 47-08-01 STK 9940A FC Tape Drive
rmt2 Available 47-08-01 STK 9940A FC Tape Drive
rmt3 Available 47-08-01 STK 9840 FC Tape Drive
rmt4 Available 47-08-01 STK 9840 FC Tape Drive
rmt5 Available 47-08-01 STK 9940A FC Tape Drive
rmt6 Available 5b-08-01 STK 9940A FC Tape Drive
rmt7 Available 5b-08-01 STK 9940B FC Tape Drive
rmt8 Available 5b-08-01 STK 9940B FC Tape Drive
rmt9 Available 5b-08-01 STK 9940A FC Tape Drive
rmt10 Defined 5b-08-01 STK 9940B FC Tape Drive
rmt11 Available 5b-08-01 STK 9940B FC Tape Drive
rmt12 Available 47-08-01 STK 9940A FC Tape Drive
rmt13 Defined 47-08-01 STK 9940A FC Tape Drive
rmt14 Defined 47-08-01 STK 9940A FC Tape Drive
rmt15 Defined 47-08-01 STK 9940A FC Tape Drive
rmt16 Defined 47-08-01 STK 9940A FC Tape Drive
rmt17 Defined 5b-08-01 STK 9940B FC Tape Drive
rmt18 Defined 5b-08-01 STK 9940A FC Tape Drive
rmt19 Defined 5b-08-01 STK 9840 FC Tape Drive
rmt20 Defined 47-08-01 Other FC SCSI Tape Drive
rmt21 Defined 47-08-01 Other FC SCSI Tape Drive
rmt22 Defined 47-08-01 Other FC SCSI Tape Drive
rmt23 Defined 47-08-01 Other FC SCSI Tape Drive
rmt24 Defined 47-08-01 Other FC SCSI Tape Drive
rmt25 Defined 47-08-01 Other FC SCSI Tape Drive
rmt26 Defined 47-08-01 Other FC SCSI Tape Drive
rmt27 Defined 47-08-01 Other FC SCSI Tape Drive
rmt28 Defined 47-08-01 Other FC SCSI Tape Drive
rmt29 Defined 47-08-01 Other FC SCSI Tape Drive
rmt30 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt31 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt32 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt33 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt34 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt35 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt36 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt37 Defined 5b-08-01 STK 9840 FC Tape Drive
rmt38 Available 5b-08-01 STK 9840 FC Tape Drive
rmt39 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt40 Defined 5b-08-01 Other FC SCSI Tape Drive
rmt41 Defined 5b-08-01 Other FC SCSI Tape Drive
$
PLEASE HELP?
Winnie
-----Original Message-----
From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu] On Behalf Of
veritas-bu-request AT mailman.eng.auburn DOT edu
Sent: Thursday, 12 August 2004 7:07 PM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Veritas-bu digest, Vol 1 #3409 - 5 msgs
Send Veritas-bu mailing list submissions to
veritas-bu AT mailman.eng.auburn DOT edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
or, via email, send a message with subject or body 'help' to
veritas-bu-request AT mailman.eng.auburn DOT edu
You can reach the person managing the list at
veritas-bu-admin AT mailman.eng.auburn DOT edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Veritas-bu digest..."
Today's Topics:
1. Re: Monitoring perfomance at the buffer level (Dave Markham)
2. Re: Listing tapes in a media server's media catalog (Jack
Forester, Jr.)
3. Re: Monitoring perfomance at the buffer level (Dave Markham)
4. Re: Monitoring perfomance at the buffer level (Ed Wilts)
5. Re: Monitoring perfomance at the buffer level (Ed Wilts)
--__--__--
Message: 1
Date: Thu, 12 Aug 2004 13:59:46 +0100
From: Dave Markham <dave.markham AT icl DOT net>
Reply-To: dave.markham AT icl DOT net
To: Ed Wilts <ewilts AT ewilts DOT org>
CC: Mark.Donaldson AT cexp DOT com, veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Monitoring perfomance at the buffer level
Ed Wilts wrote:
>On Tue, Aug 10, 2004 at 03:59:35PM -0600, Mark.Donaldson AT cexp DOT com
wrote:
>
>
>>Here's a quick and dirty script that sweeps the bptm logs on a media
server
>>for a supplied policy name and reports the "fill_buffer, waiting on
empty
>>buffer" and "write_backup, waiting on full buffer" statistics.
>>
>>Output looks like this:
>>
>>
>>
>>>policy_perf Hot_PRD
>>>
>>>
>>## Gathering data..........Done.
>>## Write to buffer waiting on available buffer:
>>Min: 0 Avg: 356 Max: 5877 with 285 samples
>>
>>## Write to tape waiting on full buffer:
>>Min: 0 Avg: 43373 Max: 290583 with 7 samples
>>
>>
>
>I've added a section to optionally pass in a date so I can go back
>through previous days logs and here's a sample:
>
>[root@osiris ewilts]# ./perf.sh osiris-vpn 081004
>Using /usr/openv/netbackup/logs/bptm/log.081004
>## Gathering data.................................................Done.
>## Write to buffer waiting on available buffer:
>Min: 0 Avg: 1216 Max: 42479 with 150 samples
>
>## Write to tape waiting on full buffer:
>Min: 317 Avg: 34382 Max: 157723 with 48 samples
>
>
>
>>If the Write to Buffer is waiting for an available empty buffer a
whole
>>bunch, then perhaps you should increase your buffer count. If you're
tape
>>writing process waiting on a full buffer a lot, then you're starving
your
>>tape drives and you should find a way to increase the delivery of
client
>>data to your media server or increase your multiplexing factor.
>>
>>
>
>So what's a "whole bunch"? Is what I'm seeing an issue I should deal
>with? Don't things like incrementals really slow down the tape
>processing?
>
>Can it be broken down by host instead of by policy? Having multiple
>hosts per policy would make it difficult to target a system to fix.
>There's also the minor issue of not knowing which hosts or policies
even
>have buffer messages in bptm. The script is an excellent start
though.
>
>My overall issue is that although we have GigE connections between many
>hosts and the media servers, and trying to drive 8 SDLT220 drives in an
>L700, we almost never exceed 11MBs of traffic coming into the media
>servers. It's like there's a cap there that we just haven't been able
to
>remove.
>
>Thanks,
> .../Ed
>
>
>
I just saw this script at beginning of this thread and my current today
log file has no policy info so i quickly added a dirty way to check
yesterdays log with a -y flag
policy=$1
yesterday=$2
today=`date +%m%d%y`
TMPFILEf=/tmp/`basename $0`.tmp.f
TMPFILEw=/tmp/`basename $0`.tmp.w
[ -f $TMPFILEf ] && rm -f $TMPFILEf
[ -f $TMPFILEw ] && rm -f $TMPFILEw
[ $2 = "" ] && yesterday = "undef"
if [ $yesterday = "-y" ]
then
if [ -n `echo $today | grep ^0` ]
then
today1=`expr $today - 1`
today=0$today1
else
today=`expr $today - 1`
fi
fi
echo "## Gathering data.\c"
...............script continues
On answer to your capping have you check the gigabit ndd settings for
the device ? This may help :-
This is nicked from a script i wrote to auto set device paremeters on
boot up or if ran manually to check them.
do_check_ge()
{
DEV=$1
INST=$2
ndd -set $DEV instance ${INST}
echo "+-------------------+"
if [ `ndd $DEV link_status` = 0 ];then
echo "$DEV${INST} status is down";else
echo "$DEV${INST} status is up"
fi
if [ `ndd $DEV link_speed` = 1000 ];then
echo "$DEV${INST} link speed 1000 Mbps";else
echo "$DEV${INST} link not up"
fi
if [ `ndd $DEV link_mode` = 0 ];then
echo "$DEV${INST} link mode Half-Duplex";else
echo "$DEV${INST} link mode Full-Duplex"
fi
if [ `ndd $DEV adv_1000autoneg_cap` = 0 ];then
echo "$DEV${INST} Auto-Negotiation-OFF";else
echo "$DEV${INST} Auto-Negotiation-ON"
fi
if [ `ndd $DEV adv_pauseTX` = 0 ];then
echo "$DEV${INST} Transmit PAUSE Not
Capable(default)";else
echo "$DEV${INST} Transmit PAUSE Capable"
fi
if [ `ndd $DEV adv_pauseRX` = 0 ];then
echo "$DEV${INST} Receive PAUSE Not Capable";else
echo "$DEV${INST} Receive PAUSE Capable(default)"
fi
}
do_set_ge()
{
DEV=$1
INST=$2
ndd -set $DEV instance ${INST}
echo "Setting $DEV${INST} adv_1000autoneg_cap 0"
ndd -set $DEV adv_1000autoneg_cap 0
echo "Setting $DEV${INST} adv_1000fdx_cap 1"
ndd -set $DEV adv_1000fdx_cap 1
echo "Setting $DEV${INST} adv_1000hdx_cap 0"
ndd -set $DEV adv_1000hdx_cap 0
echo "Setting $DEV${INST} adv_pauseTX 0"
ndd -set $DEV adv_pauseTX 0
echo "Setting $DEV${INST} adv_pauseRX 1"
ndd -set $DEV adv_pauseRX 1
}
# Workings
case "$1" in
'check')
### Ge gigabit interface different
GE_=`nawk '$NF == "\"ge\"" {print $2}' /etc/path_to_inst | uniq`
if [ "$GE_" != "" ];then
for x in ${GE_};do
do_check_ge /dev/ge $x
done
ANS=`ckyorn -p "Do you want to force all nics 1000 Mbps ,
Full-Duplex, Auto Negotiation off?~"`
if [ $ANS = y ] || [ $ANS = Y ] || [ $ANS = YES ] || [
$ANS = yes ];then
echo "Setting Interfaces"
for x in ${GE_};do
do_set_ge /dev/ge $x
done
fi
fi
;;
'start')
### Ge gigabit interface different
GE_=`nawk '$NF == "\"ge\"" {print $2}' /etc/path_to_inst | uniq`
if [ "$GE_" != "" ];then
for x in ${GE_};do
do_set_ge /dev/ge $x
done
fi
;;
*)
echo "Usage: $0 { check | start }"
exit 1
esac
exit 0
Thanks
--__--__--
Message: 2
Date: Thu, 12 Aug 2004 09:04:00 -0400
From: "Jack Forester, Jr." <jack.l.forester AT lmco DOT com>
Subject: Re: [Veritas-bu] Listing tapes in a media server's media
catalog
To: veritas-bu <veritas-bu AT mailman.eng.auburn DOT edu>
Organization: Lockheed Martin
I want to thank everyone who emailed me with suggestions. I was able to
hack together a script to suspend all of our active tapes.
Kendall, Scott APX wrote:
>
> " I know I can use vmquery to find out if a tape is assigned, but how,
then,
> do you determine which media server owns the tape so I can give
bpexpdate
> all the information it needs to suspend the tape?"
>
> bpmedialist will give you this. If you don't sepcify the media server
to
> run it against with -h, it will run down the list against each media
server
> with a configured Storare Unit, which will eventually hit the one that
owns
> it and report information on it.
>
>
> - Scott
>
> -----Original Message-----
> From: Jack Forester, Jr. [mailto:jack.l.forester AT lmco DOT com]
> Sent: Tuesday, August 10, 2004 8:09 AM
> To: veritas-bu
> Subject: [Veritas-bu] Listing tapes in a media server's media catalog
>
> We are going to be replacing our 9940A tape drives with 9940B tape
> drives, and I will want to suspend all of the tapes that are currently
> assigned to the media servers. This is to prevent NBU from attempting
> to use a tape written by the older drives in the new drives when doing
a
> backup. The "B" drives can read tapes written by the "A" drives but
> cannot append to them (as would be the case when doing a backup on a
> partially used tape).
>
> My experience shows that if a partially used "A" tape is used in a "B"
> drive for a backup, NBU will find the tape "read only" and spit it out
> and try another tape. I want to avoid this tape search by suspending
> all of our currently assigned tapes until the backups on those tapes
> expire naturally.
>
> How do I find a list of assigned tapes and the media servers they are
> assigned to? I'd like to write a single script that I run on the
master
> server to do all the tapes if possible. I know I can use vmquery to
> find out if a tape is assigned, but how, then, do you determine which
> media server owns the tape so I can give bpexpdate all the information
> it needs to suspend the tape?
--
Jack L. Forester, Jr.
UNIX Systems Administrator, Stf
Lockheed Martin Information Technology
(304) 625-3946
--__--__--
Message: 3
Date: Thu, 12 Aug 2004 14:08:13 +0100
From: Dave Markham <dave.markham AT icl DOT net>
Reply-To: dave.markham AT icl DOT net
To: dave.markham AT icl DOT net
CC: Ed Wilts <ewilts AT ewilts DOT org>, Mark.Donaldson AT cexp DOT com,
veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Monitoring perfomance at the buffer level
Dave Markham wrote:
> Ed Wilts wrote:
>
>> On Tue, Aug 10, 2004 at 03:59:35PM -0600, Mark.Donaldson AT cexp DOT com
wrote:
>>
>>
>>> Here's a quick and dirty script that sweeps the bptm logs on a media
>>> server
>>> for a supplied policy name and reports the "fill_buffer, waiting on
>>> empty
>>> buffer" and "write_backup, waiting on full buffer" statistics.
>>>
>>> Output looks like this:
>>>
>>>
>>>
>>>> policy_perf Hot_PRD
>>>>
>>>
>>> ## Gathering data..........Done.
>>> ## Write to buffer waiting on available buffer:
>>> Min: 0 Avg: 356 Max: 5877 with 285 samples
>>>
>>> ## Write to tape waiting on full buffer:
>>> Min: 0 Avg: 43373 Max: 290583 with 7 samples
>>>
>>
>>
>> I've added a section to optionally pass in a date so I can go back
>> through previous days logs and here's a sample:
>>
>> [root@osiris ewilts]# ./perf.sh osiris-vpn 081004
>> Using /usr/openv/netbackup/logs/bptm/log.081004
>> ## Gathering
data.................................................Done.
>> ## Write to buffer waiting on available buffer:
>> Min: 0 Avg: 1216 Max: 42479 with 150 samples
>>
>> ## Write to tape waiting on full buffer:
>> Min: 317 Avg: 34382 Max: 157723 with 48 samples
>>
>>
>>
>>> If the Write to Buffer is waiting for an available empty buffer a
whole
>>> bunch, then perhaps you should increase your buffer count. If
>>> you're tape
>>> writing process waiting on a full buffer a lot, then you're starving
>>> your
>>> tape drives and you should find a way to increase the delivery of
>>> client
>>> data to your media server or increase your multiplexing factor.
>>>
>>
>>
>> So what's a "whole bunch"? Is what I'm seeing an issue I should deal
>> with? Don't things like incrementals really slow down the tape
>> processing?
>>
>> Can it be broken down by host instead of by policy? Having multiple
>> hosts per policy would make it difficult to target a system to fix.
>> There's also the minor issue of not knowing which hosts or policies
even
>> have buffer messages in bptm. The script is an excellent start
though.
>>
>> My overall issue is that although we have GigE connections between
many
>> hosts and the media servers, and trying to drive 8 SDLT220 drives in
an
>> L700, we almost never exceed 11MBs of traffic coming into the media
>> servers. It's like there's a cap there that we just haven't been able
to
>> remove.
>> Thanks,
>> .../Ed
>>
>>
>>
> I just saw this script at beginning of this thread and my current
> today log file has no policy info so i quickly added a dirty way to
> check yesterdays log with a -y flag
>
> policy=$1
> yesterday=$2
>
>
> today=`date +%m%d%y`
> TMPFILEf=/tmp/`basename $0`.tmp.f
> TMPFILEw=/tmp/`basename $0`.tmp.w
>
> [ -f $TMPFILEf ] && rm -f $TMPFILEf
> [ -f $TMPFILEw ] && rm -f $TMPFILEw
>
> [ $2 = "" ] && yesterday = "undef"
>
> if [ $yesterday = "-y" ]
> then
> if [ -n `echo $today | grep ^0` ]
> then
> today1=`expr $today - 1`
> today=0$today1
> else
> today=`expr $today - 1`
> fi
> fi
>
> echo "## Gathering data.\c"
> ...............script continues
>
>
>
> On answer to your capping have you check the gigabit ndd settings for
> the device ? This may help :-
>
> This is nicked from a script i wrote to auto set device paremeters on
> boot up or if ran manually to check them.
>
> do_check_ge()
> {
> DEV=$1
> INST=$2
>
> ndd -set $DEV instance ${INST}
> echo "+-------------------+"
> if [ `ndd $DEV link_status` = 0 ];then
> echo "$DEV${INST} status is down";else
> echo "$DEV${INST} status is up"
> fi
>
> if [ `ndd $DEV link_speed` = 1000 ];then
> echo "$DEV${INST} link speed 1000 Mbps";else
> echo "$DEV${INST} link not up"
> fi
>
> if [ `ndd $DEV link_mode` = 0 ];then
> echo "$DEV${INST} link mode Half-Duplex";else
> echo "$DEV${INST} link mode Full-Duplex"
> fi
>
> if [ `ndd $DEV adv_1000autoneg_cap` = 0 ];then
> echo "$DEV${INST} Auto-Negotiation-OFF";else
> echo "$DEV${INST} Auto-Negotiation-ON"
> fi
> if [ `ndd $DEV adv_pauseTX` = 0 ];then
> echo "$DEV${INST} Transmit PAUSE Not
> Capable(default)";else
> echo "$DEV${INST} Transmit PAUSE Capable"
> fi
> if [ `ndd $DEV adv_pauseRX` = 0 ];then
> echo "$DEV${INST} Receive PAUSE Not Capable";else
> echo "$DEV${INST} Receive PAUSE Capable(default)"
> fi
> }
>
> do_set_ge()
> {
> DEV=$1
> INST=$2
> ndd -set $DEV instance ${INST}
> echo "Setting $DEV${INST} adv_1000autoneg_cap 0"
> ndd -set $DEV adv_1000autoneg_cap 0
> echo "Setting $DEV${INST} adv_1000fdx_cap 1"
> ndd -set $DEV adv_1000fdx_cap 1
> echo "Setting $DEV${INST} adv_1000hdx_cap 0"
> ndd -set $DEV adv_1000hdx_cap 0
> echo "Setting $DEV${INST} adv_pauseTX 0"
> ndd -set $DEV adv_pauseTX 0
> echo "Setting $DEV${INST} adv_pauseRX 1"
> ndd -set $DEV adv_pauseRX 1
> }
>
> # Workings
>
> case "$1" in
> 'check')
>
> ### Ge gigabit interface different
> GE_=`nawk '$NF == "\"ge\"" {print $2}' /etc/path_to_inst |
uniq`
> if [ "$GE_" != "" ];then
> for x in ${GE_};do
> do_check_ge /dev/ge $x
> done
>
> ANS=`ckyorn -p "Do you want to force all nics 1000 Mbps
> , Full-Duplex, Auto Negotiation off?~"`
> if [ $ANS = y ] || [ $ANS = Y ] || [ $ANS = YES ] || [
> $ANS = yes ];then
> echo "Setting Interfaces"
> for x in ${GE_};do
> do_set_ge /dev/ge $x
> done
> fi
> fi
> ;;
>
> 'start')
>
> ### Ge gigabit interface different
> GE_=`nawk '$NF == "\"ge\"" {print $2}' /etc/path_to_inst |
uniq`
> if [ "$GE_" != "" ];then
> for x in ${GE_};do
> do_set_ge /dev/ge $x
> done
> fi
>
> ;;
>
> *)
> echo "Usage: $0 { check | start }"
> exit 1
> esac
> exit 0
>
>
>
> Thanks
>
>
>
>
>
>
>
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
I must apologize i hadnt tested properly. the -y function takes 1 off
the year bit of date not the day. Sorry ill fix
Dave
--__--__--
Message: 4
Date: Thu, 12 Aug 2004 08:16:45 -0500
From: Ed Wilts <ewilts AT ewilts DOT org>
To: Dave Markham <dave.markham AT icl DOT net>
Cc: Mark.Donaldson AT cexp DOT com, veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Monitoring perfomance at the buffer level
Organization: (ewilts)
On Thu, Aug 12, 2004 at 01:59:46PM +0100, Dave Markham wrote:
> Ed Wilts wrote:
>
> >My overall issue is that although we have GigE connections between
many
> >hosts and the media servers, and trying to drive 8 SDLT220 drives in
an
> >L700, we almost never exceed 11MBs of traffic coming into the media
> >servers. It's like there's a cap there that we just haven't been able
to
> >remove.
>
> On answer to your capping have you check the gigabit ndd settings for
> the device ? This may help :-
It doesn't appear to be a network issue - I've down some FTP testing and
I have no problem cranking through 30MB/sec.
We avoid autonegotiation as much as we can and force everything to its
maximum speed at full duplex on both ends of the connections.
--
Ed Wilts, Mounds View, MN, USA
mailto:ewilts AT ewilts DOT org
--__--__--
Message: 5
Date: Thu, 12 Aug 2004 09:08:57 -0500
From: Ed Wilts <ewilts AT ewilts DOT org>
To: Dave Markham <dave.markham AT icl DOT net>
Cc: List Veritas List <veritas-bu AT mailman.eng.auburn DOT edu>
Subject: Re: [Veritas-bu] Monitoring perfomance at the buffer level
Organization: (ewilts)
On Thu, Aug 12, 2004 at 02:08:13PM +0100, Dave Markham wrote:
> I must apologize i hadnt tested properly. the -y function takes 1 off
> the year bit of date not the day. Sorry ill fix
[massive snipping of irrelevant messages]
Just add the following snippet to the script right after the policy=$1
line:
if [ -z $2 ]
then
today=`date +%m%d%y`
else
today=$2
fi
echo "Using /usr/openv/netbackup/logs/bptm/log.$today"
This lets you pass in an arbitrary date as the second parameter.
.../Ed
--
Ed Wilts, Mounds View, MN, USA
mailto:ewilts AT ewilts DOT org
--__--__--
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
End of Veritas-bu Digest
NOTE: This e-mail message is subject to the MTN Group disclaimer see
http://www.mtn.co.za/default.aspx?pid=34411
|
