Recovery log getting pinned nightly, TSM 5.5.6

[dangel42]

Pretty much at a loss here. I think I've tried just about everything but obviously haven't hit the nail on the head yet.

TSM 5.5.6 on AIX > upgrade to 6.x is planned for this year
DB size is ~100GB running in normal mode > rollforward was causing DB backups to run constantly

- show logpinned does state that I have dirty buffer pool pages
- trying to kick off occasional incr db backups has no effect as they can't even get started the system is so locked up
- running the flush command does seem to do it's job, but obviously not well enough to avoid the system from crashing
- consultant says it's possible that I don't have enough network bandwidth available on the server itself(two bonded 1Gb links) > another dual-port card is on it's way to make it 4
- I've split up the backup jobs so that not all of my servers are backed up every night...has had no lasting effect

Any ideas would be greatly appreciated.

[moon-buddy]

How many nodes are you backing up on this TSM server?

One main cause of the logs getting full is a node or nodes has so many files to back up and the primary pool (I hope you have disk pools) is full (cache enabled taking time to migrate) or not big enough to hold the backup.

If you have disk pools, and you are sure it is really big to hold the backup data, check for performance issues. What does AIX 'errpt' show? If your disk pool is on SAN, checkout the LUNs.

If you have a lot of nodes to backup you may want to split your TSM server.

[dangel42]

500+ nodes

Primary pool for the great majority of these systems is disk and it does have room for a night's worth of incrementals at least when split up across different nights.

Disk pool is on a SAN, errpt does mention occasionally "PATH HAS FAILED" for some of the LUNs.

[moon-buddy]

500+ nodes

Primary pool for the great majority of these systems is disk and it does have room for a night's worth of incrementals at least when split up across different nights.

500 is what I still consider a low medium environment. I had an environment with 480 nodes and I can complete a nights' backup within the 10 hour window. I only had 7 3592 tape drives, 8 TB of disk pool and 6 HBA split between the SAN and tape drives - 2 for SAN and 4 for tape drives. The HBA setup is far from ideal as it is required to have 1 HBA per two drives. The environment is a p520 (4 CPU) with 4 GB of RAM, 2 fiber attached TCP/IP ports at 1 GB and an internal disk of 180 GB for the TSM DB and OS. AIX is 6.1 and TSM version is 5.5.6

Disk pool is on a SAN, errpt does mention occasionally "PATH HAS FAILED" for some of the LUNs.

This seems to be your problem. Troubleshoot the path failed errors. I have encountered this and had the log pinned for two days.

6 HBA split between the SAN and tape drives - 2 for SAN and 4 for tape drives.

Correction: I had only 4 HBAs: 2 were for SAN and 2 for the tape drives.

[dangel42]

I'll check out the path failed errors, thanks.

[dangel42]

Do you recall what the resolution was for your path failed errors?

[moon-buddy]

That was a SAN fabric issue.

See if you might have a disk issue on the SAN.

[dangel42]

Ended up using the third post from this link...

http://groups.google.com/group/comp....700914d47e34a5

Hopefully this helps us.

[dangel42]

Fixing the failed paths has not fixed my problem, unfortunately.

[sgabriel62]

Here is an idea as we have had a bit of the same problem with our TSM DB running on local disk and we then moved it to the SAN and disabled mirroring.
Since your DB is on the SAN already..... have your SAN Administrator create new LUNs on a different channel. From here - one DBvolume at a time - mirror that volume to the new LUN.
Once sync'd - disable the primary volume and force the movement to the new LUN - Run a DB backup. Repeat until you find your LUN(disk) with the dirty pages.
Reformat that LUN and bring that back into the picture as your DBcopy. Or leave mirroring off as the SAN will take over from there in regards to disk managment - failed disk hot spares all that jazz.

Next - on your HBAs - if they are dual ported - or quads - run one HBA for disk - One HBA for tape - leaving multi-pathing on the HBA/port switch. This also helps in troubleshooting failing switch ports. Quads are usually best as two ports are readily available at all times for hot fixes/swaps.

Hope this helps

[dangel42]

Actually the DB is mirrored on local disk. Primary pool for the majority of the systems is SAN-based.

[moon-buddy]

Fixing the failed paths has not fixed my problem, unfortunately.

Have you rebooted the TSM server? Have you checked the local disk that houses the TSM DB and LOG volumes? Have you determined which node is sending large data and has isolated that node?

[dangel42]

Server has not been rebooted recently. Local disk appears to be fine, however my AIX knowledge is a bit limited. I do know the nodes that send large data(mostly Exchange and SQL flat files as we don't use TDP agents).

[moon-buddy]

Server has not been rebooted recently. Local disk appears to be fine, however my AIX knowledge is a bit limited. I do know the nodes that send large data(mostly Exchange and SQL flat files as we don't use TDP agents).

When these nodes send large data, what happens to the disk pool that receives the data? Do these get full and migration starts over to the next (presumably tape) pool?

Can you have an AIX admin check your system?

[dangel42]

The largest of the bunch goes straight to tape(exchange), the others go to a diskpool large enough to handle a night's worth of backups...a scheduled migration to tape is then run later on outside of the backup window.

We don't have an AIX admin on staff, but I'll see what I can do. Our consultant's firm has AIX admins.

One other thing to note, I currently have 14 DB volumes...would it help to attempt to reduce that number and/or move those DB volumes off of local disk to the SAN, if possible?

[moon-buddy]

The largest of the bunch goes straight to tape(exchange), the others go to a diskpool large enough to handle a night's worth of backups...a scheduled migration to tape is then run later on outside of the backup window.

This is where the log pin may be happening - it takes a long time to backup Exchaage directly to disk. Can you not move this to disk pool first?

We don't have an AIX admin on staff, but I'll see what I can do. Our consultant's firm has AIX admins.

One other thing to note, I currently have 14 DB volumes...would it help to attempt to reduce that number and/or move those DB volumes off of local disk to the SAN, if possible?

How big are the volumes? Generally, the number does not not affect or have any direct effect with log pinning unless one volume goes 'wild'.

[rmazzon]

If you are not sure it is the Exchange client, or just want proof, consider a shell script that will perform a "sh logpin" command every 5 minutes or so. That script should also show the log utilization since the log will be pinned often, but you only need to concern yourself when the %util starts to rise quickly.

Doing so in my environment has over the years helped to quickly pinpoint problem nodes or processes. Here is the guts of my script, suitably adjusted. It actually queries all servers in our environment, hence the outer do loop

USER=<admin account>
PASSWD=<Admin Password>
HISTFILE=<path and file name of output file>
cat SERVERLIST.FILE | while read INST
do
dsmadmc -id=${USER} -pa=${PASSWD} -dataonly=yes -se=${INST} \
"show logpin" | grep "Type=Node" | sed 's/Session //' | \
sed 's/Type=Node,//' | sed 's/Id=//' | tr -d ":" | read LINE2
SESS=$(echo $LINE2 | awk '{print $2}')
NODE=$(echo $LINE2 | awk '{print $3}')
## Note the spaces in the tr command is a SPACE and a TAB
UTIL=$(dsmadmc -id=${USER} -pa=${PASSWD} -dataonly=yes -tabdelim -se=${INST} \
"select LOG_POOL_PCT_UTIL from log" | tr -d " ")
if [[ -n ${NODE} ]]
then
# tab delimited output, only if a node was actually discerned
echo "$INST $(date '+%Y%m%d') $(date '+%H:%M') $SESS $NODE $UTIL" >> $HISTFILE
fi
dsmadmc -id=${USER} -pa=${PASSWD} -dataonly=yes -se=${INST} \
"show logpin" | grep -E "descr=|procNum" | paste - - | cut -d"=" -f2,5 \
| cut -d"," -f1,2 | sed 's/status=//' | while read LINE3
do
if [[ -n ${LINE3} ]]
then
## Note the spaces in the sed command is a TAB, and the other output is tab delim
echo "$INST $(date '+%Y%m%d') $(date '+%H:%M') $LINE3 $UTIL" | sed 's/, / /' >> $HISTFILE
fi
done
done

[dangel42]

This is where the log pin may be happening - it takes a long time to backup Exchaage directly to disk. Can you not move this to disk pool first?

Unfortunately no due to the sheer size. Also, it's not being run every night and we're still seeing the pinning.

How big are the volumes? Generally, the number does not not affect or have any direct effect with log pinning unless one volume goes 'wild'.

All but two are 10 GB.

[dangel42]

Thanks rmazzon, I'll give it a shot.

[moon-buddy]

Unfortunately no due to the sheer size. Also, it's not being run every night and we're still seeing the pinning.
All but two are 10 GB.

Ok, so the Exchange backups are not run daily.

1) Have you checked the health of your network?
2) Ask an AIX admin to checkout the TSM host server. Check for local disk issues with the disk itself and/or the SCSI interface. Again, errrpt will assist on this
3) Look for a node that stays on wait during backup and see what is causing this to wait

I presume you had checked out the SAN environment based on your last post. Have you done benchmarks on the SAN?

[dangel42]

Ok, so the Exchange backups are not run daily.

1) Have you checked the health of your network?
2) Ask an AIX admin to checkout he TSM host server. Check for local disk issues with the disk itself and/or the SCSI interface. Again, errrpt will assist on this
3) Look for a node that stays on wait during backup and see what is causing this to wait

I presume you had checked out the SAN environment based on your last post. Have you done benchmarks on the SAN?

1 > Adding more network bandwidth to the server today/tomorrow
2 > Will do. Just noticed that the last entry in the errpt is "ERROR LOGGING TURNED OFF"...not sure why that would be. Know how to re-enable that?
3 > Will do.

Correct, SAN is checking out fine at the moment. Having them check for errors again just to be safe.

[moon-buddy]

1 > 2> Will do. Just noticed that the last entry in the errpt is "ERROR LOGGING TURNED OFF"...not sure why that would be. Know how to re-enable that?

How are you accessing the ERRPT? Root ERRPT messages are different than other user's messages. ERRPT is always ON. This must be from something else. Error logging turned OFF might from be an application or from TSM.

[dangel42]

How are you accessing the ERRPT? Root ERRPT messages are different than other user's messages. ERRPT is always ON. This must be from something else. Error logging turned OFF might from be an application or from TSM.

Running the command 'errpt' as root. And I have no new entries since 6/13.

[moon-buddy]

Running the command 'errpt' as root. And I have no new entries since 6/13.

Look here for starting ERROR Logging: http://publib.boulder.ibm.com/infoce...errlogtsks.htm