TSM 6.3.4 DB backup gets slower

[brwolfe]

We have a problem that has been going on for three weeks now. Our DB backup of 1.2 tb is taking 4-6 hours now. It use to take 90 minutes in all the previous days before that. It's like some one turns a switch off about 500gig into the backup. the configuration hasn't changed, but basically be have a 750g Fusion I/O card with a bulk of the DB on and the remaining DB volumes are on SAS disk (400g). I know this is not optimal to begin with, but it was working fine.
The DB backup gets written to some of the same SAS disk in a subdirectory of where the FILE storage pools exist.
So the DB backup is all getting written to SAS, but it is being read from the FIO and SAS.
Client backups, expiration, reclamation all seem to be running at the same speed they were processing previous.
We have updated firmware on the FIO, HP arrays and HP server, upgraded RHEL to 2.6.18-348.12.1.el5 to no avail.
We have run the dbbackup to another file system all on it's own with the same results.
One note that baffles us also, the day after the firmware upgrade to the HP arrays(booting everything), we saw the DB backup return to 90 minutes for one day, then the next day it was back to 4-5 hours again.
We now have opened an HP ticket after the firmware upgrade, they wouldn't talk to us before doing that.

Regards;

Brian

 

[moon-buddy]

Did the speed issue surfaced after the firmware update? If so, the firmware is buggy and restricts I/O traffic at a certain level when disk has been shared with other files/data.

Can you move the DB on its own I/O and separate disks? What happens to the speed if you?

 

[brwolfe]

Turns out that after we got HP involved they said we had three disks going bad in three different shelfs, and after we replaced those, everything is now fine.
We are still investigating why our normal monitoring with the HP utilities is not picking these disk failures up.

Regards;

Brian