Re: backup performance with db and log on a SAN

You are correct the at 3:1 compression you will not do better than
42MB/sec. Now compression does vary quite a bit. I have one customer who
gets 5:1 compression with some of his Oracle databases; the others only
get a bit more than 2:1. To stream at 100MB/s you would have to get
compression better than 6:1. The tape drive always writes at the same
speed 14MB/sec. If the data is does not compress than your throughput
will be 14MB/sec at 2:1 compression you will get 28MB/sec and so on.

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Eliza Lau
Sent: Monday, September 02, 2002 1:23 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: backup performance with db and log on a SAN

Daniel,


> 
> Hi Eliza
> 
> As I understand it, each "tape-HBA" has 3 3590E FC connnected to it.
The 
> two 21G database disks, are each connected to it's own HBA?

Yes, you are correct.

> 
> According to spec sheets, the 3590E FC could handle speed up to
100MB/s 
> with 3:1 compression. This means that with 3 drives, you could have up
to 
> 300MB/s. However, the HBA will only theoretically handle 125MB/s, or 
> 250MB/s with 2Gb FC.

We have 1Gbit FC adapters.  Are you sure that the 3590E FC drives can
stream at 100MB/s?   We were told that the tape drives are the
bottleneck
and not the adapters.

> 
> Thie means that the tape drives will not stream the data, cause the
data 
> flow will be queued(Some data to drive1, some data to drive2, some
data to 
> drive3 and so on). There should be some wait % on the drives.
> 
> What kind of a machine are you running? The smallest machine I know of

> that is sold today and could handle this amount of adapters, is the 
> P-Series 660, which has 3 or 6 PCI busses. Normally, you don't want 
> multiple FC HBA and Gb Ethernet adapters mixed in the same PCI bus, as

> theese are considered high-performance adapters that can utilize the
whole 
> bandwidth of the bus.

We have a P-Series 660 7026-6H1 with 12 PCI slots. The other adapters
being used
besides the 4 HBAs are a SCSIRaid adapter and a graphics monitor
adapter.
There are also 4 SCSI adapters left over from before the tape drives
were converted to FC. They should be taken out.

> 
> How many other adapters are installed in the machine? Gigabit
Ethernet, 
> 10/100 ethernet and so on.

The 10/100 ethernet has its own adapter, not in the PCI slots.

> 
> Best Regards
> 
> Daniel Sparrman
> -----------------------------------
> Daniel Sparrman
> Exist i Stockholm AB
> Propellervägen 6B
> 183 62 HÄGERNÄS
> Växel: 08 - 754 98 00
> Mobil: 070 - 399 27 51
> 
> 
> 
> 
> Eliza Lau <lau AT VTCAT.CC.VT DOT EDU>
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> 2002-09-02 12:19
> Please respond to "ADSM: Dist Stor Manager"
> 
>  
>         To:     ADSM-L AT VM.MARIST DOT EDU
>         cc: 
>         Subject:        Re: backup performance with db and log on a
SAN
> 
> 
> Thanks Daniel,
> 
> The host has 4 HBAs. Two attached to each of the two switches.  One
HBA is 
> 
> zoned for tape traffic and the other for disk traffic.  Only the db
and 
> log
> are on the Shark F20.  The disk stgpools are on attached SCSI disk
drives.
> The disks are 36G.  There are 8 LSSes.  Each LSS is striped across 8 
> drives
> and sliced into 21G partitions.  The db is on two of these 21G slices.
> Tpae library is a 3494 with 6 3590E FC drives.  All drives are
connected
> to the two switches, three each.
> 
> Come to think of it, the Shark only has 2 HBAs.  I have to verify
this, 
> since
> I am typing this from home.  But it only has to read from the Shark
and 
> write
> to tape through another port in the switch.
> 
> Eliza
> 
> > Hi
> > 
> > The large disks you are talking about, are you meaning large as
36GB, 
> 72GB 
> > an so on, or are you talking about LUN-sizes?
> > 
> > In a shark, you can have very large LUN:s, but they will consist of
a 
> > large number of smaller SSA-based hard drives. This means that you
will 
> > not have a performance impact on the disks.
> > 
> > Normally performance issues on TSM / SAN has to do with having disks
and 
> 
> > tapes on the same HBA. DB transactions is very randomly written, so
if 
> you 
> > for example are doing migration, TSM will write to both disks and 
> tape(DB 
> > transactions to disk, migration from disk, migration to tape). This
will 
> 
> > have a huge impact, as the HBA is arbitrated(which means only one
write 
> > can be done at a time). Also, doing backups and migration directly
to 
> > tape, assumes the ablility to write continous, sequential data. If
you 
> > have the DB on the same card, your clients, or the TSM server, won't
be 
> > able to stream data to the tapes, leading to poor performance.
> > 
> > Eliza, I'd suggest you use somekind of monitoring tool, like the 
> Storwatch 
> > specialist, to see throughput from/to disks and tape. I'm sure that
if 
> you 
> > separate the disks from the tape, you will see a performance
upgrade.
> > 
> > How many HBA:s do you have in your Shark?
> > 
> > Also, check to see that the logs, diskpook and database is not on
the 
> same 
> > volumes. This will also generate bad performance.
> > 
> > The best practice is to have db & log on one HBA, diskpool on one
HBA, 
> and 
> > tape drives on one or more HBA:s(depending on amount of tapes
drives). 
> > This is however recommended for large environments, with > 200
mid-size 
> > clients.
> > 
> > Best Regards
> > 
> > Daniel Sparrman
> > -----------------------------------
> > Daniel Sparrman
> > Exist i Stockholm AB
> > Propellervägen 6B
> > 183 62 HÄGERNÄS
> > Växel: 08 - 754 98 00
> > Mobil: 070 - 399 27 51
> > 
> > 
> > 
> > 
> > Remco Post <r.post AT SARA DOT NL>
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> > 2002-09-02 10:48
> > Please respond to "ADSM: Dist Stor Manager"
> > 
> > 
> >         To:     ADSM-L AT VM.MARIST DOT EDU
> >         cc: 
> >         Subject:        Re: backup performance with db and log on a
SAN
> > 
> > 
> > On zaterdag, augustus 31, 2002, at 05:18 , Eliza Lau wrote:
> > 
> > > I recently moved the 36G TSM database and 10G log from attached
SCSI
> > > disk
> > > drives to a SAN. Backing the db now takes twice as long as it used
to
> > > (from 40 minutes to 90 minutes).  The old
> > > attached disk drives are non-RAID and TSM mirrored.  The SAN
drives 
> are
> > > RAID-5 and TSM mirrored.  I know I have to pay a penalty for
writing 
> to
> > > RAID-5.  But considering the massive cache of the SAN it should
not be
> > > too bad.  In fact, performance of client backups hasn't suffered.
> > >
> > > However, the day after the move, I noticed that backup db ran for 
> twice
> > > as long.  It just doesn't make sense it will take a 100%
performance 
> hit
> > > from reading from RAID-5 disks.  Our performance guys looked at
the 
> sar
> > > data and didn't find any bottlenecks, no excessive iowait, paging,

> etc.
> > > The solution is to move the db and log
> > > back to where they were.  But now management says: "We purchased
this
> > > very expensive 2T IBM SAN and you are saying that you can't use
it."
> > > Meanwhile, our Oracle people happily report that they are seeing
> > > the performance of their applications enjoy a 10% increase.
> > >
> > > Has anyone put their db and log on a SAN and what is your
experience?
> > 
> > Yes we have. SAN is very bad for database performance. We used it as
a
> > temp storage space while we were reorganizing the ssa disks on our
TSM
> > server. The reason SAN attached storage gives poor performance is
the
> > size of the disks. You now have just a few large disks for your
> > database, while in the past you probably had a few more smaller
disks to
> > hold the same amount of data. Disk seek times have not kept pace
with
> > disk size, so while the disks are a lot bigger, they take about the
same
> > amount of time to find a block of data. Since almost each database
read
> > access requires a seek, more spindles give more bandwith and this
better
> > performance. Even worse in raid5, since now all disks must have read
the
> > block of data before the raid5 controller can return anything.
> > 
> > Raid5 will give you another performance hit, especially if you have
a
> > write-trough cache (or no cache at all) and not a write back cache.
> > 
> > You probably want to look into the cache-hit ratio on the raid5
> > controller, and see how you could improve that. Other than that, I
still
> > feel jbod is the way to go for databases. More spindles are more
> > better... ;)
> > 
> > > I have called it in to Tivoli support but has yet to get a
callback.
> > > Has anyone noticed that support is now very non-responsive?
> > >
> > 
> > 
> > 
> > > server; AIX 4.3.3,  TSM 4.2.1.15
> > >
> > > Thanks,
> > > Eliza Lau
> > > Virginia Tech Computing Center
> > > 1700 Pratt Drive
> > > Blacksburg, VA 24060
> > > lau AT vt DOT edu
> > >
> > ---
> > Met vriendelijke groeten,
> > 
> > Remco Post
> > 
> > SARA - Stichting Academisch Rekencentrum Amsterdam
http://www.sara.nl
> > High Performance Computing  Tel. +31 20 592 8008    Fax. +31 20 668
3167
> > PGP keys at http://home.sara.nl/~remco/keys.asc
> > 
> > "I really didn't foresee the Internet. But then, neither did the 
> computer
> > industry. Not that that tells us very much of course - the computer
> > industry
> > didn't even foresee that the century was going to end." -- Douglas
Adams
> > 
>