ADSM-L

Re: Why do database restores take so long?

2005-01-25 13:45:16
Subject: Re: Why do database restores take so long?
From: Tab Trepagnier <Tab.Trepagnier AT LAITRAM DOT COM>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Tue, 25 Jan 2005 12:44:43 -0600
Wanda,

Since that DB restore was almost a week ago, I don't have any live data
for you.  But *if I recall correctly*, each disk was about 50-60% busy.

Also, the DB and log re-mirrored themselves automatically after the
restore, yet I thought I saw the mirror volumes being written in parallel.
 Note that the DB and log are TSM-mirrored; the disks are not mirrored to
each other.  The TSM re-mirror operations showed 25-50 MB/s according to
topas; indeed, the entire mirror-sync operation took only a few minutes to
complete.

I've restored data from the same DLT and it comes in at its usual average
of 8 MB/s per drive (file data rate; DB backup rate is 11 MB/s), so I
doubt the DLT is the culprit.  Even during the DB restore, I saw rare
spikes to 6 MB/s.  Since the restore took 3-1/4 hours vs. a 50-minute
backup means the average restore rate was 11 x 50 / 195 = 2.8 MB/s,
despite periods when data was being restored as slowly as 300 KB/s.

Thanks for the feedback.

Tab





"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 01/25/2005
12:23:05 PM:

> What does topas report during the restore about disk device busy?
>
> Some things  I would try, if I had enough of a test environment to try
> it, is a DB restore from a disk backup instead of tape; if it takes
> longer from tape, then the DLT is part of the problem.
>
> If topas reports a lot of disk busy during the restore, I'd try breaking
> the mirror before doing the restore, and reestablishing the mirror
> afterwards.  Cutting the I/O in half has gotta help somewhere....
>
> Wanda Prather
> "I/O, I/O, It's all about I/O"  -(me)
>
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Tab Trepagnier
> Sent: Friday, January 21, 2005 6:16 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: Why do database restores take so long?
>
>
> Dave,
>
> Our database disk system consists of four 18 GB SCSI disks in a split
> (dual-bus) disk pod connected to a dual-channel LVDS adapter.  The
> Recovery Log in on another pair of such disks.  One set of disks on one
> bus holds the primary database and log volumes; two DB and one log disk.
> Each DB/log volume is mirrored to an identical volume/disk on the other
> bus.  So the mirror read/writes occur on independent buses downstream of
> the PCI slot.  The database consists of four volumes per disk of about 4
> GB each.  All database and log volumes are defined on JFS2 file systems.
> The LVDS adapter is the IBM 64-bit dual-channel adapter called "PCI DUAL
> CHANNEL ULTRA3 SCSI ADAPTER", Part Number 09P2544.
> The disks are "16 Bit LVD SCSI Disk Drive (18200 MB)", Machine Type and
> Model = ST318305LC which I *think* is 15K rpm.
> The server is a 2-way 6H1 64-bit 600 MHz with 4 GB RAM, tuned for just
> about zero paging. TSM DB buffer pool is 1 GB.
> Like I said, when backing up to our HP 4/40 DLT via HVD SCSI, topas
> reports a consistent 11 MB/s, such that a typical backup takes just
> about
> 45-50 minutes including the tape mount.
> The restore took slightly over three hours; and the 3:1 ratio is not out
> of line based on other responses.
>
> Thanks.
>
> Tab
>
>
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 01/20/2005
> 11:27:02 PM:
>
> > Can you provide a little more information? I would like to know:
> >
> >          1. What kind of disk subsystem are you restoring the data to?
> >          2. How many disks, and what kind of disks are they?
> >          3. How is the database laid out database volume wise?
> >          4. Does the disk have write cache behind it?
> >          5.  What kind of controller do you have for the disk
> subsystem?
> >
> > At 07:23 PM 1/20/2005 -0600, you wrote:
> > >TSM 5.1.9.0 on AIX 5.2 ML4; DLT8000 media on SCSI
> > >
> > >Why do TSM database restores take so much longer than the backups?
> Our
> > >system backs up our 24 GB database to DLT in about 50 minutes with a
> > >sustained rate of 11 MB/s reported by topas.
> > >
> > >Tonight I'm performing a database restore.  It is occurring at an
> average
> > >rate of less than 1 MB/s with rare peaks no higher than 6 MBs.  At
> times
> > >the rate is as low as 300 KB/s.
> > >
> > >I realize the system is writing data to disk rather than reading so
> its
> > >not doing as much caching, but these are new disks and I've seen them
> > >stream write at better than 25 MB/s on the inner edge and as fast as
> 50
> > >MB/s on the outer edge.  So I doubt the disk is the problem.
> > >
> > >This restore has been running for almost 2-1/2 hours now and is about
> 90%
> > >complete.
> > >
> > >This is the fourth or fifth time I've restored the TSM database in
> the
> > >seven years I've operated the system.  I have never seen database
> restores
> > >exceed 1/3 the backup rate - sustained - on two different servers,
> three
> > >different media and going back to ADSM 3.1.
> > >
> > >Just curious.
> > >
> > >Tab Trepagnier
> > >TSM Administrator
> > >Laitram, L.L.C.
> >
> > Dave Canan
> > TSM Performance
> > IBM Advanced Technical Support
> > ddcanan AT us.ibm DOT com

<Prev in Thread] Current Thread [Next in Thread>