Veritas-bu

Re: [Veritas-bu] Destaging going slow

2010-06-25 11:49:23
Subject: Re: [Veritas-bu] Destaging going slow
From: "Martin, Jonathan" <JMARTI05 AT intersil DOT com>
To: "WALLEBROEK Bart" <Bart.WALLEBROEK AT swift DOT com>, <veritas-bu AT mailman.eng.auburn DOT edu>
Date: Fri, 25 Jun 2010 11:49:13 -0400
Great.  I would say that is your "baseline", and that is just about what I get 
out of 15x1TB SATA disks in a Raid-5 at 8 streams. What kind of Raid controller 
are you using?  What kind of block sizes on that raid volume? What kind, and 
how many physical disks are in your raid set? Is this a SAN or DAS?  If it is a 
SAN, do you have other things on the same raid group? Are you using / what are 
your values in the touch files? Have you looked at your disk counters while the 
destaging is running? (Logical and Physical disk counters for individual 
disks.) Are you seeing high levels of disk queuing? Does your throughput to 
tape match the read/sec? 

35-40 MB/sec is not ideal, but I could consider it acceptable if you were using 
huge SATA disks. If you think you are not getting the performance you deserve 
from the hardware then I would suggest digging.  It took me two weeks to find 
the optimal settings that I run now.  In the mean time, don't listen to Ed.  
There are plenty of users on this forum who have DSSUs running just fine. I for 
one have something like 12 sites running DSSUs without issue.

-J

-----Original Message-----
From: WALLEBROEK Bart [mailto:Bart.WALLEBROEK AT swift DOT com] 
Sent: Thursday, June 24, 2010 11:19 PM
To: Martin, Jonathan; veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Destaging going slow

Martin,

Even when we fully format the DSSU disk and we then run 1 backup job to this 
disk and directly afterwards we duplicate it to tape we get these speeds 
(35-40MB/sec).  So at that time no fragmentation at all is involved.

Best Regards,
Bart WALLEBROEK
Backup Admin & Systems & Applications Management & Support Specialist 
Enterprise Applications Delivery - Infrastructure Management

-----Original Message-----
From: Martin, Jonathan [mailto:JMARTI05 AT intersil DOT com] 
Sent: Thursday, June 24, 2010 5:34 PM
To: WALLEBROEK Bart; veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Destaging going slow

I regularly push LTO3 to 100+MB/sec. It just depends on your configuration 
(hardware and software.) For example, I've got a 4TB Oracle DB that writes to a 
DSSU at 40MB/sec then destages to tape at 100+MB/sec.  I understand that you 
are running 1 deduplication job at a time to a single tape drive.  How many 
concurrent backups are you writing to the DSSU? That is where the 
"fragmentation" comes from. It's not so much fragmentation, as non-contiguous 
files that cause the majority of my speed issues.

-Jonathan

-----Original Message-----
From: WALLEBROEK Bart [mailto:Bart.WALLEBROEK AT swift DOT com] 
Sent: Thursday, June 24, 2010 2:46 AM
To: Martin, Jonathan; veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Destaging going slow

Jonathan,

Actually we already tried out different sets of block sizes on the DSSU.  32 
seems to be the fastest we can find.  We do not use compression and there is no 
fragmentation whatsoever (newly formatted disk).  Only 1 duplication is done at 
a time (only 1 tape drive attached to the DSSU Media Server) while no backups 
are pointing to the DSSU while we perform the destaging.

But so far no luck.  Could this be due to the backend disk system where the 
DSSU disk is located on ?  Although we see some impressive speeds when backing 
up files from DSSU to that same tape drive.  We use an HP XP 20000 (Hitachi).

Best Regards,
Bart WALLEBROEK
Backup Admin & Systems & Applications Management & Support Specialist 
Enterprise Applications Delivery - Infrastructure Management


-----Original Message-----
From: Martin, Jonathan [mailto:JMARTI05 AT intersil DOT com] 
Sent: Wednesday, June 23, 2010 3:59 PM
To: WALLEBROEK Bart; veritas-bu AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Destaging going slow

Format your DSSUs with a 64k block size.
Use 64K block size on raid group.
Disable compression (if you can't use 64k block size.)
Lower the number of concurrent writes to the DSSU.
Occasionally "clear" the DSSU to relieve fragmentation.

Clear the DSSU, write a single backup to the disk, then stream the backup to 
tape. Whatever performance you get from that single stream is the best you are 
going to get from that configuration.  Any additional concurrent writes will 
likely slow your performance.

We had a lively discussion on this topic a few weeks back. Here's a link to a 
post I made about fragmentation and wet blankets.

http://www.adsm.org/lists/html/Veritas-bu/2010-05/msg00029.html

-Jonathan

-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu 
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of 
WALLEBROEK Bart
Sent: Wednesday, June 23, 2010 9:23 AM
To: veritas-bu AT mailman.eng.auburn DOT edu
Subject: [Veritas-bu] Destaging going slow

We have a couple of windows (dedicated) Media Servers that act as NetBackup 
DSSU (Disk Staging Storage Unit) servers with each having from 4 to 16 TB of 
SAN disks.  Backing up to them is no issue and goes up to the limit of the 
network connected clients.  Destaging to tape (LTO4) however is slow (20 - 35 
MB/sec).

We have been in contact with Symantec where they advised us to lower the 
fragment size, change the data buffers size and numbers but we seem to get 
stuck to the current speed.  Backing up data from this DSSU disk to the tape 
drives goes up to physical tape speed limitation.

Does anyone have an idea where to look further to get to a reasonable speed ?

We currently are testing a Solaris DSSU server but we do not have numbers for 
that one.

Best Regards,
Bart WALLEBROEK
Backup Admin & Systems & Applications Management & Support Specialist 
Enterprise Applications Delivery - Infrastructure Management
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu