I started with a calculation like that, but I also incorporate destaging speed. I've never used anything higher than 16 concurrent writes, and that is for incremental backups. With incremental backups, I don’t care as much because even if the destaging speed is poor, the amount of data is so small that it will still be written to tape quickly.

 

However for full backups, I’ve found that the more streams you have, the slower your destaging will be.  I created the graphic below to help explain it to management.  The issue isn’t fragmentation as much as it is sequential blocks of data. If you were to write one sequential stream to a raid group then reading it back would be a sequential operation.  And in my case, SATA disks read sequential data very quickly. But as soon as you multistream, the data “fragments” or becomes non-sequential (really, the file sizes are so large every file is fragmented).

 

Image removed due to size limitation.

 

So to find your “sweet” spot you will have to test.  I started by building my environment and then testing with a single stream, from client to disk, then disk to tape.  This was my best case scenario, and I was able to drive LTO3 with it.

 

My configuration: Dell MD1000 w/ 12 500GB DATA disks in a Raid 5. PERC 5/e card with Adaptive Read Ahead and Write Back enabled. GPT Partition table w/ 64KB block sizes.

 

The 64KB block size and adaptive read ahead settings were key to my performance. I was going to test the same hardware with a linux ext2 file system and 4KB block sizes for comparison, but I never got the chance.

 

Once you have a baseline you can run more streams. You also need to consider job length.  For example, I backup an NFS server with 16 streams to a DSSU that allows all 16 streams because 12 of those streams take less than an hour each.  I’ve also tested with 1TB SATA Disks and 300GB 15K SAS Disks.  I still rely on the 500GB SATA as my work horse because I can drive LTO3 if I need to. Most of my DSSUs are 8 streams and push 70MB/sec to tape.

 

-Jonathan

 

 

 

-----Original Message-----
From: Victor Engle [mailto:victor.engle AT gmail DOT com]
Sent: Monday, April 26, 2010 3:39 PM
To: Martin, Jonathan
Cc: veritas-bu AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Architectural question (staging)

 

Thanks Martin. Good info. What criteria do you use to determine the

number of concurrent jobs for a DSSU? Is it reasonable to determine

the concurrent DSSU sessions based on the speed of the clients? For

example, If I have a disk to which I can write 100MB/s and clients

that can write 5MB/s then I would use 20 concurrent sessions?

 

Thanks all for your responses!

 

Vic

 

 

On Mon, Apr 26, 2010 at 10:14 AM, Martin, Jonathan

<JMARTI05 AT intersil DOT com> wrote:

> We use Disk Stage Storage Units (DSSU) almost exclusively for our backups.

> As someone has already mentioned, you can stream many slow clients to a DSSU

> without impacting your speed significantly. To do this with tape you would

> have to use multiplexing, which is a real performance killer come restore

> time.  The DSSUs essentially allow you to “multiplex” to disk, then single

> stream to tape. Regarding Ed’s speed issue below, I’ve got data here that

> directly correlates number of concurrent streams written to a DSSU with the

> performance to tape.  You’ll need to balance this when setting this on the

> DSSU, but we get away with 8-12 streams on 14x500GB SATA Disks in a Raid-5

> and still drive LTO3.

> 

> 

> 

> Here we were also able to purchase a much smaller tape library.  I replaced

> a library with 8 x SDLT220 drives with a library with 2 x LTO3 drives (and

> disk storage.)  The disk storage is far more reliable than tapes and

> libraries. As Ed noted below you also benefit from having your backups on

> disk which makes restores very snappy.

> 

> 

> 

> -Jonathan

> 

> 

> 

> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu

> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Ed Wilts

> Sent: Sunday, April 25, 2010 3:45 PM

> To: Victor Engle

> Cc: veritas-bu AT mailman.eng.auburn DOT edu

> Subject: Re: [Veritas-bu] Architectural question (staging)

> 

> 

> 

> On Sun, Apr 25, 2010 at 2:14 PM, Victor Engle <victor.engle AT gmail DOT com>

> wrote:

> 

> Just wanted to get some opinions about whether disk staging units are

> worthwhile. My backup server has two BasicDisk staging units with the

> storage units configured such that the data goes to disk and is then

> moved to tape. I have a tape library with four LTO-3 drives connected

> via FC. So what I'm wondering is, since the LTO drives are reasonably

> fast, and since I'm writing the data ultimately to tape anyway, would

> it be better to just write directly to tape. The disk is just old

> fashioned spinning disk with no de-duplication so there are

> operational costs for the disks. All tape and disk storage units are

> local to the backup server. I'm thinking it would be better to add LTO

> drives and eliminate the disk for now and maybe later add a

> de-duplicating disk unit.

> 

> We worked with disk staging units for at least a year before we mostly gave

> up.  The biggest challenge we ran into was that destaging was too slow. Even

> though we proved to Symantec that we could read from those disk drives at

> over 100MB/sec, we could never destage even half that fast.  We had an open

> case with Symantec for a VERY long time before we agreed that it wasn't

> going to get fixed.

> 

> Under what circumstances does it make sense to stage data on disk. I

> would appreciate hearing what your thoughts and experiences are with

> regard to disk staging.

> 

> There are times when DSSUs make sense.  1.  If you don't have a tape drive

> free but want to do a backup anyway - we still use DSSUs for things like

> small backups of Oracle archive logs. 2.  If you need to throttle your

> backups, especially across things like a bunch of virtual servers on the

> same physical server.  NBU only allows you to set the maximum jobs per

> client name, not per client.  DSSUs make an acceptable choke point for

> clusters.  3.  If you have small backups, but don't have a lot of them at

> once, multiplexing may not buy you enough performance boosts.  Use DSSUs to

> write those little jobs to disk and then destage them at once.

> 

> If you currently multiplex, realize that your restores are going to be

> slower than if you don't multiplex.  All tapes created from a DSSU destage

> are non-multiplexed so your restores can go faster.

> 

> DSSUs also give you a staging area for restores.  If your tapes go offsite,

> you may still be able to do a restore from the staging unit the next day (or

> longer) depending on how big your stagig units are.  NBU is smart enough to

> realize that if the same data is on both disk and tape and you kick off a

> restore, the restore will automatically come from disk.

> 

> In general, I'd say that there is a place for DSSUs but it's not the great

> benefit we thought it was going to be.

> 

>    .../Ed

> 

> _______________________________________________

> Veritas-bu maillist  -  Veritas-bu AT mailman.eng.auburn DOT edu

> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu

> 

>