Networker

Re: [Networker] Query in Staging from adv_file

2008-08-20 07:33:00
Subject: Re: [Networker] Query in Staging from adv_file
From: Francis Swasey <Frank.Swasey AT UVM DOT EDU>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Wed, 20 Aug 2008 07:29:33 -0400
Curtis,

To be clear, I don't hate VTL's either. I just haven't found one yet that I feel comfortable using here.

I don't think I have a small environment. I have a Qualstar XLS tape library with 12 LTO4 tape drives. I have a NetWorker Server and 8 storage nodes. I have 238 clients defined (about 225 active). We run monthly fulls, weekly differentials, and daily incrementals. Our monthly backup volume is about 150 TB. Our nightly incrementals are about 2 TB, our weekend full/differential cycle is about 25 TB per weekend.

I have three NexSAN SATABeasts. Each SATABeast is attached to a single storage node and holds four AFTDs. Each of those storage nodes has two LTO4 tape drives attached.

I have three DataDomain DDRs. Each DDR is attached to at least one storage node and houses from one to three AFTDs. Each of these storage nodes that needs to do cloning has one LTO4 tape drive attached.

Two of the storage nodes each have two LTO4 tape drives and no AFTDs -- they are for those poor clients that can't or refuse to spend money on a disk buffer, so I want those tape drives to be worn out first so they can pay for them.

I have scripted the weekly clones so a list of the SSID's on each AFTD that needs to be cloned is generated and I then issue nsrclone commands for as many AFTD's as I have tape drives and that way, I get the cloning done in less than 24 hours from the AFTDs. (at least from the SATABeast AFTDs I do. The DataDomain devices are old and they are not capable of keeping up with our new LTO4 drives -- they were great before we upgraded from the AIT3 drives). Replacing the almost EOSL DataDomains with something new is my next priority.... (shhh... don't tell my manager... he keeps telling me I'm out of money for backup).

As for staging savesets off the AFTDs... Those AFTDs that are set up to receive incrementals -- yes, that staging definition runs 24x7. However, for the AFTDs that are set up to receive full saves -- NO! I turn that staging definition on after the clones finish and turn it off again before the weekend full saves start. I still have enough head-room that works. My reason for doing that is to prevent contention for the tape drives when I need/want them for cloning.

I have also defined the read node on the jukebox definition to be one of the two storage nodes that does not have any AFTDs attached -- to further attempt to reduce the contention for the tape drives.

To be honest, I don't hate VTLs either. I wouldn't have spent the month working with the sales folks last December if I did. I am convinced that management would be easier (well -- at least I believe there would be less manual intervention required) if I were using a VTL instead of the AFTDs. However, since I was unable to make the VTL that I believed was the best perform as I needed, I chose the complex cloning/staging scripting of the AFTDs over a poor performing VTL. I will one day go back and re-evaluate the VTLs -- but right now, with NW 7.4 looming over me and the new VTL licensing requirements that make paying for AFTD space look pleasant..... I'm going to wait a while longer.

To address your summary;

1) I am assuming by "destage" you mean moving the savesets from the VTL to real tape. Yes, some of my AFTDs do that every day. Others do not. It depends on their usage.

2) I agree whole heartedly. If I bought a device today that couldn't keep an LTO4 tape drive streaming -- I made a mistake! I didn't (whew).

3) Yeah, I'll concede that point. However, I do see more recovers than you are expecting. Because NetWorker really doesn't keep tabs (it counts staging and cloning operations as recovers) -- I don't know how much more and how often being able to have multiple restores reading from the same AFTD (granted they still have to be different savesets) actually is a benefit.

And -- yes, healthy discussion is always wonderful. I've even been known to change my mind a time or two.

Frank




On 8/19/08 11:27 PM, Curtis Preston wrote:
If the VTLs I test could only supply data at 40 MB/s, I wouldn't recommend them 
either.  The ability of a disk device to stream to tape is job #1, AFAIC.  Just 
remember that not all VTLs are created equal.

Don't get me wrong.  I'm not an AFTD hater.  I think they're fine for smaller 
shops that want an easy solution.  But when I get to a larger shop, I need the 
ability to make things go more concurrently, and I think it's possible to do 
that with a VTL and VERY difficult to impossible to do it with an AFTD.

VTLs aren't the end-all-be-all, and some of them are crap.  (The same is true 
of the disk systems behind a lot of AFTDs.)  They're also more management in a 
SMALL NW environment than AFTDs.  But in a large environment, where 
staging/cloning performance is king, I'll take a VTL any day.  Yes, I'll be 
doing some scripting, but at least I can clone/destage multiple backups at the 
same time.  That's NOT the case with an AFTD unless you make a bunch of them 
and that creates other management hassles, and decreases disk utilization.

I think the concurrent restore thing is a red herring.  Sure, you can't do two 
restores at once from a tape, but:

1. Most people don't do more than one restore a month, let alone more than one 
a day, let ALONE two at the same time.
2. Add to that the odds that the two restores you do at the same time would 
need the same tape at the same time. (Multiplexing would increase the chances, 
but one should not multiplex to virtual tapes.)
3. If you want better odds, just use smaller virtual tapes.  It costs you 
nothing, adds no management issues (like more AFTDs do), and significantly 
decreases the chance that two restores would be grabbing for the same tape.

So, to summarize:
1. I think it's easier to destage multiple backups simultaneously from a VTL 
than it is to do it from an AFTD, and that's something you do every night.
2. Performance counts.  If your device, regardless of type, can't stream your 
tape drive type, you bought the wrong one.
3. Sure I can do concurrent restores from the same "tape" with AFTDs and I 
can't with VTLs, but I don't think that happens much and if it does I can address it with 
smaller virtual tape sizes.

Isn't healthy discussion fun?


Curtis Preston | VP Data Protection GlassHouse Technologies, Inc. T: +1 760 710 2004 | C: +1 760 419 5838 | F: F: +1 760 710 2009 cpreston AT glasshouse DOT com | www.glasshouse.com
Infrastructure :: Optimized

-----Original Message-----
From: EMC NetWorker discussion [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On 
Behalf Of Francis Swasey
Sent: Tuesday, August 19, 2008 6:13 PM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: Re: [Networker] Query in Staging from adv_file

With VTL, Networker limits you to one operation at a time per virtual Tape. I've heard tales that if you are lucky enough to issue two recover commands at the exact same instant, that NetWorker can be tricked into doing two restores from the same tape at once.

With the AFTD, you can do multiple concurrent writes and either multiple restores or a stage or a clone at the same time per AFTD. In my environment, I've tested a VTL and it was a lousy performer. I wasted an entire month and had to put up with a lot of pressure from the sales people who really wanted me to spend a small fortune on their appliance when they couldn't make it perform at even one third of the I/O performance of the AFTD on a NexSAN SATABeast.

If I had to do my weekly clones with a VTL that was flat out retrieving data at 35-38MB/second to write to LTO4 tape drives that can do 120MB/second -- I'd be looking for a new job! Thankfully, on my big fileservers with the big savesets, I'm pushing the LTO4's to their limits and my clones are finished in less than 24 hours. Back when I was doing clones from LTO4 to LTO4, these same clones ran for three days. If I was doing that with that bad VTL, I don't know that I'd successfully be able to clone the data during the week before the next round of weekend fulls were written and needed cloning.

In my environment, AFTD wins.

Frank

On 8/19/08 6:16 PM, Curtis Preston wrote:
Well, that significantly decreases the value of AFTDs over VTLs in
NetWorker, now doesn't it?

________________________________________________________
Curtis Preston | VP Data Protection
GlassHouse Technologies, Inc.

T: +1 760 710 2004 | C: +1 760 419 5838 | F: +1 760 710 2009
cpreston AT glasshouse DOT com | www.glasshouse.com
<http://www.glasshouse.com/> Infrastructure :: Optimized



________________________________

From: Francis Swasey [mailto:Frank.Swasey AT uvm DOT edu] Sent: Tuesday, August 19, 2008 2:22 PM
To: EMC NetWorker discussion; Curtis Preston
Subject: Re: [Networker] Query in Staging from adv_file

Sadly, no.  The nsrstage will serialize their use of the single RO side
of the AFTD.  With multiple nsrstage processes trying to access the same
AFTD, you will get alerts that they are waiting for the AFTD.RO
device/volume.

Frank

On 8/19/08 5:10 PM, Curtis Preston wrote:
I understand that each staging process will only use one drive at a
time, but is there a way that he can get more than one staging process
reading from the same AFTD? What if he doesn't do it automatically, but kicks off several nsrstaging
processes manually?  Is that possible?  Will they then each read some
backups and stage them to tape?  (I haven't done much with nsrstage.)






--
Frank Swasey                    | http://www.uvm.edu/~fcs
Sr Systems Administrator        | Always remember: You are UNIQUE,
University of Vermont           |    just like everyone else.
 "I am not young enough to know everything." - Oscar Wilde (1854-1900)


To sign off this list, send email to listserv AT listserv.temple DOT edu and type 
"signoff networker" in the body of the email. Please write to networker-request 
AT listserv.temple DOT edu if you have any problems with this list. You can access the 
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER