Re: [Networker] Query in Staging from adv_file
2008-08-20 07:33:00
Curtis,
To be clear, I don't hate VTL's either. I just haven't found one yet
that I feel comfortable using here.
I don't think I have a small environment. I have a Qualstar XLS tape
library with 12 LTO4 tape drives. I have a NetWorker Server and 8
storage nodes. I have 238 clients defined (about 225 active). We run
monthly fulls, weekly differentials, and daily incrementals. Our
monthly backup volume is about 150 TB. Our nightly incrementals are
about 2 TB, our weekend full/differential cycle is about 25 TB per weekend.
I have three NexSAN SATABeasts. Each SATABeast is attached to a single
storage node and holds four AFTDs. Each of those storage nodes has two
LTO4 tape drives attached.
I have three DataDomain DDRs. Each DDR is attached to at least one
storage node and houses from one to three AFTDs. Each of these storage
nodes that needs to do cloning has one LTO4 tape drive attached.
Two of the storage nodes each have two LTO4 tape drives and no AFTDs --
they are for those poor clients that can't or refuse to spend money on a
disk buffer, so I want those tape drives to be worn out first so they
can pay for them.
I have scripted the weekly clones so a list of the SSID's on each AFTD
that needs to be cloned is generated and I then issue nsrclone commands
for as many AFTD's as I have tape drives and that way, I get the cloning
done in less than 24 hours from the AFTDs. (at least from the SATABeast
AFTDs I do. The DataDomain devices are old and they are not capable of
keeping up with our new LTO4 drives -- they were great before we
upgraded from the AIT3 drives). Replacing the almost EOSL DataDomains
with something new is my next priority.... (shhh... don't tell my
manager... he keeps telling me I'm out of money for backup).
As for staging savesets off the AFTDs... Those AFTDs that are set up to
receive incrementals -- yes, that staging definition runs 24x7.
However, for the AFTDs that are set up to receive full saves -- NO! I
turn that staging definition on after the clones finish and turn it off
again before the weekend full saves start. I still have enough
head-room that works. My reason for doing that is to prevent contention
for the tape drives when I need/want them for cloning.
I have also defined the read node on the jukebox definition to be one of
the two storage nodes that does not have any AFTDs attached -- to
further attempt to reduce the contention for the tape drives.
To be honest, I don't hate VTLs either. I wouldn't have spent the month
working with the sales folks last December if I did. I am convinced
that management would be easier (well -- at least I believe there would
be less manual intervention required) if I were using a VTL instead of
the AFTDs. However, since I was unable to make the VTL that I believed
was the best perform as I needed, I chose the complex cloning/staging
scripting of the AFTDs over a poor performing VTL. I will one day go
back and re-evaluate the VTLs -- but right now, with NW 7.4 looming over
me and the new VTL licensing requirements that make paying for AFTD
space look pleasant..... I'm going to wait a while longer.
To address your summary;
1) I am assuming by "destage" you mean moving the savesets from the VTL
to real tape. Yes, some of my AFTDs do that every day. Others do not.
It depends on their usage.
2) I agree whole heartedly. If I bought a device today that couldn't
keep an LTO4 tape drive streaming -- I made a mistake! I didn't (whew).
3) Yeah, I'll concede that point. However, I do see more recovers than
you are expecting. Because NetWorker really doesn't keep tabs (it
counts staging and cloning operations as recovers) -- I don't know how
much more and how often being able to have multiple restores reading
from the same AFTD (granted they still have to be different savesets)
actually is a benefit.
And -- yes, healthy discussion is always wonderful. I've even been
known to change my mind a time or two.
Frank
On 8/19/08 11:27 PM, Curtis Preston wrote:
If the VTLs I test could only supply data at 40 MB/s, I wouldn't recommend them
either. The ability of a disk device to stream to tape is job #1, AFAIC. Just
remember that not all VTLs are created equal.
Don't get me wrong. I'm not an AFTD hater. I think they're fine for smaller
shops that want an easy solution. But when I get to a larger shop, I need the
ability to make things go more concurrently, and I think it's possible to do
that with a VTL and VERY difficult to impossible to do it with an AFTD.
VTLs aren't the end-all-be-all, and some of them are crap. (The same is true
of the disk systems behind a lot of AFTDs.) They're also more management in a
SMALL NW environment than AFTDs. But in a large environment, where
staging/cloning performance is king, I'll take a VTL any day. Yes, I'll be
doing some scripting, but at least I can clone/destage multiple backups at the
same time. That's NOT the case with an AFTD unless you make a bunch of them
and that creates other management hassles, and decreases disk utilization.
I think the concurrent restore thing is a red herring. Sure, you can't do two
restores at once from a tape, but:
1. Most people don't do more than one restore a month, let alone more than one
a day, let ALONE two at the same time.
2. Add to that the odds that the two restores you do at the same time would
need the same tape at the same time. (Multiplexing would increase the chances,
but one should not multiplex to virtual tapes.)
3. If you want better odds, just use smaller virtual tapes. It costs you
nothing, adds no management issues (like more AFTDs do), and significantly
decreases the chance that two restores would be grabbing for the same tape.
So, to summarize:
1. I think it's easier to destage multiple backups simultaneously from a VTL
than it is to do it from an AFTD, and that's something you do every night.
2. Performance counts. If your device, regardless of type, can't stream your
tape drive type, you bought the wrong one.
3. Sure I can do concurrent restores from the same "tape" with AFTDs and I
can't with VTLs, but I don't think that happens much and if it does I can address it with
smaller virtual tape sizes.
Isn't healthy discussion fun?
Curtis Preston | VP Data Protection
GlassHouse Technologies, Inc.
T: +1 760 710 2004 | C: +1 760 419 5838 | F: F: +1 760 710 2009
cpreston AT glasshouse DOT com | www.glasshouse.com
Infrastructure :: Optimized
-----Original Message-----
From: EMC NetWorker discussion [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
Behalf Of Francis Swasey
Sent: Tuesday, August 19, 2008 6:13 PM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: Re: [Networker] Query in Staging from adv_file
With VTL, Networker limits you to one operation at a time per virtual
Tape. I've heard tales that if you are lucky enough to issue two
recover commands at the exact same instant, that NetWorker can be
tricked into doing two restores from the same tape at once.
With the AFTD, you can do multiple concurrent writes and either multiple
restores or a stage or a clone at the same time per AFTD.
In my environment, I've tested a VTL and it was a lousy performer. I
wasted an entire month and had to put up with a lot of pressure from the
sales people who really wanted me to spend a small fortune on their
appliance when they couldn't make it perform at even one third of the
I/O performance of the AFTD on a NexSAN SATABeast.
If I had to do my weekly clones with a VTL that was flat out retrieving
data at 35-38MB/second to write to LTO4 tape drives that can do
120MB/second -- I'd be looking for a new job! Thankfully, on my big
fileservers with the big savesets, I'm pushing the LTO4's to their
limits and my clones are finished in less than 24 hours. Back when I
was doing clones from LTO4 to LTO4, these same clones ran for three
days. If I was doing that with that bad VTL, I don't know that I'd
successfully be able to clone the data during the week before the next
round of weekend fulls were written and needed cloning.
In my environment, AFTD wins.
Frank
On 8/19/08 6:16 PM, Curtis Preston wrote:
Well, that significantly decreases the value of AFTDs over VTLs in
NetWorker, now doesn't it?
________________________________________________________
Curtis Preston | VP Data Protection
GlassHouse Technologies, Inc.
T: +1 760 710 2004 | C: +1 760 419 5838 | F: +1 760 710 2009
cpreston AT glasshouse DOT com | www.glasshouse.com
<http://www.glasshouse.com/>
Infrastructure :: Optimized
________________________________
From: Francis Swasey [mailto:Frank.Swasey AT uvm DOT edu]
Sent: Tuesday, August 19, 2008 2:22 PM
To: EMC NetWorker discussion; Curtis Preston
Subject: Re: [Networker] Query in Staging from adv_file
Sadly, no. The nsrstage will serialize their use of the single RO side
of the AFTD. With multiple nsrstage processes trying to access the same
AFTD, you will get alerts that they are waiting for the AFTD.RO
device/volume.
Frank
On 8/19/08 5:10 PM, Curtis Preston wrote:
I understand that each staging process will only use one drive at a
time, but is there a way that he can get more than one staging process
reading from the same AFTD?
What if he doesn't do it automatically, but kicks off several nsrstaging
processes manually? Is that possible? Will they then each read some
backups and stage them to tape? (I haven't done much with nsrstage.)
--
Frank Swasey | http://www.uvm.edu/~fcs
Sr Systems Administrator | Always remember: You are UNIQUE,
University of Vermont | just like everyone else.
"I am not young enough to know everything." - Oscar Wilde (1854-1900)
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
|
|