Networker

Re: [Networker] Legato Networker 7.4.3 and retention/recycling

2009-10-13 22:27:30
Subject: Re: [Networker] Legato Networker 7.4.3 and retention/recycling
From: Joel Fisher <jfisher AT WFUBMC DOT EDU>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Tue, 13 Oct 2009 22:21:55 -0400
Thanks for the response Mike.

Just a couple more questions if you don't mind.

Does that 200TB of data include the tape part or is that just the online
part?

So does legato address the entire samqfs space with one file device?  Or
do you have several directories that get written to?

Thanks!

Joel

-----Original Message-----
From: Mike Cannon [mailto:mikec AT clemson DOT edu] 
Sent: Tuesday, October 13, 2009 9:08 AM
To: Joel Fisher; NETWORKER AT LISTSERV.TEMPLE DOT EDU
Cc: Matthew Powell
Subject: Re: [Networker] Legato Networker 7.4.3 and retention/recycling


On Oct 8, 2009, at 8:09 PM, Joel Fisher wrote:

> Hey Matt/Mike,
>
> I'm in the process of figuring out how to backup our data in a sane
> manner without the purchase of really expensive data domain(like)
> devices.  I got to thinking about using a really large sam-qfs
> filesystem with Nexsan arrays on the back and backed up by LTO-4  
> drives.
> Then just using them as adv_file type

I don't think we are using adv_file types, matt would need to answer  
that

> devices and letting sam copy them
> to tape for me.  The more I thought about it, it almost seemed  
> ideal...
> I'm sure I'm missing a lot of the details.  So, anyway I decided to  
> look
> through the archives to see if anyone else was doing this and it looks
> to me like that's exactly what you guys are doing.  So I was wondering
> if you wouldn't mind answering a few questions for me?
>
> 1) What amount of data are you backing up to the samqfs?

We have about 200TB of data within samqfs part of the active 42 day   
retention window owned by legato savesets.  We are 2/3 of our way  
through the migration.

> 2) How is it architected?  1 huge sam-qfs shared across several  
> storage
> nodes and striped across several disk arrays, or segmented in some  
> way?


Currently 1 huge samqfs volume striped across four 2540 controller /  
tray pairs comprised of sixteen 4+1 raid sets.  These are all SAS  
300GB drives.  The next one we are building will be SATA.   The meta  
data server is shared to one one samqfs client which is dedicated to  
running legato.  I have learned that this does not need to be on  
separate machines.  We use the sun 5220 hardware and there is plenty  
of IO and processor bandwidth to do both on the same machine, at least  
with this line of sparc chips.  We may introduce a storage node, so  
far we have not hit any performance bottle necks that would require a  
storage node.  We actively monitor the performance and add things as  
needed.  We have upgraded since we started building this a couple  
times.  We have doubled the disks, both data and meta data devices and  
gone from one 1Gb to Several 1Gb snd finally to 10Gb Ethernet.  I am  
prepared to add a storage node to the same or new disks as needed.

We write the legato data and metadata to the same file system.


> 3) What type of throughput are you able to achieve?

We are able to run legato wide open (2 Gb of inbound Ethernet) writing  
to qfs and sustain 4 tape drives at full 120MB/s bandwidth (read and  
or write) simultaneously.  We also allow users to write to the same  
samqfs file system via NFS.  I think there is some limiter within  
legato potentially related to parallelism that is preventing moving  
more concurrent data on certain jobs.

> 4) If you had it to do over, would you do it this way again? Why?

We are still building these same solutions.  I do recommend this to  
everyone, regardless of the backup software it works.  Before the  
migration to the new architecture started our legacy legato system  
using traditional disk and tape had a storage shortage.  We plumed up  
an NFS connection to the legacy server and streamed the data to samqfs  
without any other modification.

> 5) What happens when Networker wants to restore a saveset that sam has
> moved to a tape and removed from the disk?  Does it just wait?  Does  
> it
> write it back to disk first then restore or restore directly from  
> tape?

Legato like all commands I have seen so far just waits.  Once the  
first part of the file has been restored (a couple meg) legato begins  
it transfer to the target of the restore.

> 6) What has the affect been on your restore performance?

I don't have before and afters comparison...  I didn't manage the  
legacy box.  The over all ethernet based legato client/server backups  
have not change in throughput capacity.

> 7) What size tape library do you need?

We have two sl8500 with six t10000 drives each.  The libraries sit in  
different buildings.

> 8) What happens it legato requests a restore of a saveset who's  
> saveset
> file is on a tape that has been ejected from the library?

We don't eject tapes.  I was told in the beginning that this can be  
tricky and we should avoid this.  But as long and you manage the tape  
archive sets you can remove tape.  For example we also write the  
legato indexes to the same samqfs file system.  You would want to  
ensure that you had a separate archive set for this to ensure an index  
was never offline.

> 9) Is sam-qfs a one cost item or based on capacity.  I've heard
> conflicting stories.

I think if you buy support it that becomes capacity based.  There is a  
free version but we don't use that version.


>
> If you guys are willing, I'm sure I may have other questions along the
> way.

sure

> Any other bits of info you might think would help me make a decision.


samqfs gives you the added benefit of the mountable file system by  
clients over nfs, samba, and the qfs client.  It also can  
automatically create multiple copies so you don't have to perform  
cloning.


> I would really appreciate the help.
>
> Thanks in advance!
>
> Joel
>
>
> -----Original Message-----
> From: EMC NetWorker discussion  
> [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
> Behalf Of Matthew Powell
> Sent: Monday, June 08, 2009 3:59 PM
> To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
> Subject: [Networker] Legato Networker 7.4.3 and retention/recycling
>
> I am currently administering an instance of Networker that writes its
> data to a giant disk volume that will almost never fill up. It is  
> backed
> by SAM/QFS and 3000 tapes so we are good there. My question is this: I
> have some save sets that are past expiration time but they are not
> getting removed by Legato. Is the reason that Legato is not releasing
> this data because the drives have plenty of space left? Just curious  
> if
> there is some sort of setting that I can set so that it will recycle
> data when it is expired even though it doesn't need to re-claim the
> space? Any suggestion/advice  would be greatly appreciated. Thanks.
>
>
>
> Matt Powell
>
> Storage Administrator
>
> Clemson University
>
> office : 865-656-0589
>
> cell: 864-650-8237
>
>
>
> "Yesterday is history. Tomorrow is a mystery. Today is a gift. That's
> why its called the present."
>
>
>
>
> To sign off this list, send email to listserv AT listserv.temple DOT edu and
> type "signoff networker" in the body of the email. Please write to
> networker-request AT listserv.temple DOT edu if you have any problems with  
> this
> list. You can access the archives at
> http://listserv.temple.edu/archives/networker.html or
> via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the body of the email. Please write to 
networker-request AT listserv.temple DOT edu if you have any problems with this 
list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER