Networker

Re: [Networker] Splitting up large directory save jobs into smaller pieces (aka. "parallelizing")

2010-03-03 13:10:46
Subject: Re: [Networker] Splitting up large directory save jobs into smaller pieces (aka. "parallelizing")
From: Tim Mooney <Tim.Mooney AT NDSU DOT EDU>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Wed, 3 Mar 2010 12:07:46 -0600
In regard to: [Networker] Splitting up large directory save jobs into...:

One customer asked me if it is possible to speed up saves of large
directories with many sub-dirs (or files). Unfortunately Networker itself
cannot automatically parallelize saving of directories. Thats quite odd.

The ability to automatically parallelize different parts of one filesystem
has been on the list of desired features for a long time.  That feature
was included in a previous release of the product (I think it was 7.3,
but I would have to check my notes to be certain) as a hidden,
experimental feature.  It was subsequently either removed or completely
disabled.

So i wrote  simple  script bash script  in order to split up large
directories in  smaller (and parallel) save jobs.

No script that's going to do it robustly is going to remain simple, I
think.  :-|

Most of the discussion on the list in the past about this problem has
centered around the "split client" trick, which does not automatically
split things (the administrator has to set up how a volume is split) but
is "safe", if done correctly, because one of the two (or more) clients
uses the All saveset.

Tim
--
Tim Mooney                                             Tim.Mooney AT ndsu DOT 
edu
Enterprise Computing & Infrastructure                  701-231-1076 (Voice)
Room 242-J6, IACC Building                             701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164

To sign off this list, send email to listserv AT listserv.temple DOT edu and type 
"signoff networker" in the body of the email. Please write to networker-request 
AT listserv.temple DOT edu if you have any problems with this list. You can access the 
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER