Amanda-Users

Re: handling unreasonably large, non-static directories

2006-01-11 20:04:53
Subject: Re: handling unreasonably large, non-static directories
From: Frank Smith <fsmith AT hoovers DOT com>
To: amanda-users AT amanda DOT org
Date: Wed, 11 Jan 2006 18:54:12 -0600
Cameron Matheson wrote:
> Hi,
> 
> Amanda has been working wonderfully for me ever since I started using it 
> about a year ago.  I do have one question though, that plagues me every 
> time I try to confront it:
> 
> I have one directory on one of my boxes that holds files for customers 
> (each customer gets a subdirectory (the subdirectory seems to just be a 
> customer number... so it's mostly sequential unless a customer gets 
> deleted.  The size of these directories varies widely (anywhere from a 
> few megabytes to 15 gigabytes).  All in all there is a little under 
> 200GB of data that needs to be backed up.  Initially I had just been 
> going through the list of directories myself and compiling 15GB chunks 
> of them to be backed up, but due to the ever-changing nature of these 
> directories it's kind of a pain to keep up w/ that.  Is there any way I 
> could have amanda automatically split this directory up into chunks to 
> be backed up?  Or, does anyone else have any keen ideas on how one might 
> approach this problem?
> 
> Thanks,
> Cameron Matheson

Try using regular expressions to split your subdirectories.
Many people on this list wildcard the end of the name, such
as /data/1*, but for your application I would suggest wildcarding
the first digits instead of the last instead, such as /data/*1,
since the first digit(s) of your customer numbers are probably
less evenly distributed than the last. If splitting it into
10 chunks isn't enough, you could try something like /data/*1[0-4]
and /data/*1[5-9] etc.  and split it into 20 DLEs.
   The man page claims that disk/directory names are glob
expressions and not regexes, and only shows regexes in the
include/exclude lists, so you might need a bunch of DLEs with
the same diskdevice name with separate disknames, and include
excludes for each using regexes.
    Don't forget to add a DLE excluding all of the above DLEs
to pick up any stray subdirectories that don't match any of your
regexes, just in case some non-numeric subdirectories are created
in the future.

Frank


-- 
Frank Smith                                      fsmith AT hoovers DOT com
Sr. Systems Administrator                       Voice: 512-374-4673
Hoover's Online                                   Fax: 512-374-4501