On Mon, Nov 19, 2012 at 10:03:03PM +0100, Markus wrote:
> Am 15.11.2012 19:20, schrieb Les Mikesell:
> > If there are top-level directories segregating the files sensibly you
> > could split it into multiple 'shares'.   Otherwise, you could switch
>[...]
> If that shouldn't work for some reason I will try these suggestions:
> 
> a) Different profiles with aliases
> b) Different shares
> c) tar
> (or a combination of a, b, c)
> 
> Unforunately almost all of the files are located in /home in thousands 
> of subdirectories, so I can't just say "backup /home or /home/1, 
> /home/2, /home/3 in another profile", but if tar won't work I guess I 
> will have to dig deeper into the subdirectories for profiles/shares 
> splitting.

What may also work is to use excludes to do your sharding. I have 4
"hosts" now with different excludes. All of them back up the same share:

   $Conf{RsyncShareName} = [ '/home1', ];

Then have different exclusion lists:

   # Use exclusion of directories rather than inclusion. This way if a
   # new directory starting say with ! or . is created it will get backed
   # up by all of the hosts. This is a slight waste of compute
   # resources, but it is also failsafe in that a new unlisted directory
   # will be backed up without any other changes by default.

   # include subdirectories starting with a, b, or c case insensitive
   $Conf{BackupFilesExclude} = {
     '/home1' => [ "- /[D-Zd-z]*/**" ],
   };

This just backs up directories under /home1 that start with a, b, c,
A, B or C. Then a second host config has:

   # include subdirectories starting with d...m case insensitive
   $Conf{BackupFilesExclude} = {
     '/home1' => [ "- /[A-Ca-c]*/**", "- /[N-Zn-z]*/**", "- /user/**" ],
   };

   # include subdirectories starting with d...m case insensitive
   $Conf{BackupFilesExclude} = {
     '/home1' => [ "- /[A-Ca-c]*/**", "- /[N-Zn-z]*/**", "- /user/**" ],
   };

I have a particular user (called user) who I want to back up alone, so
I exclude him from the backups here.

The third host completes the normal directories:

   # include subdirectories starting with n...z case insensitive
   $Conf{BackupFilesExclude} = {
     '/home1' => [ "- /[A-Ma-m]*/**",  "- /user/**"
                    ],
   };

The fourth host backs up only the problem user:

   $Conf{BackupFilesExclude} = {
     '/home1' => [ "+ /user/**", "- /[A-Za-z]*/**" ],
   };


The exclude lists are passed through to the underlying rsync mechanism
and depend on that mechanism to work. So the setup above won't work
for tar.

You should be able to use the xferlog stats (e.g.
BACKUPPC_ROOT/pc/<host>/XferLOG.<run>.z) from the backups (use
BackupPC_zcat to dump the log file) to see what the size is of the
various directory trees/files.

Hopefully you can create roughly equal sized (either in bytes or in
number of files) groups and then make exclude lists that subtract all
the subdirectories that do not fit in that group.

That may be easier to specify than each individual directory you want
to back up.

-- 
                                -- rouilj

John Rouillard       System Administrator
Renesys Corporation  603-244-9084 (cell)  603-643-9300 x 111

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/