On 11/19/2012 4:35 PM, John Rouillard wrote:
>
> What may also work is to use excludes to do your sharding. I have 4
> "hosts" now with different excludes. All of them back up the same share:
>
>     $Conf{RsyncShareName} = [ '/home1', ];
>
> Then have different exclusion lists:
>
>     # Use exclusion of directories rather than inclusion. This way if a
>     # new directory starting say with ! or . is created it will get backed
>     # up by all of the hosts. This is a slight waste of compute
>     # resources, but it is also failsafe in that a new unlisted directory
>     # will be backed up without any other changes by default.
>
>     # include subdirectories starting with a, b, or c case insensitive
>     $Conf{BackupFilesExclude} = {
>       '/home1' => [ "- /[D-Zd-z]*/**" ],
>     };
>
> This just backs up directories under /home1 that start with a, b, c,
> A, B or C. Then a second host config has:
>
>     # include subdirectories starting with d...m case insensitive
>     $Conf{BackupFilesExclude} = {
>       '/home1' => [ "- /[A-Ca-c]*/**", "- /[N-Zn-z]*/**", "- /user/**" ],
>     };
>
>     # include subdirectories starting with d...m case insensitive
>     $Conf{BackupFilesExclude} = {
>       '/home1' => [ "- /[A-Ca-c]*/**", "- /[N-Zn-z]*/**", "- /user/**" ],
>     };
>
> I have a particular user (called user) who I want to back up alone, so
> I exclude him from the backups here.
>
> The third host completes the normal directories:
>
>     # include subdirectories starting with n...z case insensitive
>     $Conf{BackupFilesExclude} = {
>       '/home1' => [ "- /[A-Ma-m]*/**",  "- /user/**"
>                      ],
>     };
>
> The fourth host backs up only the problem user:
>
>     $Conf{BackupFilesExclude} = {
>       '/home1' => [ "+ /user/**", "- /[A-Za-z]*/**" ],
>     };
>
>
> The exclude lists are passed through to the underlying rsync mechanism
> and depend on that mechanism to work. So the setup above won't work
> for tar.
>
> You should be able to use the xferlog stats (e.g.
> BACKUPPC_ROOT/pc/<host>/XferLOG.<run>.z) from the backups (use
> BackupPC_zcat to dump the log file) to see what the size is of the
> various directory trees/files.
>
> Hopefully you can create roughly equal sized (either in bytes or in
> number of files) groups and then make exclude lists that subtract all
> the subdirectories that do not fit in that group.
>
> That may be easier to specify than each individual directory you want
> to back up.

That seems a bit overly complex.  Wouldn't it be easier to use includes?

# include subdirectories starting with a, b, or c case insensitive
    $Conf{BackupFilesOnly} = {
      '/home1' => [ "/[A-Ca-c]*/**" ],
    };

    # include subdirectories starting with d...m case insensitive
    $Conf{BackupFilesOnly} = {
      '/home1' => [ "/[D-Md-m]*/**" ],
    };

    # include subdirectories starting with n...z case insensitive
    $Conf{BackupFilesOnly} = {
      '/home1' => [ "/[N-Zn-z]*/**" ],
    };
    # exclude your problem case
    $Conf{BackupFilesExclude} = {
      '/home1' => [ "- /user/**" ],
    };

    # back up problem user and other misc directories (non-alphabetic 
first char)
    $Conf{BackupFilesExclude} = {
      '/home1' => [ "+ /user/**", "- /[A-Za-z]*/**" ],
    };

This way, it is much more obvious what is being backed up by each host.

This is off the top of my head and not tested, so it may need to be 
tweaked a bit.

-- 
Bowie

------------------------------------------------------------------------------
Monitor your physical, virtual and cloud infrastructure from a single
web console. Get in-depth insight into apps, servers, databases, vmware,
SAP, cloud infrastructure, etc. Download 30-day Free Trial.
Pricing starts from $795 for 25 servers or applications!
http://p.sf.net/sfu/zoho_dev2dev_nov
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/