Re: [BackupPC-users] Splitting up large directories.

On Tue, May 18, 2010 at 02:04:46PM -0700, Robin Lee Powell wrote:
> A customer we're backing up has a directory with ~500 subdirs and
> hundreds of GiB of data.  We're using BackupPC in rsync+ssh mode.
> 
> As a first pass at breaking that up, I made a bunch of seperate host
> entries like /A/*0, /A/*1, ... (all the dirs have numeric names).
> 
> That seems to select the right files, but it doesn't work because
> BackupPC ends up running a bunch of them at once, hammering that
> customer's machine.

You can set it up so only a few of those hosts will run at the same
time using

  $Conf{UserCmdCheckStatus} = 1;

and a $Conf{DumpPreUserCmd}/$Conf{DumpPostUserCmd} that know the host
names and implement a counting semaphore to make sure only some number
of them are running at the same time. I posted a longer sketch of how
I limit the number of parallel backups to remote sites in the archives
some time ago.
 
> I could make those into share names, but I'm worried about running
> out of command-line argument space in that case; that is, that
> "rsync /A/*0" will at some point in the future expand to a hundred
> or more directories and break.
> 
> What I want to do is have a bunch of shares like [ "/A", "/A", ...],
> and have something like:
> 
> $Conf{BackupFilesOnly} = {
>   "/A" => "/A/*0",
>   "/A" => "/A/*1",
>   "/A" => "/A/*2",
>   }
> 
> But obviously that's not going to work.
> 
> Does anyone have any other way to handle this?

How about a set of symbolic links on the host:

  /backup_slices/A0
  /backup_slices/A1
  /backup_slices/A2
  /backup_slices/A3
  /backup_slices/A4
  /backup_slices/A5
  ...

which all point to /A. Then a spec:

 $Conf{BackupFilesOnly} = {
   "/backup_slices/A0/." => "/*0",
   "/backup_slices/A1/." => "/*1",
   "/backup_slices/A2/." => "/*2",
   ...
   }

Not pretty, but I think it will work as the trailing . should cause
rsync to dereference the symbolic link, and the resulting rsync will
start in /A.

-- 
                                -- rouilj

John Rouillard       System Administrator
Renesys Corporation  603-244-9084 (cell)  603-643-9300 x 111

------------------------------------------------------------------------------

_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/