Re: [BackupPC-users] rsync building file list takes forever

On Sat, Mar 1, 2014 at 10:20 AM, Dr. Boris Neubert <omega AT online DOT de> 
wrote:
>>
> On the other hand, rsync 3.0.6 on the Virtual Machine Server takes hours
> and hours to build the file list - actually I never waited for this to
> finish. We talk about 1,800 files, most of them small log files and the
> bulk of the 578 GB of data in maybe 50 large virtual disk files and a
> handful of ISO image dumps of DVDs. The underlying file system for
> backup is a snapshot of a logical volume (LVM2). File system type is xfs.
>
> This renders it impossible to backup the Virtual Machine Server.

'Hours" to start a backup doesn't seem like a real showstopper.   What
happens if you just let it run over a weekend?

Rsync wants to read the entire directory tree into memory at both ends
before walking it to do the comparisons, although I thought the latest
rsync versions and backuppc 4.x were supposed to improve this.   That
doesn't sound like a huge file list, though.    How long does it take
for something like 'find . -ctime -1' to complete starting at the top
of that directory (or maybe ls -lR .  - anything that makes it read
all the filenames and stat all the files)?

> While I am writing this I can observe the output of strace -p <PID of
> rsyncd on client #2> scrolling through. There are some intermittent
> calls to select that timeout. The hosts XferLOG is still empty.

There is always a chance you have filesystem corruption.

> I tried running rsync on the BackupPC server with and without the
> --checksum option but did not see any difference.

Checksum caching only works after the 2nd full run - and it only saves
work on the server side.

-- 
  Les Mikesell
    lesmikesell AT gmail DOT com

------------------------------------------------------------------------------
Flow-based real-time traffic analytics software. Cisco certified tool.
Monitor traffic, SLAs, QoS, Medianet, WAAS etc. with NetFlow Analyzer
Customize your own dashboards, set traffic alerts and generate reports.
Network behavioral analysis & security monitoring. All-in-one tool.
http://pubads.g.doubleclick.net/gampad/clk?id=126839071&iu=/4140/ostg.clktrk
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/