Hi everyone, thanks for the help!
Today around noon I remounted the backup disk with noatime, and then
it only took another three hours, rather than another 10, which is
exciting. I just remounted with nodiratime so we'll see if that makes
any difference in tonight's run.
I'm pretty sure that the hard links were not exploded into multiple
files, because there's several terabytes of overlapped data between
computers, but the pool is much smaller. Of course, this doesn't
preclude some other brokenness in the backup. To make the copy of the
pool, I stopped backuppc (/etc/init.d/backuppc stop), then made an LVM
snapshot of the backup logical volume, then made a tarfile of that
(with GNU tar, which handles hard links!).
The XFS filesystem is only 1/2 full, so that shouldn't be a problem.
The computer has 2G of memory, and yeah, there were some backups
running at the time. I killed those today, so that might also have
helped it finish faster. There were no errors in the backuppc log,
though there were some backups happening---not long-running ones,
though, I think. Nothing else happens on this computer, which is a
Core 2 Duo machine with three SATA-connected 750GB 7200RPM disks, with
the backup volume a 3-way stripe LVM volume with 4MB physical extents.
One weird thing about the files I back up is that a lot of them are
very similar PGM images, which don't differ in the first and last 128K
or whatever, so there are some very long hash chains: "Pool hashing
gives 523 repeated files with longest chain 182". I don't think that
this would cause hours and hours of problems, though.
If the problem is out-of-order inodes, is there some sort of XFS
tuning program that can optimize this?
If it gets down to a more manageable time, I'll probably just split
the nightly clean across several days, as you say.
Thank you again!
-Thomas
On Thu, Dec 18, 2008 at 8:33 PM, Holger Parplies <wbppc AT parplies DOT de>
wrote:
> Hi,
>
> Adam Goryachev wrote on 2008-12-19 10:56:44 +1100 [Re: [BackupPC-users]
> backuppc 3.0.0: another xfs problem?]:
>> Jeffrey J. Kosowsky wrote:
>> >
>> > I don't think that BackupPC_nightly checks for hard link dups between
>> > the pc/ and pool/ directories.
>
> I fully agree on that point.
>
>> I would advise that you confirm whether or not your hard links were
>> restored properly:
>> cd /var/lib/backuppc/pool/3/3/3
>> for file in `ls`
>
> Don't you trust shell globbing? ;-)
>
>> do
>> stat $file|grep Links|awk '{print $5" "$6}'
>> done
>
> You mean
>
> cd /var/lib/backuppc/cpool/3/3/3
> perl -e 'foreach (<*>) { $l {(stat $_) [3]} ++; }
> foreach (sort {$a <=> $b} keys %l) {
> print "$l{$_} files have $_ links\n";
> }'
>
> ? ;-)
>
> Actually, if your hard links have *not* been restored correctly, your 700GB
> tar file will have been unpacked to occupy significantly more space on your
> destination device (at least twice the amount, checking *before the first run
> of BackupPC_nightly*). I would almost be surprised if a 700GB pool was, in
> fact, restored correctly - see all of the "copying the pool" discussions for
> details.
>
>> The other possibility is that xfs is that much slower on your hardware,
>> with your mount options, etc... Perhaps look at the backuppc wiki for
>> some suggestions on improving performance on xfs.
>
> I can think of one additional point to note. Your files will have been created
> in a different order than before (probably the first instance of each inode
> encountered is "beneath" a different directory than it was on your old pool).
> If a directory references inodes scattered all over the disk and all these
> inodes need to be read (such as for determining their link count), this will
> be significantly slower than if the inodes are all neatly stored near each
> other. Even reading the inodes in numerical order (instead of by appearance
> in the directory) speeds up things (IO::Dirent optimization).
> So, while your source pool was *not* in the ideal condition (for
> BackupPC_nightly) of having the files stored neatly near each other (with
> respect to appearance in the pool directories), your current pool may (or may
> not) behave significantly worse - though I doubt we're talking of a factor of
> 22 here. It might be a combination of several things.
>
> - How much memory does your BackupPC server have? How much of that is
> available as disk cache?
> - What happens between 01:00:01 and 23:11:42 (the "[...]" in your quote from
> the log file)? Are there backups running during this time? Many? Any error
> messages?
> - Is the machine doing anything significant that is unrelated to BackupPC
> during this time?
> - Sorry if this is a stupid question: are any read errors reported for the
> disk in question?
>
> You *can* spread out the nightly cleanup process over several nights
> ($Conf{BackupPCNightlyPeriod}). If you do that, does the time it takes
> decrease proportionally? Do all passes take the same time?
>
> Regards,
> Holger
>
> ------------------------------------------------------------------------------
> SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
> The future of the web can't happen without you. Join us at MIX09 to help
> pave the way to the Next Web now. Learn more and register at
> http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users AT lists.sourceforge DOT net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
--
http://resc.smugmug.com/
------------------------------------------------------------------------------
SF.Net email is Sponsored by MIX09, March 18-20, 2009 in Las Vegas, Nevada.
The future of the web can't happen without you. Join us at MIX09 to help
pave the way to the Next Web now. Learn more and register at
http://ad.doubleclick.net/clk;208669438;13503038;i?http://2009.visitmix.com/
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
