Actually, one of the files in question is the fformat file of a .svn/
directory which appears many times in one of the backed up
filesystems. Since it will have the same md5 sum, BackupPC links it to
the same file in cpool...
This works fine in the local server since backuppc is aware of the
link limit, but fails when rsyncing to the remote server since new
backups would increase this link count...
Best regards,
Johannes H. Jensen
On Sun, Feb 28, 2010 at 10:56 PM, Johannes H. Jensen
<joh AT pseudoberries DOT com> wrote:
> Yep, seems like I have 32000 links to the file in question:
>
> ls -l cpool/7/c/6/7c67493bd72ceff21059c3d924d17518
> -rw-r----- 32000 backuppc backuppc 10 2009-01-25 19:02
> cpool/7/c/6/7c67493bd72ceff21059c3d924d17518
>
> Which seems a bit weird to me... I mean, at maximum it should only be
> one link per backup?
>
> Best regards,
>
> Johannes H. Jensen
>
>
>
> On Sun, Feb 28, 2010 at 9:20 PM, Michael Kuss <michael.w.kuss AT gmail DOT
> com> wrote:
>>
>>
>> On Sun, Feb 28, 2010 at 7:44 PM, Johannes H. Jensen <joh AT pseudoberries
>> DOT com>
>> wrote:
>>>
>>> $ df -i
>>> Filesystem Inodes IUsed IFree IUse% Mounted on
>>> /dev/mapper/BACKUP 37429248 4138304 33290944 12% /backup
>>>
>>> 31 is an error code as far as I can see. It's the same for all the
>>> link attempts...
>>
>> Well, probably you're right, but it would be the same number too if it is a
>> hard link count you're trying to exceed. Can you confirm that you have
>> 32000 hard links to one of the files in question (ls -l ...)?
>>
>> Michael
>>
>>> On Sun, Feb 28, 2010 at 7:29 PM, Michael Kuss <michael.w.kuss AT gmail DOT
>>> com>
>>> wrote:
>>> >
>>> >
>>> > On Sat, Feb 27, 2010 at 5:22 PM, dan <dandenson AT gmail DOT com> wrote:
>>> >>
>>> >> Are you running into the actual hardlink limit or an inode limit? ext3
>>> >> has
>>> >> a hard coded hardlink limit but hardlinks are also limited by available
>>> >> inodes. you can check your available inodes with
>>> >>
>>> >> tune2fs -l /dev/disk|grep -e "Free inodes" -e "Inode count"
>>> >
>>> > df -i does the same, and you don't need to be root.
>>> >
>>> > Michael
>>> >
>>> >>
>>> >> if you have very few or none left then this is your problem. You cant
>>> >> change the inode count on an existing ext3 filesystem as far as I know
>>> >> but
>>> >> if you re-create the filesystem you can do
>>> >> mkfs.ext3 -N ##### /dev/disk
>>> >> change the ##### to suite your needs. You should know the current
>>> >> number
>>> >> for the tune2fs command above. I would just take your current
>>> >> filesystem
>>> >> usage (lets say 62% for the math) then take the `current number` * 3 /
>>> >> .62
>>> >> so that you have enough inodes for today PLUS you are compensated for
>>> >> when
>>> >> the disks are fuller.
>>> >>
>>> >>
>>> >>
>>> >> On Sat, Feb 27, 2010 at 6:12 AM, Johannes H. Jensen
>>> >> <joh AT pseudoberries DOT com> wrote:
>>> >>>
>>> >>> Thank you for your input,
>>> >>>
>>> >>> On Sat, Feb 27, 2010 at 3:38 AM, dan <dandenson AT gmail DOT com> wrote:
>>> >>> > if [ -e /var/lib/backuppc/testfile ];
>>> >>> > then rsync xxxx;
>>> >>> > else echo "uh oh!";
>>> >>> > fi
>>> >>> >
>>> >>> > should make sure that the filesystem is mounted.
>>> >>>
>>> >>> Yes, that's definitely a good idea. However it does not check to make
>>> >>> sure that the integrity of the BackupPC pool is okay. If only a small
>>> >>> subset of the backup pool gets removed/corrupted/etc, this would still
>>> >>> get reflected in the remote mirror. I would prefer some
>>> >>> BackupPC-oriented way of doing this (maybe BackupPC_serverMesg status
>>> >>> info?) if someone could provide me with the details.
>>> >>>
>>> >>> > you could also first do a try run
>>> >>> > rsync -avnH --delete /source /destination > /tmp/list
>>> >>> > then identify what will be deleted:
>>> >>> > cat /tmp/list|grep deleting|sed 's/deleting /\//g'
>>> >>> >
>>> >>> > now you have a list of everything that WOULD be deleted with the
>>> >>> > --delete
>>> >>> > option. Run your normal sync and save this file for later
>>> >>> >
>>> >>> > You could save take this file list and send it to the remote system
>>> >>> >
>>> >>> > scp /tmp/list remotehost:/list-`date -%h%m%s`
>>> >>> >
>>> >>> > on remote system
>>> >>> >
>>> >>> > cat /list-* | xargs rm
>>> >>> >
>>> >>> > to delete the file list. You could do this weekly or monthly or
>>> >>> > whenever
>>> >>> > you needed.
>>> >>>
>>> >>> That's a good idea. My original thought was to manually run the rsync
>>> >>> with the --delete option once a week or so, but we've already run into
>>> >>> filesystem (ext3) problems where we exceed the maximum links after a
>>> >>> few days because we don't --delete... I guess we could use another
>>> >>> filesystem with a higher limit instead...
>>> >>>
>>> >>>
>>> >>> Best regards,
>>> >>>
>>> >>> Johannes H. Jensen
>>> >>>
>>> >>>
>>> >>>
>>> >>> > On Fri, Feb 26, 2010 at 6:27 AM, Johannes H. Jensen
>>> >>> > <joh AT pseudoberries DOT com>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Hello,
>>> >>> >>
>>> >>> >> We're currently syncing our local BackupPC pool to a remote server
>>> >>> >> using rsync -aH /var/lib/backuppc/ remote:/backup/backuppc/
>>> >>> >>
>>> >>> >> This is executed inside a script which takes care of stopping
>>> >>> >> BackupPC
>>> >>> >> while rsync is running as well as logging and e-mail notification.
>>> >>> >> The
>>> >>> >> script nightly as a cronjob.
>>> >>> >>
>>> >>> >> This works fairly well, except it won't remove old backups from the
>>> >>> >> remote server. Apart from using up unnecessary space, this has also
>>> >>> >> caused problems like hitting the remote filesystems hard link
>>> >>> >> limit.
>>> >>> >>
>>> >>> >> Now I'm aware of rsync's --delete option, but I find this very
>>> >>> >> risky.
>>> >>> >> If for some reason the local backup server fails and
>>> >>> >> /var/lib/backuppc/ is somehow empty (disk fail etc), then --delete
>>> >>> >> would cause rsync to remove *all* of the mirrored files on the
>>> >>> >> remote
>>> >>> >> server. This kind of ruins the whole point of having a remote
>>> >>> >> mirror...
>>> >>> >>
>>> >>> >> So my question is then - how to make sure that the local backup
>>> >>> >> pool
>>> >>> >> is sane and up-to-date without risking loosing the entire remote
>>> >>> >> pool?
>>> >>> >>
>>> >>> >> I have two ideas of which I'd love some input:
>>> >>> >>
>>> >>> >> 1. Perform some sanity check before running rsync to ensure that
>>> >>> >> the
>>> >>> >> local backuppc directory is indeed healthy. How this sanity check
>>> >>> >> should be performed I'm unsure of. Maybe check for existence of
>>> >>> >> some
>>> >>> >> file or examine the output of `BackupPC_serverMesg status info'?
>>> >>> >>
>>> >>> >> 2. Run another instance of BackupPC on the remote server, using the
>>> >>> >> same pc and hosts configuration as the local server but with
>>> >>> >> $Conf{BackupsDisable} = 2 in the global config. This instance
>>> >>> >> should
>>> >>> >> then keep the remote pool clean (with BackupPC_trashClean and
>>> >>> >> BackupPC_nightly), or am I mistaken? Of course, this instance also
>>> >>> >> has
>>> >>> >> to be stopped while rsyncing from the local server.
>>> >>> >>
>>> >>> >> If someone could provide some more info on how this can be done
>>> >>> >> safely, it would be greatly appreciated!
>>> >>> >>
>>> >>> >>
>>> >>> >> Best regards,
>>> >>> >>
>>> >>> >> Johannes H. Jensen
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> ------------------------------------------------------------------------------
>>> >>> >> Download Intel® Parallel Studio Eval
>>> >>> >> Try the new software tools for yourself. Speed compiling, find bugs
>>> >>> >> proactively, and fine-tune applications for parallel performance.
>>> >>> >> See why Intel Parallel Studio got high marks during beta.
>>> >>> >> http://p.sf.net/sfu/intel-sw-dev
>>> >>> >> _______________________________________________
>>> >>> >> BackupPC-users mailing list
>>> >>> >> BackupPC-users AT lists.sourceforge DOT net
>>> >>> >> List:
>>> >>> >> https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> >>> >> Wiki: http://backuppc.wiki.sourceforge.net
>>> >>> >> Project: http://backuppc.sourceforge.net/
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > ------------------------------------------------------------------------------
>>> >>> > Download Intel® Parallel Studio Eval
>>> >>> > Try the new software tools for yourself. Speed compiling, find bugs
>>> >>> > proactively, and fine-tune applications for parallel performance.
>>> >>> > See why Intel Parallel Studio got high marks during beta.
>>> >>> > http://p.sf.net/sfu/intel-sw-dev
>>> >>> > _______________________________________________
>>> >>> > BackupPC-users mailing list
>>> >>> > BackupPC-users AT lists.sourceforge DOT net
>>> >>> > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> >>> > Wiki: http://backuppc.wiki.sourceforge.net
>>> >>> > Project: http://backuppc.sourceforge.net/
>>> >>>
>>> >>>
>>> >>>
>>> >>> ------------------------------------------------------------------------------
>>> >>> Download Intel® Parallel Studio Eval
>>> >>> Try the new software tools for yourself. Speed compiling, find bugs
>>> >>> proactively, and fine-tune applications for parallel performance.
>>> >>> See why Intel Parallel Studio got high marks during beta.
>>> >>> http://p.sf.net/sfu/intel-sw-dev
>>> >>> _______________________________________________
>>> >>> BackupPC-users mailing list
>>> >>> BackupPC-users AT lists.sourceforge DOT net
>>> >>> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> >>> Wiki: http://backuppc.wiki.sourceforge.net
>>> >>> Project: http://backuppc.sourceforge.net/
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> ------------------------------------------------------------------------------
>>> >> Download Intel® Parallel Studio Eval
>>> >> Try the new software tools for yourself. Speed compiling, find bugs
>>> >> proactively, and fine-tune applications for parallel performance.
>>> >> See why Intel Parallel Studio got high marks during beta.
>>> >> http://p.sf.net/sfu/intel-sw-dev
>>> >> _______________________________________________
>>> >> BackupPC-users mailing list
>>> >> BackupPC-users AT lists.sourceforge DOT net
>>> >> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> >> Wiki: http://backuppc.wiki.sourceforge.net
>>> >> Project: http://backuppc.sourceforge.net/
>>> >>
>>> >
>>> >
>>> >
>>> > ------------------------------------------------------------------------------
>>> > Download Intel® Parallel Studio Eval
>>> > Try the new software tools for yourself. Speed compiling, find bugs
>>> > proactively, and fine-tune applications for parallel performance.
>>> > See why Intel Parallel Studio got high marks during beta.
>>> > http://p.sf.net/sfu/intel-sw-dev
>>> > _______________________________________________
>>> > BackupPC-users mailing list
>>> > BackupPC-users AT lists.sourceforge DOT net
>>> > List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> > Wiki: http://backuppc.wiki.sourceforge.net
>>> > Project: http://backuppc.sourceforge.net/
>>> >
>>> >
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Download Intel® Parallel Studio Eval
>>> Try the new software tools for yourself. Speed compiling, find bugs
>>> proactively, and fine-tune applications for parallel performance.
>>> See why Intel Parallel Studio got high marks during beta.
>>> http://p.sf.net/sfu/intel-sw-dev
>>> _______________________________________________
>>> BackupPC-users mailing list
>>> BackupPC-users AT lists.sourceforge DOT net
>>> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>>> Wiki: http://backuppc.wiki.sourceforge.net
>>> Project: http://backuppc.sourceforge.net/
>>
>>
>> ------------------------------------------------------------------------------
>> Download Intel® Parallel Studio Eval
>> Try the new software tools for yourself. Speed compiling, find bugs
>> proactively, and fine-tune applications for parallel performance.
>> See why Intel Parallel Studio got high marks during beta.
>> http://p.sf.net/sfu/intel-sw-dev
>> _______________________________________________
>> BackupPC-users mailing list
>> BackupPC-users AT lists.sourceforge DOT net
>> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
>> Wiki: http://backuppc.wiki.sourceforge.net
>> Project: http://backuppc.sourceforge.net/
>>
>>
>
------------------------------------------------------------------------------
Download Intel® Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
