BackupPC-users

[BackupPC-users] Too many links : where is the problem ?

2011-08-09 09:23:20
Subject: [BackupPC-users] Too many links : where is the problem ?
From: friedmann <backuppc-forum AT backupcentral DOT com>
To: backuppc-users AT lists.sourceforge DOT net
Date: Tue, 09 Aug 2011 06:21:56 -0700
thanks all for your answers, i will try to be more specific and precise. I was 
somehow hoping  that the problem was common, and that a simple answer existed.

i tried the piece of script you provided : 
perl -e 'for (my $i = 2; ; $i ++) { link "1", $i or die "Cant create ${i}th 
link: $!\n"; }' 

it returned, within 1-2 sec : 
"Cant create 32001th link: Trop de liens"
So yes, notice that i translated the error msg, for communication and googling 
purposes : "Trop de liens" is the actual message, and now i realize the 
importance of that detail ...

I launched one BackupPC_dump this night, with the limit moved to 100000 : it 
failed the same way, as expected.

[Question] How can i check that "the number of links on one or more pool files 
exceeds
HardLinkMax", and figure out what is special/unique about such a file ? 
[Question]

For your information, here are the last lines of the log generated by the 
"backupPC_dump -v -f 192.168.2.105" command : 
LOG
log ERROR: opendir 
(/var/lib/backuppc/pc/192.168.2.105/27/f%2fmedia%2fRhenovia/fVBox_Data_until_January_2011/fSBML_221010_IP3R_ForArnaud/fsbml2java_distrib/fDATE27octobre2010TIME09h17_huge9/fSim9996-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP32.4244620171E7e-9.0)
 failed
  same     777 1000/1000          86 
VBox_Data_until_January_2011/SBML_221010_IP3R_ForArnaud/sbml2java_distrib/DATE27octobre2010TIME09h17_huge9/Sim9996-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP32.4244620171E7e-9.0/Ca.txt
  same     777 1000/1000          80 
VBox_Data_until_January_2011/SBML_221010_IP3R_ForArnaud/sbml2java_distrib/DATE27octobre2010TIME09h17_huge9/Sim9996-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP32.4244620171E7e-9.0/IP3.txt
  same     777 1000/1000        2317 
VBox_Data_until_January_2011/SBML_221010_IP3R_ForArnaud/sbml2java_distrib/DATE27octobre2010TIME09h17_huge9/Sim9996-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP32.4244620171E7e-9.0/IP3R.txt
  same     777 1000/1000        2909 
VBox_Data_until_January_2011/SBML_221010_IP3R_ForArnaud/sbml2java_distrib/DATE27octobre2010TIME09h17_huge9/Sim9996-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP32.4244620171E7e-9.0/IP3R_Open.txt
  same     777 1000/1000        2316 
VBox_Data_until_January_2011/SBML_221010_IP3R_ForArnaud/sbml2java_distrib/DATE27octobre2010TIME09h17_huge9/Sim9996-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP32.4244620171E7e-9.0/StepSize.txt
  create d 700 1000/1000        4096 
VBox_Data_until_January_2011/SBML_221010_IP3R_ForArnaud/sbml2java_distrib/DATE27octobre2010TIME09h17_huge9/Sim9997-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP33.8881551803E7e-9.0
[Arrow] mkdir 
/var/lib/backuppc/pc/192.168.2.105/new//f%2fmedia%2fRhenovia/fVBox_Data_until_January_2011/fSBML_221010_IP3R_ForArnaud/fsbml2java_distrib/fDATE27octobre2010TIME09h17_huge9/fSim9997-singleInput1193.777e-9.0-J0_k188586.679e-3.0-IP33.8881551803E7e-9.0:
 Trop de liens at /usr/share/backuppc/lib/BackupPC/Xfer/RsyncFileIO.pm line 641
Parent read EOF from child: fatal error!
Done: 0 files, 0 bytes
[Arrow] Got fatal error during xfer (Child exited prematurely)
cmdSystemOrEval: about to system /bin/ping -c 1 192.168.2.105
cmdSystemOrEval: finished: got output PING 192.168.2.105 (192.168.2.105) 56(84) 
bytes of data.
64 bytes from 192.168.2.105: icmp_seq=1 ttl=64 time=1.96 ms

--- 192.168.2.105 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.960/1.960/1.960/0.000 ms

cmdSystemOrEval: about to system /bin/ping -c 1 192.168.2.105
cmdSystemOrEval: finished: got output PING 192.168.2.105 (192.168.2.105) 56(84) 
bytes of data.
64 bytes from 192.168.2.105: icmp_seq=1 ttl=64 time=1.23 ms

--- 192.168.2.105 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.238/1.238/1.238/0.000 ms

CheckHostAlive: returning 1.238
[Arrow] Backup aborted (Child exited prematurely)
Not saving this as a partial backup since it has fewer files than the prior one 
(got 3 and 0 files versus 3)
dump failed: Child exited prematurely
ENDLOG

"Trop de liens at /usr/share/backuppc/lib/BackupPC/Xfer/RsyncFileIO.pm line 
641" seems to be the most insightful message : would you tell me what this 
RsyncFileIO.pm deals with ?


I also noticed a good clue, pointing out the number of subdirectories
hence, i have counted the number of subdirectories that are on the backuped 
drive, using : "find . -type d | wc -l", that returned me :
82779

And, maybe more insightful, i have 32002 subdirectories in the same directory, 
i.e. in the "most populated" one (the one on which the backup process fails : 
/var/lib/backuppc/pc/192.168.2.105/new//f%2fmedia%2fRhenovia/fVBox_Data_until_January_2011/fSBML_221010_IP3R_ForArnaud/fsbml2java_distrib/fDATE27octobre2010TIME09h17_huge9)

I found out that, using ext3, a limit for the number of subdirectories in one 
directory could be 31998 
(http://superuser.com/questions/66331/what-is-the-maximum-number-of-folders-allowed-in-a-folder-in-linux).
I checked this with a shell script the following shell script : 
SCRIPT
#!/bin/bash
for i in {1..32010}
do
   mkdir test$i
done
END SCRIPT

It actually fails after 31998 created subdirectories on the ext3 partition
tested on the host's ntfs partition : the test is passed. I even tried with a 
value of 500000 : I stopped it after 296000 subdirectories were created.

[Idea] So i consider the problem as identified : It is relatively not related 
to backuppc itself, but deals with a sort of filesystems incompatibility. [Idea]
[Question] And finally, the only question that lasts is : how can i increase 
the number of subdirectories tolerated by ext3 ? [Question]

 :o 

Thanks again for your help, i hope there exists a workaround for that limit ... 
!

+----------------------------------------------------------------------
|This was sent by alefriedx AT gmail DOT com via Backup Central.
|Forward SPAM to abuse AT backupcentral DOT com.
+----------------------------------------------------------------------



------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model 
configuration take the hassle out of deploying and managing Subversion and 
the tools developers use with it. Learn more about uberSVN and get a free 
download at:  http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/