[BackupPC-users] Problem: infinite backup on some hosts

Hello to everyone!

Sorry for my English.

We have a lot of backuppc server installations. And thousands of backup jobs.

But sometimes backup job runs infinitely, and killed by backuppc after some time (time out option in config file).

In final we have partial backup with dirs and file, but not all.

Log records:

2014-06-23 13:32:40 full backup started for directory /home/httpd/wgw-eu/logs

2014-06-23 14:27:18 full backup started for directory /home/httpd/wgw-eu/wg_wallet

2014-06-24 16:27:19 Aborting backup up after signal ALRM

2014-06-24 16:27:20 Got fatal error during xfer (aborted by signal=ALRM)

2014-06-24 16:27:20 Saved partial dump 0

2014-06-25 03:00:01 full backup started for directory /home/httpd/wgw-eu/logs

2014-06-25 03:32:58 full backup started for directory /home/httpd/wgw-eu/wg_wallet

Deleting and adding backup job didn't help.

Server side environment:

OS: Ubuntu 12.04.4 (precise) LTS (kernel: 3.2.0-56-generic)

backuppc: 3.2.1-2ubuntu1.1 (in Ubuntu notation)

openssh-client: 1:5.9p1-5ubuntu1.4 (in Ubuntu notation)

rsync: 3.0.9-1ubuntu1 (in Ubuntu notation), 3.0.9 protocol version 30 (in rsync --version)

rsync capabilities:

64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,

socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,

append, ACLs, xattrs, iconv, symtimes

Process list part (ps -eFH):

backuppc 61504 1 0 18908 19712 10 Apr11 ? 00:54:16 /usr/bin/perl /usr/share/backuppc/bin/BackupPC -d

backuppc 61507 61504 0 14098 13324 10 Apr11 ? 09:28:39 /usr/bin/perl /usr/share/backuppc/bin/BackupPC_trashClean

backuppc 15459 61504 0 32526 66276 15 03:00 ? 00:03:27 /usr/bin/perl /usr/share/backuppc/bin/BackupPC_dump jobname

backuppc 15512 15459 0 0 0 2 03:00 ? 00:00:58 [ssh] <defunct>

backuppc 15637 15459 0 0 0 13 03:00 ? 00:04:00 [BackupPC_dump] <defunct>

backuppc 19583 15459 0 11786 6508 2 03:32 ? 00:00:00 /usr/bin/ssh -q -x somehost.com /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . /home/httpd/wgw-eu/wg_wallet/

backuppc 19584 15459 0 42870 102052 4 03:32 ? 00:00:18 /usr/bin/perl /usr/share/backuppc/bin/BackupPC_dump jobname

Traces parts (strace -f -p <PID>):

/usr/bin/perl /usr/share/backuppc/bin/BackupPC_dump jobname (PID: 15459):

Process 15459 attached - interrupt to quit

select(8, [6], NULL, [6], NULL

/usr/bin/perl /usr/share/backuppc/bin/BackupPC_dump jobname (PID: 19584):

Process 19584 attached - interrupt to quit

select(16, [10], NULL, [10], NULL

/usr/bin/ssh -q -x somehost.com /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . /home/httpd/wgw-eu/wg_wallet/ (PID: 19583):

Process 19583 attached - interrupt to quit

select(7, [3 4], [], NULL, NULL

Client side environment:

OS: CentOS 5.10

openssh-server: 0:4.3p2-82.el5.x86_64 (in CentOS notation)

sudo: 0:1.7.2p1-29.el5_10.x86_64 (in CentOS notation), 1.7.2p1 (in sudo -V)

rsync: 0:3.0.6-4.el5_7.1.x86_64 (in CentOS notation), 3.0.6 protocol version 30 (in rsync --version)

rsync capabilities:

64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,

socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,

append, ACLs, xattrs, iconv, no symtimes

Process list part (ps -eFH):

root 6460 1 0 16202 1212 28 Apr15 ? 00:00:00 /usr/sbin/sshd

root 5111 6460 0 22719 3320 15 03:32 ? 00:00:00 sshd: bpc [priv]

bpc 5113 5111 0 22755 1868 31 03:32 ? 00:00:00 sshd: bpc@notty

root 5114 5113 0 26640 2252 7 03:32 ? 00:00:00 /usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . /home/httpd/wgw-eu/wg_wallet/

root 5115 5114 0 16597 1376 8 03:32 ? 00:00:00 /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . /home/httpd/wgw-eu/wg_wallet/

Traces parts (strace -f -p <PID>):

sshd: bpc [priv] (PID 5111):

Process 5111 attached - interrupt to quit

read(6,

sshd: bpc@notty (PID 5113):

Process 5113 attached - interrupt to quit

select(12, [3 6 9 11], [], NULL, NULL

/usr/bin/sudo /usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . /home/httpd/wgw-eu/wg_wallet/ (PID 5114):

Process 5114 attached - interrupt to quit

select(7, [6], [], NULL, {0, 881000}) = 0 (Timeout)

select(7, [6], [], NULL, {1, 0}) = 0 (Timeout)

...

/usr/bin/rsync --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --checksum-seed=32761 --ignore-times . /home/httpd/wgw-eu/wg_wallet/ (PID 5115):

Process 5115 attached - interrupt to quit

select(1, [0], [], NULL, {22, 636000}) = 0 (Timeout)

select(1, [0], [], NULL, {60, 0}

Config files (also attached to email):

grep -vE '(^#|^$)' /etc/backuppc/jobname.pl:

$Conf{RsyncShareName} = [

'/home/httpd/wgw-eu/logs',

'/home/httpd/wgw-eu/wg_wallet'

];

$Conf{BackupFilesExclude} = {

'*' => [

'.subversion'

]

};

$Conf{BlackoutPeriods} = [

{

'hourEnd' => '2.9',

'weekDays' => [

'0',

'1',

'2',

'3',

'4',

'5',

'6'

'hourBegin' => '5.8'

}

];

$Conf{FullKeepCnt} = [

'2',

'0',

'1'

];

$Conf{ClientNameAlias} = 'somehost.com';

grep -vE '(^#|^$)' /etc/backuppc/config.pl:

$ENV{'PATH'} = '/bin:/usr/bin';

delete @ENV{'IFS', 'CDPATH', 'ENV', 'BASH_ENV'};

$Conf{ServerHost} = `hostname`;

chomp($Conf{ServerHost});

$Conf{ServerPort} = -1;

$Conf{ServerMesgSecret} = '';

$Conf{MyPath} = '/bin';

$Conf{UmaskMode} = 23;

$Conf{WakeupSchedule} = [7.35, 8, 8.35, 9, 9.35, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 0.51, 1, 1.51, 2, 3, 4, 5, 6, 7];

$Conf{MaxBackups} = 25;

$Conf{MaxUserBackups} = 10;

$Conf{MaxPendingCmds} = 40;

$Conf{CmdQueueNice} = 10;

$Conf{MaxBackupPCNightlyJobs} = 8;

$Conf{BackupPCNightlyPeriod} = 1;

$Conf{MaxOldLogFiles} = 90;

$Conf{DfPath} = '/bin/df';

$Conf{DfCmd} = '$dfPath $topDir';

$Conf{SplitPath} = '/usr/bin/split';

$Conf{ParPath} = undef;

$Conf{CatPath} = '/bin/cat';

$Conf{GzipPath} = '/usr/bin/pigz';

$Conf{Bzip2Path} = '/usr/bin/lbzip2';

$Conf{DfMaxUsagePct} = 97;

$Conf{TrashCleanSleepSec} = 300;

$Conf{DHCPAddressRanges} = [];

$Conf{BackupPCUser} = 'backuppc';

$Conf{TopDir} = '/var/lib/backuppc';

$Conf{ConfDir} = '/etc/backuppc';

$Conf{LogDir} = '/var/lib/backuppc/log';

$Conf{InstallDir} = '/usr/share/backuppc';

$Conf{CgiDir} = '/usr/share/backuppc/cgi-bin';

$Conf{BackupPCUserVerify} = 1;

$Conf{HardLinkMax} = 31999;

$Conf{PerlModuleLoad} = undef;

$Conf{ServerInitdPath} = '';

$Conf{ServerInitdStartCmd} = '';

$Conf{FullPeriod} = 6.97;

$Conf{IncrPeriod} = 0.79;

$Conf{FullKeepCnt} = [4, 0, 6, 0, 0, 2];

$Conf{FullKeepCntMin} = 1;

$Conf{FullAgeMax} = 90;

$Conf{IncrKeepCnt} = 12;

$Conf{IncrKeepCntMin} = 1;

$Conf{IncrAgeMax} = 30;

$Conf{IncrLevels} = [1];

$Conf{BackupsDisable} = 0;

$Conf{PartialAgeMax} = 3;

$Conf{IncrFill} = 0;

$Conf{RestoreInfoKeepCnt} = 50;

$Conf{ArchiveInfoKeepCnt} = 50;

$Conf{BackupFilesOnly} = {

'/root/' => [

'.bash_history',

'.mysql_history',

'.my.cnf',

'bin/'

'/var/log/' => [

'messages',

'syslog',

'dmesg',

'secure'

]

};

$Conf{BackupFilesExclude} = {

'*' => [

'.subversion'

]

};

$Conf{BlackoutBadPingLimit} = 3;

$Conf{BlackoutGoodCnt} = 0;

$Conf{BlackoutPeriods} = [

{

hourBegin => 6.5,

hourEnd => 0.9,

weekDays => [0, 1, 2, 3, 4, 5, 6],

];

$Conf{BackupZeroFilesIsFatal} = 0;

$Conf{XferMethod} = 'rsync';

$Conf{XferLogLevel} = 1;

$Conf{ClientCharset} = '';

$Conf{ClientCharsetLegacy} = 'iso-8859-1';

$Conf{SmbShareName} = 'C$';

$Conf{SmbShareUserName} = '';

$Conf{SmbSharePasswd} = '';

$Conf{SmbClientPath} = '/usr/bin/smbclient';

$Conf{SmbClientFullCmd} = '$smbClientPath \\\\$host\\$shareName'

. ' $I_option -U $userName -E -d 1 -N'

. ' -c tarmode\\ full -Tc$X_option - $fileList';

$Conf{SmbClientIncrCmd} = '$smbClientPath \\\\$host\\$shareName'

. ' $I_option -U $userName -E -d 1 -N'

. ' -c tarmode\\ full -TcN$X_option $timeStampFile - $fileList';

$Conf{SmbClientRestoreCmd} = '$smbClientPath \\\\$host\\$shareName'

. ' $I_option -U $userName -E -d 1 -N'

. ' -c tarmode\\ full -Tx -';

$Conf{TarShareName} = '/';

$Conf{TarClientCmd} = '$sshPath -x -n $host'

. ' env LC_ALL=C $tarPath -c -v -f - -C $shareName+'

. ' --totals';

$Conf{TarFullArgs} = '$fileList+';

$Conf{TarIncrArgs} = '--newer=$incrDate+ $fileList+';

$Conf{TarClientRestoreCmd} = '$sshPath -x $host'

. ' env LC_ALL=C $tarPath -x -p --numeric-owner --same-owner'

. ' -v -f - -C $shareName+';

$Conf{TarClientPath} = '/usr/bin/sudo /bin/tar';

$Conf{RsyncClientPath} = '/usr/bin/sudo /usr/bin/rsync';

$Conf{RsyncClientCmd} = '$sshPath -q -x $host $rsyncPath $argList+';

$Conf{RsyncClientRestoreCmd} = '$sshPath -q -x $host $rsyncPath $argList+';

$Conf{RsyncShareName} = [

'EMPTY_CONFIG_USED_CONFIGURE_ME_PLEASE',

'/etc/',

'/var/spool/cron/',

'/root/',

'/var/log/'

];

$Conf{RsyncdClientPort} = 873;

$Conf{RsyncdUserName} = '';

$Conf{RsyncdPasswd} = '';

$Conf{RsyncdAuthRequired} = 1;

$Conf{RsyncCsumCacheVerifyProb} = 0.01;

$Conf{RsyncArgs} = [

# Do not edit these!

'--numeric-ids',

'--perms',

'--owner',

'--group',

'-D',

'--links',

'--hard-links',

'--times',

'--block-size=2048',

'--recursive',

# Rsync >= 2.6.3 supports the --checksum-seed option

# which allows rsync checksum caching on the server.

# Uncomment this to enable rsync checksum caching if

# you have a recent client rsync version and you want

# to enable checksum caching.

'--checksum-seed=32761',

];

$Conf{RsyncArgsExtra} = [];

$Conf{RsyncRestoreArgs} = [

# Do not edit these!

'--numeric-ids',

'--perms',

'--owner',

'--group',

'-D',

'--links',

'--hard-links',

'--times',

'--block-size=2048',

'--relative',

'--ignore-times',

'--recursive',

# Rsync >= 2.6.3 supports the --checksum-seed option

# which allows rsync checksum caching on the server.

# Uncomment this to enable rsync checksum caching if

# you have a recent client rsync version and you want

# to enable checksum caching.

#'--checksum-seed=32761',

# Add additional arguments here

];

$Conf{FtpShareName} = '';

$Conf{FtpUserName} = '';

$Conf{FtpPasswd} = '';

$Conf{FtpPassive} = 1;

$Conf{FtpBlockSize} = 10240;

$Conf{FtpPort} = 21;

$Conf{FtpTimeout} = 120;

$Conf{FtpFollowSymlinks} = 0;

$Conf{ArchiveDest} = '/tmp';

$Conf{ArchiveComp} = 'gzip';

$Conf{ArchivePar} = 0;

$Conf{ArchiveSplit} = 0;

$Conf{ArchiveClientCmd} = '$Installdir/bin/BackupPC_archiveHost'

. ' $tarCreatePath $splitpath $parpath $host $backupnumber'

. ' $compression $compext $splitsize $archiveloc $parfile *';

$Conf{SshPath} = '/usr/bin/ssh' if -x '/usr/bin/ssh';

$Conf{NmbLookupPath} = '/usr/bin/nmblookup';

$Conf{NmbLookupCmd} = '$nmbLookupPath -A $host';

$Conf{NmbLookupFindHostCmd} = '$nmbLookupPath $host';

$Conf{FixedIPNetBiosNameCheck} = 0;

$Conf{PingPath} = '/bin/ping';

$Conf{Ping6Path} = '';

$Conf{PingCmd} = '$pingPath -c 1 -w 3 $host';

$Conf{PingMaxMsec} = 200;

$Conf{CompressLevel} = 3;

$Conf{ClientTimeout} = 93600;

$Conf{MaxOldPerPCLogFiles} = 12;

$Conf{DumpPreUserCmd} = undef;

$Conf{DumpPostUserCmd} = undef;

$Conf{DumpPreShareCmd} = undef;

$Conf{DumpPostShareCmd} = undef;

$Conf{RestorePreUserCmd} = undef;

$Conf{RestorePostUserCmd} = undef;

$Conf{ArchivePreUserCmd} = undef;

$Conf{ArchivePostUserCmd} = undef;

$Conf{UserCmdCheckStatus} = 1;

$Conf{ClientNameAlias} = undef;

$Conf{SendmailPath} = '/usr/sbin/sendmail';

$Conf{EMailNotifyMinDays} = 0.5;

$Conf{EMailFromUserName} = 'backuppc';

$Conf{EMailAdminUserName} = 'backuppc';

$Conf{EMailUserDestDomain} = '';

$Conf{EMailNoBackupEverSubj} = undef;

$Conf{EMailNoBackupEverMesg} = undef;

$Conf{EMailNotifyOldBackupDays} = 1.4;

$Conf{EMailNoBackupRecentSubj} = undef;

$Conf{EMailNoBackupRecentMesg} = undef;

$Conf{EMailNotifyOldOutlookDays} = 5.0;

$Conf{EMailOutlookBackupSubj} = undef;

$Conf{EMailOutlookBackupMesg} = undef;

$Conf{EMailHeaders} = <<EOF;

MIME-Version: 1.0

Content-Type: text/plain; charset="utf-8"

EOF

$Conf{CgiAdminUserGroup} = '';

$Conf{CgiAdminUsers} = '*';

$Conf{CgiURL} = 'http://'.$Conf{ServerHost}.'/backuppc/index.cgi';

$Conf{Language} = 'en';

$Conf{CgiUserHomePageCheck} = '';

$Conf{CgiUserUrlCreate} = 'mailto:%s';

$Conf{CgiDateFormatMMDD} = 2;

$Conf{CgiNavBarAdminAllHosts} = 1;

$Conf{CgiSearchBoxEnable} = 1;

$Conf{CgiNavBarLinks} = [

{

link => "?action="">

lname => "Documentation", # actually displays $Lang->{Documentation}

{

link => "http://backuppc.wiki.sourceforge.net",

name => "Wiki", # displays literal "Wiki"

{

link => "http://backuppc.sourceforge.net",

name => "SourceForge", # displays literal "SourceForge"

];

$Conf{CgiStatusHilightColor} = {

Reason_backup_failed => '#ffcccc',

Reason_backup_done => '#ccffcc',

Reason_no_ping => '#ffff99',

Reason_backup_canceled_by_user => '#ff9900',

Status_backup_in_progress => '#66cc99',

Disabled_OnlyManualBackups => '#d1d1d1',

Disabled_AllBackupsDisabled => '#d1d1d1',

};

$Conf{CgiHeaders} = '<meta http-equiv="pragma" content="no-cache">';

$Conf{CgiImageDir} = '/usr/share/backuppc/image';

$Conf{CgiExt2ContentType} = { };

$Conf{CgiImageDirURL} = '/backuppc/image';

$Conf{CgiCSSFile} = 'BackupPC_stnd.css';

$Conf{CgiUserConfigEditEnable} = 1;

$Conf{CgiUserConfigEdit} = {

FullPeriod => 1,

IncrPeriod => 1,

FullKeepCnt => 1,

FullKeepCntMin => 1,

FullAgeMax => 1,

IncrKeepCnt => 1,

IncrKeepCntMin => 1,

IncrAgeMax => 1,

IncrLevels => 1,

IncrFill => 1,

PartialAgeMax => 1,

RestoreInfoKeepCnt => 1,

ArchiveInfoKeepCnt => 1,

BackupFilesOnly => 1,

BackupFilesExclude => 1,

BackupsDisable => 1,

BlackoutBadPingLimit => 1,

BlackoutGoodCnt => 1,

BlackoutPeriods => 1,

BackupZeroFilesIsFatal => 1,

ClientCharset => 1,

ClientCharsetLegacy => 1,

XferMethod => 1,

XferLogLevel => 1,

SmbShareName => 1,

SmbShareUserName => 1,

SmbSharePasswd => 1,

SmbClientFullCmd => 0,

SmbClientIncrCmd => 0,

SmbClientRestoreCmd => 0,

TarShareName => 1,

TarFullArgs => 1,

TarIncrArgs => 1,

TarClientCmd => 0,

TarClientRestoreCmd => 0,

TarClientPath => 0,

RsyncShareName => 1,

RsyncdClientPort => 1,

RsyncdPasswd => 1,

RsyncdUserName => 1,

RsyncdAuthRequired => 1,

RsyncCsumCacheVerifyProb => 0,

RsyncArgs => 1,

RsyncArgsExtra => 1,

RsyncRestoreArgs => 1,

RsyncClientCmd => 0,

RsyncClientRestoreCmd => 0,

RsyncClientPath => 0,

FtpShareName => 1,

FtpUserName => 1,

FtpPasswd => 1,

FtpBlockSize => 1,

FtpPort => 1,

FtpTimeout => 1,

FtpFollowSymlinks => 1,

FtpRestoreEnabled => 1,

ArchiveDest => 1,

ArchiveComp => 1,

ArchivePar => 1,

ArchiveSplit => 1,

ArchiveClientCmd => 0,

FixedIPNetBiosNameCheck => 1,

NmbLookupCmd => 0,

NmbLookupFindHostCmd => 0,

PingMaxMsec => 1,

PingCmd => 0,

ClientTimeout => 1,

MaxOldPerPCLogFiles => 1,

CompressLevel => 1,

ClientNameAlias => 1,

DumpPreUserCmd => 0,

DumpPostUserCmd => 0,

RestorePreUserCmd => 0,

RestorePostUserCmd => 0,

ArchivePreUserCmd => 0,

ArchivePostUserCmd => 0,

DumpPostShareCmd => 0,

DumpPreShareCmd => 0,

UserCmdCheckStatus => 0,

EMailNotifyMinDays => 1,

EMailFromUserName => 1,

EMailAdminUserName => 1,

EMailUserDestDomain => 1,

EMailNoBackupEverSubj => 1,

EMailNoBackupEverMesg => 1,

EMailNotifyOldBackupDays => 1,

EMailNoBackupRecentSubj => 1,

EMailNoBackupRecentMesg => 1,

EMailNotifyOldOutlookDays => 1,

EMailOutlookBackupSubj => 1,

EMailOutlookBackupMesg => 1,

EMailHeaders => 1,

};

config.pl
Description: Text Data

jobname.pl
Description: Text Data

------------------------------------------------------------------------------
Open source business process management suite built on Java and Eclipse
Turn processes into business applications with Bonita BPM Community Edition
Quickly connect people, data, and systems into organized workflows
Winner of BOSSIE, CODIE, OW2 and Gartner awards
http://p.sf.net/sfu/Bonitasoft

_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/