BackupPC-users

Re: [BackupPC-users] 100,000+ errors in last nights backup

2009-08-13 02:26:48
Subject: Re: [BackupPC-users] 100,000+ errors in last nights backup
From: Steve Blackwell <zephod AT cfl.rr DOT com>
To: backuppc-users AT lists.sourceforge DOT net
Date: Thu, 13 Aug 2009 00:26:20 -0400
Holger,

I started to reply to your e-mail but my system crashed. The messages
log suggests that backuppc may have been the culprit. See below.

On Thu, 13 Aug 2009 02:19:25 +0200 Holger Parplies <wbppc AT parplies DOT de>
wrote:

> Hi,
> 
> Jeffrey J. Kosowsky wrote on 2009-08-12 18:12:09 -0400 [Re:
> [BackupPC-users] 100,000+ errors in last nights backup]:
> > Steve Blackwell wrote at about 14:33:54 -0400 on Wednesday, August
> > 12, 2009:
> >  > >  > > Steve Blackwell wrote at about 11:18:36 -0400 on
> >  > >  > > Wednesday, August 12, 2009:
> >  > >  > >  > On Wed, 12 Aug 2009 10:06:37 -0400 "Jeffrey J.
> >  > >  > >  > Kosowsky" <backuppc AT kosowsky DOT org> wrote:
> >  > >  > >  > > Try manually doing something like the following from
> >  > >  > >  > > the command line: link
> >  > >  > >  > > b32585c3cc30b7ebb556d335a08554e3 
> > /media/disk/pc/steve/151/f%2f/froot/f.gconf/fdesktop/fgnome/faccessibility/fkeyboard/f%25gconf.xml
> >  > >  > >  > 
> >  > >  > >  > Where do I need to be to run this? 
> 
> you need to be in $TopDir/cpool/b/3/2 ... or rather, it should be
> 
>       sudo -u backuppc ln
> $TopDir/cpool/b/3/2/b32585c3cc30b7ebb556d335a08554e3 
> /media/disk/pc/steve/151/f%2f/froot/f.gconf/fdesktop/fgnome/faccessibility/fkeyboard/f%25gconf.xml

my $TopDir is /media/disk. I ran the command above and it appeared to
work OK. I don't have the exact result anymore because of the crash.

> (or use 'link' if you prefer). You need to get $TopDir right though.
> See below.
> 
> >  > # sudo -u backuppc link
> >  > b32585c3cc30b7ebb556d335a08554e3 
> > /media/disk/pc/steve/151/f%2f/froot/f.gconf/fdesktop/fgnome/faccessibility/fkeyboard/f%25gconf.xml
> >  > link: cannot create link
> >  > 
> > `/media/disk/pc/steve/151/f%2f/froot/f.gconf/fdesktop/fgnome/faccessibility/fkeyboard/f%25gconf.xml'
> >  > to `b32585c3cc30b7ebb556d335a08554e3': No such file or directory
> 
> Yes, most directories won't contain a file with that name ;-).
> 
> >  > Jeffrey, I'm now thinking that two backups have somehow been
> >  > scheduled at the same. See the Les Miskell thread.
> 
> I agree with that except for the name. But I don't think it's the
> problem, at least not the one you're looking for. I can't imagine why
> two backups of the same host would be scheduled simultaneously, and I
> don't think it's a good idea to do so :-). BackupPC doesn't usually
> do this. Have you changed the code in any way? Have you seen such a
> thing happen before?

Could I have started the backuppc service twice somehow? I certainly
haven't changed any of the code. I haven't written any perl since perl
4 circa 1995.


8><---- [snip]

> So, I believe we're back to the issue of what you did wrong when
> moving $TopDir. I don't remember reading which version of BackupPC

When moving $TopDir? I haven't moved it. It's always been /media/disk
ever since I installed backuppc.

> you are using. What did you do to move $TopDir? Have a look at the
> old location, wherever that was. Are there files with recent
> modification times below the cpool/ directory?
> Below /media/disk/cpool? BackupPC_nightly reports files there, but

# ls -l /media/disk
total 48
drwxr-x--- 18 backuppc root  4096 2009-08-12 20:00 cpool
drwx------  2 root     root 16384 2008-07-16 23:54 lost+found
drwxr-x---  4 backuppc root  4096 2009-08-09 23:06 pc
drwxr-x---  2 backuppc root  4096 2008-07-26 18:53 pool
drwxr-x---  2 backuppc root  4096 2009-08-11 01:25 trash

So cpool has today's date

]# ls -l /media/disk/cpool
total 128
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 0
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 1
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 2
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 3
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 4
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 5
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 6
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 7
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 8
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 9
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 a
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 b
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 c
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 d
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 e
drwxr-x--- 18 backuppc backuppc 4096 2009-06-28 15:30 f

Nothing since 6/28.

This is where I got to when the machine crashed. Looking
at /var/log/messages I see this:

Aug 12 20:00:12 steve kernel: ------------[ cut here ]------------
Aug 12 20:00:12 steve kernel: WARNING: at lib/list_debug.c:51
list_del+0x41/0x60() 
Aug 12 20:00:12 steve kernel: Hardware name: To Be Filled By O.E.M. 
Aug 12 20:00:12 steve kernel: list_del corruption. next->prev should be
c14c2838, but was c14c2878 
Aug 12 20:00:12 steve kernel: Modules linked in: snd_seq_midi vfat fat
autofs4 w83627ehf hwmon_vid hwmon nf_conntrack_netbios_ns
nf_conntrack_ipv6 ip6t_ipv6header ip6t_REJECT ip6table_filter
ip6_tables ipv6 p4_clockmod fuse dm_multipath uinput snd_usb_audio
snd_usb_lib snd_emu10k1_synth snd_emux_synth snd_seq_virmidi
snd_seq_midi_emul snd_emu10k1 snd_rawmidi snd_ac97_codec ac97_bus
snd_seq_dummy snd_hda_intel snd_seq_oss snd_seq_midi_event snd_seq
snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_device snd_timer
snd_page_alloc snd_util_mem snd_hwdep floppy ppdev snd iTCO_wdt
iTCO_vendor_support firewire_ohci emu10k1_gp gameport firewire_core
crc_itu_t pcspkr nvidia(P) i2c_i801 soundcore parport_pc parport
i2c_core sky2 usb_storage sata_sil pata_it821x pata_sil680 [last
unloaded: microcode] 
Aug 12 20:00:12 steve kernel: Pid: 17353, comm: BackupPC_tarExt
Tainted: P   M 2.6.27.29-170.2.78.fc10.i686 #1 
Aug 12 20:00:12 steve kernel: [<c042db38>] warn_slowpath+0x69/0x89 
Aug 12 20:00:12 steve kernel: [<c04d3866>] ? brelse+0xc/0xe 
Aug 12 20:00:12 steve kernel: [<c04d5a8a>] ?
ext3_get_blocks_handle+0x772/0x791 
Aug 12 20:00:12 steve kernel: [<c04770e0>] ?
mark_page_accessed+0x4c/0x57 
Aug 12 20:00:12 steve kernel: [<c04af9bb>] ?
__find_get_block+0x1ab/0x1b5 
Aug 12 20:00:12 steve kernel: [<c04d53a1>] ?
ext3_get_blocks_handle+0x89/0x791 
Aug 12 20:00:12 steve kernel: [<c04770e0>] ?
mark_page_accessed+0x4c/0x57 
Aug 12 20:00:12 steve kernel: [<c04af9bb>] ?
__find_get_block+0x1ab/0x1b5 
Aug 12 20:00:12 steve kernel: [<c04e2458>] ?
do_get_write_access+0x339/0x36a 
Aug 12 20:00:12 steve kernel: [<c052394d>] list_del+0x41/0x60 
Aug 12 20:00:12 steve kernel: [<c0473516>]
__rmqueue_smallest+0x82/0x12d 
Aug 12 20:00:12 steve kernel: [<c04735d8>] __rmqueue+0x17/0x1a6 
Aug 12 20:00:12 steve kernel: [<c0473795>] rmqueue_bulk+0x2e/0x6c 
Aug 12 20:00:12 steve kernel: [<c0474565>]
get_page_from_freelist+0x22b/0x427 
Aug 12 20:00:12 steve kernel: [<c04749c8>]
__alloc_pages_internal+0xb0/0x399 
Aug 12 20:00:12 steve kernel: [<c06aa932>] ? _cond_resched+0x8/0x32 
Aug 12 20:00:12 steve kernel: [<c0483864>] ? anon_vma_prepare+0x22/0xb3
Aug 12 20:00:12 steve kernel: [<c047d673>] handle_mm_fault+0x14c/0x6d1
Aug 12 20:00:12 steve kernel: [<c04a5eb9>] ? mntput_no_expire+0x1a/0xf7
Aug 12 20:00:12 steve kernel: [<c051b079>] ?
_atomic_dec_and_lock+0x29/0x4c 
Aug 12 20:00:12 steve kernel: [<c0498d1e>] ? mntput+0x13/0x15 
Aug 12 20:00:12 steve kernel: [<c0498dfa>] ? path_put+0x15/0x18 
Aug 12 20:00:12 steve kernel: [<c04a5eb9>] ? mntput_no_expire+0x1a/0xf7
Aug 12 20:00:12 steve kernel: [<c06adaef>] do_page_fault+0x33d/0x710
Aug 12 20:00:12 steve kernel: [<c04432d1>] ?
sched_clock_cpu+0x12c/0x13b 
Aug 12 20:00:12 steve kernel: [<c041fbc4>] ? update_curr+0x8d/0xf0 
Aug 12 20:00:12 steve kernel: [<c0444627>] ? getnstimeofday+0x54/0xe1
Aug 12 20:00:12 steve kernel: [<c0414a2b>] ? lapic_next_event+0x15/0x1c
Aug 12 20:00:12 steve kernel: [<c06abc89>] ? _spin_lock+0x8/0xb 
Aug 12 20:00:12 steve kernel: [<c04808f0>] ? vma_adjust+0x320/0x380 
Aug 12 20:00:12 steve kernel: [<c04fe2e5>] ?
selinux_vm_enough_memory+0x57/0x5f 
Aug 12 20:00:12 steve kernel: [<c0482636>] ? do_mremap+0x29b/0x4c8 
Aug 12 20:00:12 steve kernel: [<c0498dfa>] ? path_put+0x15/0x18 
Aug 12 20:00:12 steve kernel: [<c0461ccd>] ?
audit_syscall_exit+0xb2/0xc7 
Aug 12 20:00:12 steve kernel: [<c06ad7b2>] ? do_page_fault+0x0/0x710
Aug 12 20:00:12 steve kernel: [<c06ac08a>] error_code+0x72/0x78 
Aug 12 20:00:12 steve kernel: ======================= 
Aug 12 20:00:12 steve kernel: ---[ end trace 66ead324e761b4c4 ]---

Note the line:
Aug 12 20:00:12 steve kernel: Pid: 17353, comm: BackupPC_tarExt
Tainted: P   M 2.6.27.29-170.2.78.fc10.i686 #1 

This is repeated for the 2nd CPU. Does this have anything to do with
that link we created?

> you might have copied them there at some point. If you
> umount /dev/sdd2, what's below /media/disk? A cpool/ directory?
> 
> > You may also want to look at ownership/permissions
> > in your pc tree to make sure that the relevant files and directories
> > are writable by user backuppc.
> 
> SElinux? But why should BackupPC_dump succeed and only BackupPC_link
> fail?
> 
> 
> All of this sounds rather strange. More information on your setup
> would probably help clear things up. You should also avoid starting
> BackupPC when $TopDir is not available (e.g. put something like
> '[ -f /media/disk/.this_is_the_real_disk ] || exit 0' in your
> /etc/init.d/backuppc and create the file on your disk). Making the
> mount point (not the root of the file system mounted on it!) an empty
> directory to which backuppc has no write permission should accomplish
> the same.
> 
> Regards,
> Holger

As I said in an earlier e-mail, the disk I backup to is an external 1TB
USB drive. My computer is old enough that the BIOS locks up when
booting with this disk connected so I have to unplug it to boot.
Backuppc fails during the boot process because it can't make the
required links. 

After the machine has booted, I plug in the USB drive and it gets
automatically mounted as disk on the mount point /media. Then I have to
restart the backuppc service.

Steve.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

<Prev in Thread] Current Thread [Next in Thread>