Hi folks, 

recently we've been seeing more and more problems with bacula-fd
messages in dmesg about a page allocation failure. 

Platform is centos 6.2 64 bit, Version 5.2.6 compiled from Source
using the stock distro gcc. 

We're using MariaDB 5.x as the db backend, here are some stats about
the bacula installation itself: 

Total clients:          128     Total bytes stored:     77.33 TB
Total files:            76905116      Database size:    111.72 GB

The server has 18G RAM, backup performance is generally rather good. 

Online backups are going to disk (33TB full of 39TB, xfs based FS). 

Here's the message from dmesg: 

May 28 12:24:16 bacula-server kernel: bacula-sd: page allocation
failure. order:1, mode:0x20
May 28 12:24:16 bacula-server kernel: Pid: 21923, comm: bacula-sd Not
tainted 2.6.32-71.29.1.el6.x86_64 #1
May 28 12:24:16 bacula-server kernel: Call Trace:
May 28 12:24:16 bacula-server kernel: <IRQ>  [<ffffffff8111eab6>]
__alloc_pages_nodemask+0x706/0x850
May 28 12:24:16 bacula-server kernel: [<ffffffff81156212>]
kmem_getpages+0x62/0x170
May 28 12:24:16 bacula-server kernel: [<ffffffff81156e2a>]
fallback_alloc+0x1ba/0x270
May 28 12:24:16 bacula-server kernel: [<ffffffff8115687f>] ?
cache_grow+0x2cf/0x320
May 28 12:24:16 bacula-server kernel: [<ffffffff81156ba9>]
____cache_alloc_node+0x99/0x160
May 28 12:24:16 bacula-server kernel: [<ffffffff8115750b>]
kmem_cache_alloc+0x11b/0x190
May 28 12:24:16 bacula-server kernel: [<ffffffff81404448>]
sk_prot_alloc+0x48/0x180
May 28 12:24:16 bacula-server kernel: [<ffffffff81404692>]
sk_clone+0x22/0x2a0
May 28 12:24:16 bacula-server kernel: [<ffffffff8144c276>]
inet_csk_clone+0x16/0xd0
May 28 12:24:16 bacula-server kernel: [<ffffffff814651c3>]
tcp_create_openreq_child+0x23/0x450
May 28 12:24:16 bacula-server kernel: [<ffffffff81462c0d>]
tcp_v4_syn_recv_sock+0x4d/0x280
May 28 12:24:16 bacula-server kernel: [<ffffffff81464f81>]
tcp_check_req+0x201/0x420
May 28 12:24:16 bacula-server kernel: [<ffffffff8146262b>]
tcp_v4_do_rcv+0x35b/0x430
May 28 12:24:16 bacula-server kernel: [<ffffffff8105c484>] ?
try_to_wake_up+0x284/0x380
May 28 12:24:16 bacula-server kernel: [<ffffffff81463e40>]
tcp_v4_rcv+0x5b0/0x7e0
May 28 12:24:16 bacula-server kernel: [<ffffffff8105c592>] ?
default_wake_function+0x12/0x20
May 28 12:24:16 bacula-server kernel: [<ffffffff81441e7d>]
ip_local_deliver_finish+0xdd/0x2d0
May 28 12:24:16 bacula-server kernel: [<ffffffff81442108>]
ip_local_deliver+0x98/0xa0
May 28 12:24:16 bacula-server kernel: [<ffffffff814415cd>]
ip_rcv_finish+0x12d/0x440
May 28 12:24:16 bacula-server kernel: [<ffffffff81441b55>]
ip_rcv+0x275/0x350
May 28 12:24:16 bacula-server kernel: [<ffffffff8140ffeb>]
netif_receive_skb+0x38b/0x670
May 28 12:24:16 bacula-server kernel: [<ffffffff8126ce48>] ?
is_swiotlb_buffer+0x18/0x50
May 28 12:24:16 bacula-server kernel: [<ffffffffa0269238>]
bnx2_poll_work+0xd18/0x1240 [bnx2]
May 28 12:24:16 bacula-server kernel: [<ffffffff8134a57a>] ?
scsi_next_command+0x4a/0x60
May 28 12:24:16 bacula-server kernel: [<ffffffff8134b36e>] ?
scsi_io_completion+0x35e/0x550
May 28 12:24:16 bacula-server kernel: [<ffffffff8105c846>] ?
update_curr+0xe6/0x1e0
May 28 12:24:16 bacula-server kernel: [<ffffffffa026979d>]
bnx2_poll_msix+0x3d/0xc0 [bnx2]
May 28 12:24:16 bacula-server kernel: [<ffffffff81410b73>]
net_rx_action+0x103/0x210
May 28 12:24:16 bacula-server kernel: [<ffffffff81073d67>]
__do_softirq+0xb7/0x1e0
May 28 12:24:16 bacula-server kernel: [<ffffffff810d8a10>] ?
handle_IRQ_event+0x60/0x170
May 28 12:24:16 bacula-server kernel: [<ffffffff81073dc4>] ?
__do_softirq+0x114/0x1e0
May 28 12:24:16 bacula-server kernel: [<ffffffff810142cc>]
call_softirq+0x1c/0x30
May 28 12:24:16 bacula-server kernel: [<ffffffff81015f35>]
do_softirq+0x65/0xa0
May 28 12:24:16 bacula-server kernel: [<ffffffff81073b65>]
irq_exit+0x85/0x90
May 28 12:24:16 bacula-server kernel: [<ffffffff814d0945>]
do_IRQ+0x75/0xf0
May 28 12:24:16 bacula-server kernel: [<ffffffff81013ad3>]
ret_from_intr+0x0/0x11

Any idea what's going wrong here? I don't see any significant swapping
or memory usage when this happens. I can provide a full dmesg dump if
that's helpful, I just didn't want to spam the list beyond measure
this time. 

All the best & thanks in advance, 

Uwe 

-- 
NIONEX --- Ein Unternehmen der Bertelsmann AG



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and 
threat landscape has changed and how IT managers can respond. Discussions 
will include endpoint security, mobile security and the latest in malware 
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users