We have been having all kinds of interesting adventures trying to get our
new DELL 2900 with 5TB with RHEL 4 (fully patched), latest engineering
level "lin" tape drivers (the only ones that would work with the 55-kernel
level). Server level is 5.4.0.0 - going to patch t o 5.4.0.3.
Our first 2900 ran for about a week and then starting having all kinds of
failures in the 5TB LZ, eventhough we all the stgvolumes (250GB each)
formatted without any problems. So we replaced the whole unit (after
talking with 5-folks at Dell, they just scratched their heads and sent us
another one). This one we partitioned into 1TB filesystems but the
volumes stayed the same 250GB each.
Now I am seeing these errors in the /var/logmessages file: When I
searched via Google, I found someone else with the same problem (posted
January 2007). They also posted a message here but I could not find a
response (http://www.mail-archive.com/adsm-l AT vm.marist DOT edu/msg70210.html)
Searching on IBM for "dsmserv: page allocation failure. order:6" didn't
yield much.
I see a lot of these message are related to memory. This box has 8GB or
RAM. One would think that was enough. The DBBuffers were at 1GB but I
dropped it down to 768M, just in case it is related.
Any thoughts ? Suggestions ?
====================================================
Jun 5 12:21:17 fireball kernel: dsmserv: page allocation failure.
order:6, mode:0xd0
Jun 5 12:21:17 fireball kernel:
Jun 5 12:21:17 fireball kernel: Call
Trace:<ffffffff8015e7a5>{__alloc_pages+777}
<ffffffff8015e83e>{__get_free_pages+11}
Jun 5 12:21:17 fireball kernel:
<ffffffffa025682f>{:lin_tape:tape_rw_buffer+136}
<ffffffffa0257817>{:lin_tape:lin_tape_drive_write+1203}
Jun 5 12:21:17 fireball kernel:
<ffffffffa0249e8d>{:lin_tape:lin_tape_write+341}
<ffffffff80179e9e>{vfs_write+207}
Jun 5 12:21:17 fireball kernel: <ffffffff80179f86>{sys_write+69}
<ffffffff8011026a>{system_call+126}
Jun 5 12:21:17 fireball kernel:
Jun 5 12:21:17 fireball kernel: Mem-info:
Jun 5 12:21:17 fireball kernel: Node 0 DMA per-cpu:
Jun 5 12:21:17 fireball kernel: cpu 0 hot: low 2, high 6, batch 1
Jun 5 12:21:17 fireball kernel: cpu 0 cold: low 0, high 2, batch 1
Jun 5 12:21:17 fireball kernel: cpu 1 hot: low 2, high 6, batch 1
Jun 5 12:21:17 fireball kernel: cpu 1 cold: low 0, high 2, batch 1
Jun 5 12:21:19 fireball kernel: cpu 2 hot: low 2, high 6, batch 1
Jun 5 12:21:19 fireball kernel: cpu 2 cold: low 0, high 2, batch 1
Jun 5 12:21:19 fireball kernel: cpu 3 hot: low 2, high 6, batch 1
Jun 5 12:21:19 fireball kernel: cpu 3 cold: low 0, high 2, batch 1
Jun 5 12:21:19 fireball kernel: Node 0 Normal per-cpu:
Jun 5 12:21:19 fireball kernel: cpu 0 hot: low 32, high 96, batch 16
Jun 5 12:21:19 fireball kernel: cpu 0 cold: low 0, high 32, batch 16
Jun 5 12:21:19 fireball kernel: cpu 1 hot: low 32, high 96, batch 16
Jun 5 12:21:19 fireball kernel: cpu 1 cold: low 0, high 32, batch 16
Jun 5 12:21:19 fireball kernel: cpu 2 hot: low 32, high 96, batch 16
Jun 5 12:21:19 fireball kernel: cpu 2 cold: low 0, high 32, batch 16
Jun 5 12:21:19 fireball kernel: cpu 3 hot: low 32, high 96, batch 16
Jun 5 12:21:19 fireball kernel: cpu 3 cold: low 0, high 32, batch 16
Jun 5 12:21:19 fireball kernel: Node 0 HighMem per-cpu: empty
Jun 5 12:21:19 fireball kernel:
Jun 5 12:21:19 fireball kernel: Free pages: 17308kB (0kB HighMem)
Jun 5 12:21:19 fireball kernel: Active:311114 inactive:1688601
dirty:229005 writeback:0 unstable:0 free:4327 slab:29289 mapped:310675
pagetables:1111
Jun 5 12:21:19 fireball kernel: Node 0 DMA free:10540kB min:4kB low:8kB
high:12kB active:0kB inactive:0kB present:16384kB pages_scanned:1885611
all_unreclaimable? yes
Jun 5 12:21:19 fireball kernel: protections[]: 0 0 0
Jun 5 12:21:19 fireball kernel: Node 0 Normal free:6768kB min:3016kB
low:6032kB high:9048kB active:1244456kB inactive:6754404kB
present:9158656kB pages_scanned:33 all_unreclaimable? no
Jun 5 12:21:19 fireball kernel: protections[]: 0 0 0
Jun 5 12:21:19 fireball kernel: Node 0 HighMem free:0kB min:128kB
low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Jun 5 12:21:19 fireball kernel: protections[]: 0 0 0
Jun 5 12:21:19 fireball kernel: Node 0 DMA: 1*4kB 1*8kB 0*16kB 1*32kB
0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 2*4096kB = 10540kB
Jun 5 12:21:19 fireball kernel: Node 0 Normal: 192*4kB 94*8kB 268*16kB
10*32kB 4*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6768kB
Jun 5 12:21:19 fireball kernel: Node 0 HighMem: empty
Jun 5 12:21:19 fireball kernel: Swap cache: add 96, delete 96, find 0/0,
race 0+0
Jun 5 12:21:19 fireball kernel: Free swap: 4192540kB
Jun 5 12:21:19 fireball kernel: 2293760 pages of RAM
Jun 5 12:21:19 fireball kernel: 252301 reserved pages
Jun 5 12:21:19 fireball kernel: 615492 pages shared
Jun 5 12:21:19 fireball kernel: 0 pages swap cached
Jun 5 12:21:19 fireball kernel: dsmserv: page allocation failure.
order:6, mode:0xd0
Jun 5 12:21:21 fireball kernel:
Jun 5 12:21:21 fireball kernel: Call
Trace:<ffffffff8015e7a5>{__alloc_pages+777}
<ffffffff8015e83e>{__get_free_pages+11}
Jun 5 12:21:21 fireball kernel:
<ffffffffa0256c8f>{:lin_tape:tape_init_sg+246}
<ffffffffa0257342>{:lin_tape:tape_build_sg+183}
Jun 5 12:21:21 fireball kernel:
<ffffffffa0257844>{:lin_tape:lin_tape_drive_write+1248}
Jun 5 12:21:21 fireball kernel:
<ffffffffa0249e8d>{:lin_tape:lin_tape_write+341}
<ffffffff80179e9e>{vfs_write+207}
Jun 5 12:21:21 fireball kernel: <ffffffff80179f86>{sys_write+69}
<ffffffff8011026a>{system_call+126}
Jun 5 12:21:21 fireball kernel:
Jun 5 12:21:21 fireball kernel: Mem-info:
Jun 5 12:21:21 fireball kernel: Node 0 DMA per-cpu:
Jun 5 12:21:21 fireball kernel: cpu 0 hot: low 2, high 6, batch 1
Jun 5 12:21:21 fireball kernel: cpu 0 cold: low 0, high 2, batch 1
Jun 5 12:21:21 fireball kernel: cpu 1 hot: low 2, high 6, batch 1
Jun 5 12:21:21 fireball kernel: cpu 1 cold: low 0, high 2, batch 1
Jun 5 12:21:21 fireball kernel: cpu 2 hot: low 2, high 6, batch 1
Jun 5 12:21:21 fireball kernel: cpu 2 cold: low 0, high 2, batch 1
Jun 5 12:21:21 fireball kernel: cpu 3 hot: low 2, high 6, batch 1
Jun 5 12:21:21 fireball kernel: cpu 3 cold: low 0, high 2, batch 1
Jun 5 12:21:21 fireball kernel: Node 0 Normal per-cpu:
Jun 5 12:21:21 fireball kernel: cpu 0 hot: low 32, high 96, batch 16
Jun 5 12:21:21 fireball kernel: cpu 0 cold: low 0, high 32, batch 16
Jun 5 12:21:21 fireball kernel: cpu 1 hot: low 32, high 96, batch 16
Jun 5 12:21:21 fireball kernel: cpu 1 cold: low 0, high 32, batch 16
Jun 5 12:21:21 fireball kernel: cpu 2 hot: low 32, high 96, batch 16
Jun 5 12:21:21 fireball kernel: cpu 2 cold: low 0, high 32, batch 16
Jun 5 12:21:21 fireball kernel: cpu 3 hot: low 32, high 96, batch 16
Jun 5 12:21:21 fireball kernel: cpu 3 cold: low 0, high 32, batch 16
Jun 5 12:21:21 fireball kernel: Node 0 HighMem per-cpu: empty
Jun 5 12:21:21 fireball kernel:
Jun 5 12:21:21 fireball kernel: Free pages: 17820kB (0kB HighMem)
Jun 5 12:21:21 fireball kernel: Active:311116 inactive:1688477
dirty:229520 writeback:0 unstable:0 free:4455 slab:29288 mapped:310675
pagetables:1111
Jun 5 12:21:21 fireball kernel: Node 0 DMA free:10540kB min:4kB low:8kB
high:12kB active:0kB inactive:0kB present:16384kB pages_scanned:1885612
all_unreclaimable? yes
Jun 5 12:21:21 fireball kernel: protections[]: 0 0 0
Jun 5 12:21:21 fireball kernel: Node 0 Normal free:7152kB min:3016kB
low:6032kB high:9048kB active:1244464kB inactive:6753964kB
present:9158656kB pages_scanned:0 all_unreclaimable? no
Jun 5 12:21:21 fireball kernel: protections[]: 0 0 0
Jun 5 12:21:21 fireball kernel: Node 0 HighMem free:0kB min:128kB
low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scanned:0
all_unreclaimable? no
Jun 5 12:21:21 fireball kernel: protections[]: 0 0 0
Jun 5 12:21:21 fireball kernel: Node 0 DMA: 1*4kB 1*8kB 0*16kB 1*32kB
0*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 1*2048kB 2*4096kB = 10540kB
Jun 5 12:21:21 fireball kernel: Node 0 Normal: 288*4kB 94*8kB 268*16kB
10*32kB 4*64kB 3*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 7152kB
Jun 5 12:21:21 fireball kernel: Node 0 HighMem: empty
Jun 5 12:21:21 fireball kernel: Swap cache: add 96, delete 96, find 0/0,
race 0+0
Jun 5 12:21:21 fireball kernel: Free swap: 4192540kB
Jun 5 12:21:21 fireball kernel: 2293760 pages of RAM
Jun 5 12:21:21 fireball kernel: 252301 reserved pages
Jun 5 12:21:21 fireball kernel: 615263 pages shared
Jun 5 12:21:21 fireball kernel: 0 pages swap cached
----------------------------------------------------
Zoltan Forray
Virginia Commonwealth University
Office of Technology Services
University Computing Center
e-mail: zforray AT vcu DOT edu
voice: 804-828-4807
|
