** email thread greatly trimmed **
Chris,
amDump time changed from 48 hours to 14 hours !!!!
On Tue, Mar 18, 2008 at 06:29:46PM -0400, Chris Hoogendyk wrote:
> Interesting that your LTO3 is doing no better than your LTO for speed. I
> would guess you should be getting 3 times the speed you are getting on
> LTO and well over 10 times the speed you are getting on LTO3.
It is curious, and indicated something common as the problem since
the drives are on separate bussed, the LTO being on a DIFF bus and
the LTO3 and multipack with the holding area (multiple spindles)
being on their own ports on an HBA that provides LVD/SE. We are
not using the standard SCSI port (meantioned later in this email).
Did a little looking in the HW manual and the online version of
the handbook, the PCI slots are not all created equally. The top
of the 4 slots will run at either 66 or 33 Mhz, the lower three
slots will run only at 33 Mhz.
We removed the now unused fiber HBA (no longer connected to massive
storage since we migrated the Lotus Notes function to the newer
T2000 system) and moved the LVD/SE HBA to the higher speed slot.
This largely if not completely solves the problems, I will provide
amdump output later in the email.
> Also, doing hardware compression, I would expect you to get more than
> you are on tape unless the data is relatively uncompressable. If Lotus
> keeps its stuff in compressed form, you could be running into slowdowns
> on the hardware compression.
I had thought of that also. We are using the Ultra device name which
should select (when there is an option) highest density and compression.
The tapes where labeled using that device, I think if that failed then
we'd have a tape being written without compression and would already
be avoiding the issues you suggest.
> With the speeds you are getting, I would be worried about shoe shining.
> Can you actually watch the tapes while they are running and see what's
> happening? That will take a slow speed and make it dismal, which is what
> you have. My AIT5 on my E250 is beating out your LTO3, and it shouldn't.
> You should be 2-3 times faster.
Unfortunately with the drives being imbedded in respective their libraries
there is no direct way for me to know if they are shoe shining, almost
makes one wish for a nice standup, glass fronted 9 track tape so we could
see what was going on. Maybe someone will start to build tapes with some
additional indicated LEDs on them.
> If any of that makes sense to you, and since you don't seem to be
> gaining much from compression, I'd be inclined to turn off compression
> altogether and see what happens. Of course, that can be a bit tricky
> with tapes that have already been used with compression. I think I saw a
> discussion somewhere about someone turning backflips to accomplish that.
> Might have been on the bacula list. Tape drives and OS level stuff
> translate almost completely between the two lists.
>
> If you can ever get any down time, I would be inclined to take one of
> the large notes partitions, recover it to the holding disk, and then
> play with dd'ing it to the tape until you figure out what's up.
One of the great values of having the amanda server separate from the
LNotes server is downtime. Fortunately I was able to juggle the HBAs
to new slots without interupting end-user services.
> If you use the standard SCSI port on the back of the E250, it is
> UltraSCSI, but only the earliest version. It can only do 40MB/s. It also
> supports single ended, wide or narrow SCSI devices. So, if there are
> other things also connected, and you are using that connection, then you
> probably have a problem with LTO3 from that perspective. I would double
> check and make sure you have connections and adapters that will support
> the throughput you need. I presume you have the E250 Server Owner's Guide?
You hit the nail on the head there.
> Good luck (intended positively ;-) ).
I see a great level of improvement, not only in the holding-area
to tape time but in the dump performance to the holding-area.
We also never filled the holding area, since we had free space as
data was able to get to tape we had greater concurrently as well
as the dumps being faster, presumably because the I/O to and from
the holding area was faster, as well as tape I/O being faster.
I do not know if we did much to improve the I/O to the LTO tape, which
we only run on weekends, having better holding area I/O will certainly
help, but there are no more "fast" PCI slots available and the LTO is
on the DIFF bus in the 33Mhz slot.
We will know more on Monday, or if it runs like it did last weekend
on Tuesday, not great for a weekly Saturday amdump.
These where the stats when the HBA was in its original position,
the slower 33 Mhz PCI slot.
Thank you for your help - and I will try to review and perhaps
change the compression settings on the LTO drive, though some of
these partitions will not fit on the tape unless compressed.
thank you,
Brian
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 0:05
Run Time (hrs:min) 48:10
Dump Time (hrs:min) 96:55 81:27 15:28
Output Size (meg) 548671.0 439359.7 109311.3
Original Size (meg) 548671.0 439359.7 109311.3
Avg Compressed Size (%) -- -- -- (level:#disks ...)
Filesystems Dumped 18 15 3 (1:3)
Avg Dump Rate (k/s) 1610.3 1534.3 2010.4
Tape Time (hrs:min) 42:23 36:20 6:03
Tape Size (meg) 548671.1 439359.8 109311.3
Tape Used (%) 142.1 113.8 28.3 (level:#disks ...)
Filesystems Taped 18 15 3 (1:3)
Avg Tp Write Rate (k/s) 3682.6 3439.9 5141.1
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
---------------------- --------------------------------------- -------------
nwcapp / 0 2104256 2104256 - 37:39 931.3 27:44 1264.8
nwcapp /nexport 0 84192 84192 - 0:25 3414.8 0:05 15806.0
wcapp / 0 11179776 11179776 - 480:04 388.1 21:23 8714.4
wcapp /db 0 71562976 71562976 - 399:16 2987.3 363:33 3280.8
wcapp /db2 0 9246208 9246208 - 467:42 329.5 254:30 605.5
wcapp /export 0 5415904 5415904 - 386:51 233.3 9:54 9120.8
wcnotes / 0 18889568 18889568 - 386:35 814.4 137:01 2297.7
wcnotes /export 1 7453632 7453632 - 76:36 1621.8 62:06 2000.6
wcnotes /maildb2/five 0 110204816 110204832 - 1176:59 1560.6 308:00 5963.5
wcnotes /maildb2/four 0 15375740 15375744 - 188:11 1361.8 72:52 3516.9
wcnotes /maildb2/one 0 62824432 62824448 - 520:52 2010.3 216:47 4829.9
wcnotes /maildb2/three 0 43245008 43245024 - 177:11 4068.0 147:22 4890.7
wcnotes /maildb2/two 0 56410712 56410720 - 303:56 3093.3 273:25 3438.7
wcnotes /space 0 895424 895424 - 33:58 439.4 98:14 151.9
wcnotes maildbAD 1 55402032 55402048 - 248:57 3709.1 257:20 3588.2
wcnotes maildbEK 1 49079128 49079136 - 602:24 1357.9 43:27 18827.4
wcnotes maildbLQ 0 32725550 32725568 - 254:06 2146.5 164:05 3324.1
wcnotes maildbRZ 0 9739760 9739776 - 73:24 2211.4 84:57 1910.9
(brought to you by Amanda version 2.4.4)
amdump stats after moving the HBA to a faster PCI slot.
Total Full Daily
-------- -------- --------
Estimate Time (hrs:min) 0:04
Run Time (hrs:min) 14:24
Dump Time (hrs:min) 34:53 9:24 25:30
Output Size (meg) 517292.1 117712.7 399579.4
Original Size (meg) 517292.1 117712.7 399579.4
Avg Compressed Size (%) -- -- -- (level:#disks ...)
Filesystems Dumped 18 6 12 (1:12)
Avg Dump Rate (k/s) 4217.4 3564.9 4457.7
Tape Time (hrs:min) 8:39 2:24 6:15
Tape Size (meg) 517292.2 117712.7 399579.5
Tape Used (%) 134.0 30.5 103.5 (level:#disks ...)
Filesystems Taped 18 6 12 (1:12)
Avg Tp Write Rate (k/s) 17019.4 13993.9 18177.1
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-KB OUT-KB COMP% MMM:SS KB/s MMM:SS KB/s
--------------------------------------------------------------- -------------
nwcapp / 1 640 640 - 0:49 13.1 0:02 374.6
nwcapp /nexport 0 84160 84160 - 0:42 2000.7 0:06 14894.3
wcapp / 1 32448 32448 - 2:42 200.8 0:03 10785.5
wcapp /db 1 71660896 71660896 - 324:07 3685.0 51:09 23349.2
wcapp /db2 0 9350528 9350528 - 74:37 2088.4 9:50 15842.3
wcapp /export 0 5438912 5438912 - 41:21 2192.1 5:09 17628.0
wcnotes / 1 5984 5984 - 3:01 33.1 0:02 2689.0
wcnotes /export 1 7140800 7140800 - 59:49 1989.7 21:03 5654.0
wcnotes /maildb2/five 1 110774048 110774080 - 382:58 4820.9 101:18 18226.6
wcnotes /maildb2/four 1 15241140 15241152 - 68:34 3704.8 14:11 17904.8
wcnotes /maildb2/one 0 63040640 63040640 - 230:50 4551.7 57:41 18213.3
wcnotes /maildb2/three 1 42685260 42685280 - 154:43 4598.2 63:04 11280.9
wcnotes /maildb2/two 1 56540368 56540384 - 223:19 4219.6 47:13 19956.1
wcnotes /space 1 61024 61024 - 0:25 2469.8 0:04 14426.6
wcnotes maildbAD 1 55578928 55578944 - 106:53 8666.0 33:34 27594.2
wcnotes maildbEK 1 49447768 49447776 - 202:30 4069.8 43:27 18968.0
wcnotes maildbLQ 0 32553520 32553536 - 143:50 3772.1 45:43 11867.5
wcnotes maildbRZ 0 10070000 10070016 - 72:12 2324.6 25:05 6691.6
(brought to you by Amanda version 2.4.4)
---
Brian R Cuttler brian.cuttler AT wadsworth DOT org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773
IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.
|
