Re: [Bacula-users] Quantum Scalar i500 slow write speed
2010-08-06 19:49:11
On Aug 6, 2010, at 4:48 AM, Christian Gaul wrote:
> Am 05.08.2010 21:56, schrieb Henry Yen:
>> On Thu, Aug 05, 2010 at 17:17:39PM +0200, Christian Gaul wrote:
>>
[[...]]
>>
>>>> /dev/urandom seems to measure about 3MB/sec or thereabouts, so creating
>>>> a large "uncompressible" file could be done sort of like:
>>>>
>>>> dd if=/dev/urandom of=tempchunk count=1048576
>>>> cat tempchunk tempchunk tempchunk tempchunk tempchunk tempchunk > bigfile
>>>>
>>>>
>>> cat-ting random data a couple of times to make one big random file wont
>>> really work, unless the size of the chunks is way bigger than the
>>> "library" size of the compression algorithm.
>>>
>> Reason 1: the example I gave yields a file size for "tempchunk" of 512MB,
>> not 1MB, as given in your counter-example. I agree that (at least
>> now-a-days)
>> catting 1MB chunks into a 6MB chunk is likely (although not assured)
>> to lead to greatly reduced size during later compression, but I disagree
>> that catting 512MB chunks into a 3GB chunk is likely to be compressible
>> by any general-purpose compressor.
>>
>
> Which is what i meant with "way bigger than the library size of the
> algorithm". Mostly my "Information" was pitfalls to look out for when
> testing the speed of your equipment, if you went ahead and cat-ted 3000
> x 1MB, i believe the hardware compression would make something highly
> compressed out of it.
> My guess is it would work for most chunks around half as large as the
> buffer size of the drive (totally guessing).
The hardware compression standard used by LTO drives specifies a buffer size of
1K (1024 bytes). See
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-321.pdf,
Section 7.3, page 4.
A 1 MB chunk is even quite large for many current compression algorithms
distributed with common operating systems. The man page for gzip would seem to
list the buffer size used by that algorithm as 32 KB. The buffer size used by
bzip2 can vary between 100,000 and 900,000 bytes (corresponding to compression
settings -1 to -9). The recent LZMA .xz format compressor would appear to vary
maximum memory usage between 6 MB and 800 MB corresponding to the -0 to -9
compression presets. (The LZMA memory requirements are adjusted downwards to
accommodate a percentage of available RAM where necessary.)
Cheers,
Paul.
------------------------------------------------------------------------------
This SF.net email is sponsored by
Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
<Prev in Thread] |
Current Thread |
[Next in Thread>
|
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, (continued)
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Thomas Mueller
- [Bacula-users] Quantum Scalar i500 slow write speed, ekke85
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Thomas Mueller
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Alan Brown
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Henry Yen
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, John Drescher
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Henry Yen
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Christian Gaul
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Henry Yen
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Christian Gaul
- Re: [Bacula-users] Quantum Scalar i500 slow write speed,
Paul Mather <=
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Henry Yen
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Christian Gaul
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Paul Mather
- Re: [Bacula-users] Quantum Scalar i500 slow write speed, Richard Scobie
Re: [Bacula-users] Quantum Scalar i500 slow write speed, Cejka Rudolf
Re: [Bacula-users] Quantum Scalar i500 slow write speed, Cejka Rudolf
|
|
|