On Aug 6, 2010, at 4:48 AM, Christian Gaul wrote:

> Am 05.08.2010 21:56, schrieb Henry Yen:
>> On Thu, Aug 05, 2010 at 17:17:39PM +0200, Christian Gaul wrote:
>> 
[[...]]
>> 
>>>> /dev/urandom seems to measure about 3MB/sec or thereabouts, so creating
>>>> a large "uncompressible" file could be done sort of like:
>>>> 
>>>>   dd if=/dev/urandom of=tempchunk count=1048576
>>>>   cat tempchunk tempchunk tempchunk tempchunk tempchunk tempchunk > bigfile
>>>> 
>>>> 
>>> cat-ting random data a couple of times to make one big random file wont
>>> really work, unless the size of the chunks is way bigger than the
>>> "library" size of the compression algorithm.
>>> 
>> Reason 1: the example I gave yields a file size for "tempchunk" of 512MB,
>> not 1MB, as given in your counter-example.  I agree that (at least 
>> now-a-days)
>> catting 1MB chunks into a 6MB chunk is likely (although not assured)
>> to lead to greatly reduced size during later compression, but I disagree
>> that catting 512MB chunks into a 3GB chunk is likely to be compressible
>> by any general-purpose compressor.
>> 
> 
> Which is what i meant with "way bigger than the library size of the
> algorithm".  Mostly my "Information" was pitfalls to look out for when
> testing the speed of your equipment, if you went ahead and cat-ted 3000
> x 1MB, i believe the hardware compression would make something highly
> compressed out of it.
> My guess is it would work for most chunks around half as large as the
> buffer size of the drive (totally guessing).

The hardware compression standard used by LTO drives specifies a buffer size of 
1K (1024 bytes).  See 
http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-321.pdf, 
Section 7.3, page 4.

A 1 MB chunk is even quite large for many current compression algorithms 
distributed with common operating systems.  The man page for gzip would seem to 
list the buffer size used by that algorithm as 32 KB.  The buffer size used by 
bzip2 can vary between 100,000 and 900,000 bytes (corresponding to compression 
settings -1 to -9).  The recent LZMA .xz format compressor would appear to vary 
maximum memory usage between 6 MB and 800 MB corresponding to the -0 to -9 
compression presets.  (The LZMA memory requirements are adjusted downwards to 
accommodate a percentage of available RAM where necessary.)

Cheers,

Paul.
------------------------------------------------------------------------------
This SF.net email is sponsored by 

Make an app they can't live without
Enter the BlackBerry Developer Challenge
http://p.sf.net/sfu/RIM-dev2dev 
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users