Jeffrey J. Kosowsky wrote at about 20:33:15 -0500 on Tuesday, February 8, 2011:
> Jeffrey J. Kosowsky wrote at about 20:15:04 -0500 on Tuesday, February 8,
> 2011:
> > Jeffrey J. Kosowsky wrote at about 16:56:30 -0500 on Tuesday, February 8,
> 2011:
> > > Jeffrey J. Kosowsky wrote at about 16:07:04 -0500 on Tuesday, February
> 8, 2011:
> > > > When I backup files on an arm (armel) architecture with rsync
> > > > checksums and then transfer the pool over to an x86 architecture,
> the
> > > > checksums no longer verify.
> > > >
> > > > In other words checksums that verify properly on the armel platform,
> > > > don't verify when run on an x86 platform and vice-versa.
> > > >
> > > > This is quite distressing since this means that one can't copy over
> > > > the pool from one architecture to another and expect it to work
> > > > properly without lots of checksum errors.
> > > >
> > > > I have double checked this by writing my own routine which uses the
> > > > subroutines in File::RsyncP::digest to verify digests on both
> > > > platforms -- and the same file gives *different* results on the
> > > > different platforms.
> > > >
> > > > I'm really at a loss to explain how and why this happens but it is
> not
> > > > a good bug ;)
> > > >
> > > > I am not sure if this is related to the bug I mentioned last week in
> > > > perl 10.0 where md5sums are miscomputed on the arm platform.
> > > >
> > >
> > > Here is the bug report identifying the problem with Digest::MD5
> > > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=289884
> > >
> > > Apparently, it has to do with arm processors requiring
> > > U32_ALIGNMENT_REQUIRED.
> > >
> >
> > OK - here is some more info...
> > Looking through the source files for Digest::MD5 and Digest::MD4, they
> > both include code to take care of both big-vs.-little endian *and* u32
> > alignment.
> >
> > It seems that perhaps this code is missing from md4c.c in
> > File::RsyncP::Digest.
> >
> >
> > I'm not sure how to fix the code, but I am guessing that the issue is
> > there.
> >
> > I think this is an IMPORTANT bug since if I am following this right,
> > then many of the md4 rsync checksums end up being wrong on arm
> > architectures.
> >
>
> I tried adding -DU32_ALIGNMENT_REQUIRED to Makefile.PL for
> File::RsyncP::Digest but that doesn't seem to have fixed the problem.
>
> So maybe there are deeper issues with the code that have some 32
> implicit bit alignment assumptions.
>
> I'm not sure how to debug and fix this further...
>
OK - I just spent the evening taking apart the rsync checksum stuff.
The problem is in the Block Digests, not the File Digests.
And the problem is actually in the Adler32 sum part and *not* in the
md4sum.
However, I'm not sure what is causing the problem there...
At first I thought it might be an Endian issue but from googling it
seems that Linux is little endian on both x86 and armel.
Any ideas on what might be wrong and how to fix it?
---------------------------------
The code to compute the Adler32 sum is pretty simple:
/*
* a simple 32 bit checksum that can be updated from either end
* (inspired by Mark Adler's Adler-32 checksum)
*/
UINT4 adler32_checksum(char *buf, int len)
{
int i;
UINT4 s1, s2;
s1 = s2 = 0;
for ( i = 0 ; i < len - 4; i += 4 ) {
s2 += 4 * (s1 + buf[i]) + 3 * buf[i+1] + 2 * buf[i+2] + buf[i+3];
s1 += (buf[i+0] + buf[i+1] + buf[i+2] + buf[i+3];
}
for ( ; i < len ; i++ ) {
s1 += buf[i];
s2 += s1;
}
return (s1 & 0xffff) + (s2 << 16);
}
/* UINT4 defines a four byte word.
We use the Perl byte-order definition to discover if a long has more than
4 bytes. If so we will try to use an unsigned int. This is OK for DEC
Alpha but may not work everywhere. See the TO32 definition below.
*/
#if (PERL_BYTEORDER <= 4321) || defined(UINT4_IS_LONG)
typedef unsigned long UINT4;
#else
typedef unsigned int UINT4;
#endif
------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
|
