|
Re: [Networker] ZFS deduplication.
2010-03-18 17:36:25
Attila Mester wrote:
I have recently made similar tests to see what dedup ratios I can get
when writing to dedup enabled ZFS filesystems as a B2D target.
I have discovered the same behavior, e.g. NetWorker saveset streams do
not really get deduplicated. Changing the underlying ZFS dedup blocksize
doesn't have any positiv effect, even if I go down to 8k blocksize.
The only explanation I can think of is, as Yaron already pointed out,
block boundaries are aligned differently because of the different
meta-data which is part of the savesets.
It seems, at current point of the code implementation the dedup can not
deal with different block bounderies and as such, not effectively usable
for deduplicating such streams.
BTW I experienced the same thing when using NetBackup as the backup
application.
If you will take a look at the file that is the save set (using od),
you will see that the name of the file is part of it. For example:
# od -bc d0e9315f-00000006-da957e9e-4b957e9e-b62e0014-4fcb6952 | head -30
0000000 000 000 000 001 003 027 130 003 000 000 000 000 000 000 000 000
\0 \0 \0 001 003 027 X 003 \0 \0 \0 \0 \0 \0 \0 \0
0000020 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000 000
\0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0
0000040 000 000 000 000 000 000 000 001 000 000 000 042 057 102 141 143
\0 \0 \0 \0 \0 \0 \0 001 \0 \0 \0 " / B a c
0000060 153 165 160 057 114 141 160 057 123 110 114 117 115 111 055 114
k u p / L a p / S H L O M I - L
0000100 101 120 057 104 145 163 153 164 157 160 056 151 156 151 000 000
A P / D e s k t o p . i n i \0 \0
0000120 000 000 000 010 000 000 026 101 000 226 200 003 000 000 000 000
\0 \0 \0 \b \0 \0 026 A \0 226 200 003 \0 \0 \0 \0
0000140 222 005 030 004 000 000 000 070 202 000 000 001 000 000 201 300
222 005 030 004 \0 \0 \0 8 202 \0 \0 001 \0 \0 201 300
0000160 000 000 000 001 000 000 011 022 000 000 000 013 000 000 000 342
\0 \0 \0 001 \0 \0 \t 022 \0 \0 \0 013 \0 \0 \0 342
0000200 000 000 000 000 000 000 000 010 000 000 026 101 000 226 200 003
\0 \0 \0 \0 \0 \0 \0 \b \0 \0 026 A \0 226 200 003
0000220 000 000 000 000 113 224 217 267 113 224 217 267 000 000 000 000
\0 \0 \0 \0 K 224 217 267 K 224 217 267 \0 \0 \0 \0
0000240 000 000 026 000 000 000 000 317 000 000 001 000 000 000 000 346
\0 \0 026 \0 \0 \0 \0 317 \0 \0 001 \0 \0 \0 \0 346
0000260 000 000 000 000 000 000 377 376 133 000 056 000 123 000 150 000
\0 \0 \0 \0 \0 \0 377 376 [ \0 . \0 S \0 h \0
0000300 145 000 040 000 154 000 154 000 103 000 241 141 000 163 000 163
e \0 \0 l \0 l \0 C \0 241 a \0 s \0 s
0000320 000 111 000 156 000 000 200 146 000 157 000 135 000 015 000 012
\0 I \0 n \0 \0 200 f \0 o \0 ] \0 \r \0 \n
0000340 000 104 000 151 000 162 260 335 210 210 143 000 164 220 211 162
\0 D \0 i \0 r 260 335 210 210 c \0 t 220 211 r
This means that files are not guaranteed to be at the same block
offset. Since ZFS (unlike dedup appliances) is not aware of this
possibility, it cannot detect duplicate blocks. This can be solved in
several ways:
. Make ZFS aware of how Networker (or Netbackup) works and see that
it can look for duplicate data (not just duplicate blocks).
. Make Networker write the save sets in such a way that files always
start at the beginning of a block.
I don't see why EMC will want to solve this problem, because it will
allow for a low cost alternative to Data Domain and VTLs product lines.
I also don't see Sun fixing this problem.
regards -attila
********************************************************************
Attila Mester 5 Digit Sun internal: x62534
Data Protection Architect Tel: (+49 89) 46 008 2534
Sun Microsystems GmbH Fax: (+49 89) 46 008 2583
Sonnenallee 1 Mobil: +49 172 812 5947
85551 Heimstetten / Germany mail: attila.mester AT sun DOT com
********************************************************************
Terry Lemons schrieb:
Hi Yaron
I know that NetWorker has a default block size for each of its output
devices. This default block size can be overridden; details on this
are in the NetWorker Administration Guide.
Does ZFS have a specific block size that it deduplicates? If so,
could it be that the NetWorker default block size for the AFTD device
is not the same as, or a multiple of, the ZFS default block size?
tl
-----Original Message-----
From: EMC NetWorker discussion [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU]
On Behalf Of Yaron Zabary
Sent: Sunday, March 14, 2010 3:59 PM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: [Networker] ZFS deduplication.
A few days ago I read a post to EMC's Networker forum by Nicholas
Bone (https://community.emc.com/thread/99839?tstart=0). He reported a
test he performed with AFTD which was running on top of ZFS with
dedup. Unfortunately, he wasn't able to get any reasonable dedup
ratios (1.03 for three full savesets of the the same file system). My
conclusion was that Networker does not align files at block level,
which confuses the ZFS dedup code. Is anyone familiar with some flag
or any configuration option which will convince save or AFTD to do the
right thing so that ZFS will be able to find identical blocks ?
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
--
-- Yaron.
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
|
|
