Since this message is pretty pro-de-dupe, I want to mention that I don't
sell any of this stuff.  I'm just excited about the technology, have
many customers large and small using it, and want to make sure it's
accurately represented.

>"We don't need tape, because disk is cheap!"
>[...hiatus...]
>"We have to save disk!  Buy (and integrate, and manage) a new product!"

I would put that history slightly differently.  I don't know anyone who
knew what they were doing that was saying "we don't need tape!"  What
they were saying is:

"Tape drives are now way too fast!  We have to stage to disk to backup
to stream the drives.  Wouldn't it be cool if we could also do away with
tape onsite, but we still need it for offsite."

[...hiatus...]

"Holy crap!  VTLs are expensive! Forget the store all onsite backups on
disk part.  Let's just do staging.  That requires a much smaller amount
of disk."

[...hiatus...]

"De-dupe is here.  Using that, we can take the amount of disk that we
would have bought just for staging and store all our onsite backups on
it.  Wow."

>I think a back-end de-dup (de do da da) would still offer advantages
>to TSM: if you've got mumblety-hundred (e.g.) Win2K boxen, then most
>of their system and app space would be identical. This could,
>concievably, end up as close to one system-images' worth of space on
>the back end.  In a fantasy. :)

This is not a fantasy.  There are products that have been GA for 3+
years that are doing just this.  These products also notice when a file
has been modified multiple times and just backs up the new blocks that
were changed each time.  In addition, these products also notice users'
files that are common between the filesystem and sitting inside Exchange
inboxes and Sent Items folders, for example.  They notice attachments
that were sent to multiple remote offices that have already been backed
up.  All of tis is reality, is GA, and is being used by many companies,
many of them very, very large.

>However, the server would need to do an awful lot of work to correlate
>all these data.

It's not easy, but it's not as hard as you may think.  The main work
comes from two things: computing a SHA-1 hash on each block of data and
looking up that hash in a big hash table.  The first is only performed
by each client (speaking of source de-dupe) on new or changed files, so
it's not as bad as you might think.  The second can handle quite a few
clients simultaneously without being a bottleneck.  At some point, you
may need multiple hash tables and servers to handle the lookup, but the
workload can be distributed.  For example, install a second lookup
server and each server handles lookups for half of the total list of
hashes.

As to how fast de-dupe backup software is, it's definitely fast enough
to keep up with remote offices and medium-sized datacenters.  Once we
start getting into many TBs of LOCAL data (i.e. a large datacenter),
there are much more efficient ways to back it up.  But if the data is
remote, de-dupe backup software is hard to beat.

(These last few comments were about de-dupe backup software -- not to be
confused with de-dupe VTLs.  Those actually go VERY fast and can handle
the largest of environments.)