Good info in the article - clears up a lot of things about VTL's and
de-dupe.  There are a few things it doesn't mention, which I still have
questions on, though.  Anyone have any insight on these questions?

What is the "sample" size for determining duplication of data?  If it's
64MB or 256MB blocks, am I going to get many hits, as the same data may not
be blocked the same way each time?

If I de-dupe, there's got to be a catalog/database.  How is it protected?
I assume that's vendor specific.  That's also a question for the VTL in
general - how do I protect the catalog, showing what blocks, in what order,
are defined to each virtual tape volume?  Physical library inventories are
re-creatable from the tape labels, but in the VTL, they're all Virtual.
And I don't think, "You don't need to worry about that" is a good enough
answer - I already don't trust disk, which is why I'm doing a backup to
begin with!  The answer may be internal to the VTL (and vendor specific),
but it would be a good thing to know.

Does de-dupe work with encrypted data?  Compression normally doesn't, so I
suspect the answer would be no, but someone may have come up with a system
to do it.

And does anyone have any real world, TSM based, numbers on the space
savings they can share?  I'm still skeptical (those who know me know that'
just me), but I know a lot of people (many represented here on the list)
are really behind the technology, and I'm willing to be converted (I
think).

Have a good weekend all, and may your pagers stay quiet!

Nick Cassimatis

----- Forwarded by Nicholas Cassimatis/Raleigh/IBM on 06/15/2007 05:50 PM
-----

"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 06/15/2007
05:26:00 PM:

> It really is all over the board as to the details...
>
> Think of it as a custom filesystem with a really large block size.  (As
> in 64 or 256 MB instead of the typical 8K you see in a normal
> filesystem.)
>
> As to how the whole deduplication thing works, you can read an article I
> wrote on the subject here:
>
> http://tinyurl.com/3588fb
>
>
> ---
> W. Curtis Preston
> Author of O'Reilly's Backup & Recovery and Using SANs and NAS
> VP Data Protection
> GlassHouse Technologies
>
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf 
> Of
> Steven Harris
> Sent: Monday, June 11, 2007 4:31 PM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: [ADSM-L] Just how does a VTL work?
>
> Allen S. Rout wrote:
> > This is irrational to some extent, I recognize.  But for the VTL, I
> > don't know where the database is; it's _magic_.
> >
> > - Allen S. Rout
> >
> Good point Allen,
>
> I realize that  VTLs come in different flavours from several
> manufacturers, but is there any information about how their gizzards are
> arranged?
>
> I'd particularly like to understand how tape blocks are mapped to disk
> blocks, and how the de-dup technology is organized.  There are a number
> of possible implementations, and I can see that DB searches/updates
> could become an enormous bottleneck particularly in a windows
> environment with potentially thousands of identical copies of windows
> system files.  This raises the spectre of another proprietary database
> to be managed, grown, reorganized and re-indexed with limited
> facilities.
>
> Does anyone have pointers to this sort of information or is it a deep,
> dark commercial secret?
>
> Thanks
>
> Steve
>
> Steven Harris
> AIX and TSM Administrator
> Brisbane Australia
>
> Moving to Sydney Mid July - Offer me a job now and avoid the rush!