TSM with NetApp data deduplication - ASIS

[TSMmer]

Hi There

I have a requirement to store 15-30 days of active data on disks for quick restore. Just want to get feedback from community if anyone has tested TSM against NetApp data deduplication technology (Advance Single Instance Storage) and if yes
1) type of data backed up
2) how much data backed up daily and
3) what sort of data reduction factor they are getting. Thanks in advance.:redface:

Regards
TSMmer

 

[Harry_Redl]

Hi,

this is how it works - NetApp ASIS is INTERNAL thing only - so all the applications see the data in FULL size. Whenever you read files, you got them in full. NDMP does read the files - so even NDMP backups read the full size.
The only method that benefits from the deduplication is the SnapmirrorToTape - subclass of NDMP backups - as it is block-level transfer encapsulated in NDMP protocol.

Hope following example says it all:

single volume with deduplication enabled - I have put one 150MB file in two folders and run the "sis start" - this is the result:

netapp1> df -h dedup
Filesystem               total       used      avail capacity  Mounted on
/vol/dedup/              475MB      151MB      323MB      32%  /vol/dedup/
/vol/dedup/.snapshot       25MB       84KB       24MB       0%  /vol/dedup/.snapshot

netapp1> df -sh dedup
Filesystem                used      saved       %saved
/vol/dedup/              151MB      150MB          50%
netapp1>

This is how "normal" NDMP works - you can see it transfers cca 300MB:

tsm: xxxx>ba node simulator /vol/dedup mode=full toc=no

Date/Time                Message
--------------------     ----------------------------------------------------------
01/03/2012 13:09:00      ANR0984I Process 88 for BACKUP NAS (FULL) started in the BACKGROUND at 01:09:00 PM. (SESSION: 6359, PROCESS: 88)
01/03/2012 13:09:00      ANR1063I Full backup of NAS node SIMULATOR, file system /vol/dedup, started as process 88 
by administrator ADMIN. (SESSION: 6359, PROCESS: 88)
01/03/2012 13:15:14      ANR1067I NAS Backup to TSM Storage process 88 completed. (SESSION: 6359, PROCESS: 88)
01/03/2012 13:15:14      ANR0986I Process 88 for BACKUP NAS (FULL) running in the BACKGROUND processed 1 items for 
a total of 316,436,480 bytes with a completion state of SUCCESS at 01:15:14 PM. (SESSION: 6359, PROCESS: 88)
01/03/2012 13:24:09      ANR2017I Administrator ADMIN issued command: QUERY ACTLOG search='process 88'  (SESSION: 6359)

and this is how SMtT works - transfers about 160MB (few MB of metadata ....)

tsm: xxxx>ba node simulator /vol/dedup mode=full type=snapmirror

Date/Time                Message
--------------------     ----------------------------------------------------------
01/03/2012 13:19:37      ANR0984I Process 89 for NAS SNAPMIRROR BACKUP started in the BACKGROUND at 01:19:37 PM. 
(SESSION: 6359, PROCESS: 89)
01/03/2012 13:19:37      ANR2685I SnapMirror backup of NAS node SIMULATOR, file system /vol/dedup, started as process 
89 by administrator ADMIN. (SESSION: 6359, PROCESS: 89)
01/03/2012 13:21:06      ANR1067I NAS SnapMirror Backup process 89 completed. (SESSION: 6359, PROCESS: 89)
01/03/2012 13:21:06      ANR0986I Process 89 for NAS SNAPMIRROR BACKUP running in the BACKGROUND processed 1 items
 for a total of 161,161,216 bytes with a completion state of SUCCESS at 01:21:06 PM. (SESSION: 6359, PROCESS: 89)
01/03/2012 13:24:20      ANR2017I Administrator ADMIN issued command: QUERY ACTLOG search='process 89'  (SESSION: 6359)

Using SMtT does not allow incrementals and single file restore.

Hope it helps

Harry

 

[TSMmer]

Harry - Thanks for your comments. I am however keen to get feedback from community on
1) How much data storage reduction have been achieved through the use of NetApp ASIS e.g. when backing up 10TB/day and keeping data (TSM storage pool) on disks for say 10,20 or 30 days?
2) What were the issues if any

With 10TB/day over 30 days TSM storage pool on NetApp will be 300TB. With ASIS dedup enabled depending on the reduction factor the size will be smaller.

This is really a data deduplication related question as other options would be to use 1) TSM client/server deduplication 2) an in-line data deduplication tool from EMC or IBM 3) Data compression hardware like IBM RTC

Regards
TSMmer

 

[Harry_Redl]

Hi,

sorry - I misunderstood the question.
Have no numbers here but few pointers.
a) Data ONTAP lower than 8.1 has a limit for volume size to be deduplicted - it used to be platform dependent with maximum of 16TB. With ONTAP 8.1 the limit is the maximum volume size you can create (which is still platform dependant)
So even with high-end filer (which I assume you have better usage for than using it for backup) you have to create such a large pool from multiple NetApp volumes - and the NetApp deduplication is done on volume level
b) NetApp deduplication works better with NAS approach - as it really frees the size in the volume (and you see the freed space immediately). When you use the ASIS on the block device (LUN) - the space is freed as well - but the filesystem on the LUN is not aware of that. So it is a bit more complicated to use this space.
c) reclamation of file volumes makes the deduplication ratio worse as for NetApp filesystem the blocks change ....


Harry