Re: [ADSM-L] *EXTERNAL* Re: deleing data from a containerpool

We have a HDS G400 behind our new TSM servers FMD/Flash behind one container 
for our primary production database (11.5 TB with 16 versions <full backup 
nightly> is fitting in 16.6TB) and the rest is in a giant nearline SAS pool.  
The flash also serves to back the logs/db and the SAS serves as a DBB/logarch 
location.

Other lessons learned from container pools...

1.  Make sure that your TSM server is big enough.  We have had to upgrade some 
old Power 750's to new boxes to handle the overhead of compression and dedupe.  
This offsets some of the finical savings from getting rid of purpose built 
appliances.

2.  In EVERY SINGLE VERSION that we have tried from 7.1.3.200, 7.1.4.x, to 
7.1.5.x we have run into container related bugs.  As such, we keep having to 
upgrade the TSM server, only to find new bugs introduced in the newer versions. 
 TSM used to be a set and forget application for us.  Now we are stuck in a 
horrible upgrade cycle for the last year and it doesn't seem to be letting up.  
Last week we were told that we may need to go to 7.1.6 since TSM thinks a 
container is full when it's not.


---------------------------------------------------
David Nixon
Storage Engineer II
Technology Services Group
Carilion Clinic
451 Kimball Ave.
Roanoke, VA 24015
Phone: 540-224-3903
cdnixon AT carilionclinic DOT org

Our mission: Improve the health of the communities we serve.



________________________________________
From: ADSM: Dist Stor Manager [ADSM-L AT VM.MARIST DOT EDU] on behalf of 
Rhodes, Richard L. [rrhodes AT FIRSTENERGYCORP DOT COM]
Sent: Wednesday, August 24, 2016 7:08 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] *EXTERNAL* Re: deleing data from a containerpool

I'm curious, what kind of storage system do ya'll use for your container pools?



-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
David Ehresman
Sent: Tuesday, August 23, 2016 8:56 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: *EXTERNAL* Re: deleing data from a containerpool

We are getting a 78% savings on a mixed workload (BA, Oracle, VM, Exchange, 
SQL) with about 350 nodes moving around 10TB of data a night.

David

-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of 
Stefan Folkerts
Sent: Tuesday, August 23, 2016 8:43 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] deleing data from a containerpool

This inline deduplication and compression thing really is amazing;

excerpt from q stgpool f=d

                Deduplication Savings: 153,840 G (90.83%)
                  Compression Savings: 8,711 G (56.07%)
                    Total Space Saved: 162,550 G (95.97%)

So on top of the 90.83% deduplication saving (that dont' require any
reclaiming) this customer saves another 56.07% of the stored data due to
compression making it a total of 95.97% space saved.
These figures require a lot of duplicate data of course but still,
containerpools FTW!
We call it the "cool pool" btw. :-)



On Mon, Aug 22, 2016 at 4:01 PM, Del Hoobler <hoobler AT us.ibm DOT com> wrote:

> Minor correction.
>
> Inline compression for container pools was added in March in 7.1.5.
>
>
> IBM Spectrum Protect 7.1.5 - Inline compression:
> - Performed in-line after deduplication to provide additional storage
> savings
> - Negligible impact on resources – uses latest and most efficient
> compression algorithms
> - Can potentially double (or more) your storage savings after
> deduplication
>
>
> Del
>
> ----------------------------------------------------
>
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 08/22/2016
> 09:53:08 AM:
>
> > From: "Loon, Eric van (ITOPT3) - KLM" <Eric-van.Loon AT KLM DOT COM>
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Date: 08/22/2016 09:53 AM
> > Subject: Re: deleing data from a containerpool
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> > Indeed, TSM 7.1.0 to 7.1.5 only supported deduplication, additional
> > compression was introduced in 7.1.6.
> > Kind regards,
> > Eric van Loon
> > Air France/KLM Storage Engineering
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> > Behalf Of David Ehresman
> > Sent: maandag 22 augustus 2016 15:12
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: deleing data from a containerpool
> >
> > At the most recent levels of TSM, it both dedups and compresses but
> > make sure you are at a level that does both.  There was a level that
> > only did dedup but not compression.
> >
> > David
> >
> > -----Original Message-----
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> > Behalf Of Rhodes, Richard L.
> > Sent: Monday, August 22, 2016 9:07 AM
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Subject: Re: [ADSM-L] deleing data from a containerpool
> >
> > >But I totally agree, everyone who is using file device
> >
> > >classes or expensive backend deduplication (like Data Domain or
> Protectier)
> >
> > >should seriously consider switching to container pools.
> >
> >
> >
> >
> >
> > We currently use DataDomains.
> >
> > With a DD it dedups what it can, then compresses the rest.
> >
> > Does TSM also try and compress what is leftover after dedup?
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> >
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> > Behalf Of Loon, Eric van (ITOPT3) - KLM
> >
> > Sent: Tuesday, August 16, 2016 3:39 AM
> >
> > To: ADSM-L AT VM.MARIST DOT EDU
> >
> > Subject: Re: deleing data from a containerpool
> >
> >
> >
> > Hi Stefan!
> >
> > Our database is on SSD in an IBM V3700, but the time needed for a
> > del filespace can be significant though. But I totally agree,
> > everyone who is using file device classes or expensive backend
> > deduplication (like Data Domain or Protectier) should seriously
> > consider switching to container pools. We are working on a design
> > for our next TSM servers and we are able to lower our costs per TB
> > by 75% compared to the old design based on the Data Domain!
> >
> > Kind regards,
> >
> > Eric van Loon
> >
> > Air France/KLM Storage Engineering
> >
> >
> >
> > -----Original Message-----
> >
> > From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On
> > Behalf Of Stefan Folkerts
> >
> > Sent: dinsdag 16 augustus 2016 8:33
> >
> > To: ADSM-L AT VM.MARIST DOT EDU
> >
> > Subject: Re: deleing data from a containerpool
> >
> >
> >
> > Yes, I too have noticed this and it is something to keep in mind.
> >
> > At the same time, I think almost everybody using this pool will be
> > using SSD's for the database the impact will be overseeable.
> >
> > But the directory containerpool is still the best thing to happen to
> > Spectrum Protect since replication came along if you ask me. great
> > performance increase over fileclass restores, no more stopping
> > reclaims during the day to increase restore performance, no more
> > messing with numopenvolsallowed and reclaim values and number of
> > processes to optimize daily operations and restore speed...oh, and
> > compression that saves an easy 30-50% storage and license cost on
> > top of the deduplication!
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Aug 15, 2016 at 11:20 AM, Loon, Eric van (ITOPT3) - KLM <
> > Eric-van.Loon AT klm DOT com> wrote:
> >
> >
> >
> > > Hi all!
> >
> > > After doing some extensive testing with a directory container
> >
> > > storagepool I noticed a significant change compared to the old
> >
> > > traditional storage pools.
> >
> > > In a traditional storage pool TSM stores a file like an object. In
> >
> > > most cases one file is one object as far as I could see. Deleting this
>
> >
> > > data is very fast: a delete filespace runs rather fast because TSM
> >
> > > only has to delete the objects. So deleting a large database client
> >
> > > with multiple TBs takes a few seconds or maybe a few minutes.
> >
> > > When you are using a container storage pool everything changes. Files
> >
> > > are still stored as objects, but objects are split into chunks. The
> >
> > > average size of a chuck is approx. 100 KB and TSM performs dedup on
> >
> > > this chuck level. So if you now delete a large file, TSM has to
> >
> > > inspect every chunk to see if it is unique or not. When it is unique
> >
> > > it will be deleted, otherwise not. If you delete a file which is for
> >
> > > instance 40 GB in size, TSM has to inspect around 420,000 chucks
> >
> > > before the object can be deleted. I noticed that this takes several
> >
> > > seconds to complete, so one has to take into consideration that the
> >
> > > deletion of large clients requires significant more time to
> > complete than one is used to.
> >
> > > Deleing a client with little more than 1 TB of Oracle data was running
>
> >
> > > for more than 20 minutes. So a delete filespace for really large
> >
> > > database clients can run for hours! Has anyone else noticed this
> > behavior too?
> >
> > > Kind regards,
> >
> > > Eric van Loon
> >
> > > Air France/KLM Storage Engineering
> >
> > > ********************************************************
> >
> > > For information, services and offers, please visit our web site:
> >
> > > https://urldefense.proofpoint.com/v2/url?
> >
> u=http-3A__www.klm.com&d=AwIGaQ&c=SgMrq23dbjbGX6e0ZsSHgEZX6A4IAf
> 1SO3AJ2bNrHlk&r=dOGCMY197NTNH1k_wcsrWS3_fxedKW4rpKJ8cHCD2L8&m=
> vUuUnchIk8qp8ANX9ecD5HSZje8iCRgiNUPhmahQWTQ&s=
> eE7XROkp9Iv02y6CJDM84muZgTbKNhpt7nAgYPrCs-0&e=
> > . This e-mail and any attachment may contain
> >
> > > confidential and privileged material intended for the addressee only.
> >
> > > If you are not the addressee, you are notified that no part of the
> >
> > > e-mail or any attachment may be disclosed, copied or distributed, and
> >
> > > that any other action related to this e-mail or attachment is strictly
>
> >
> > > prohibited, and may be unlawful. If you have received this e-mail by
> >
> > > error, please notify the sender immediately by return e-mail, and
> > delete this message.
> >
> > >
> >
> > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or
> >
> > > its employees shall not be liable for the incorrect or incomplete
> >
> > > transmission of this e-mail or any attachments, nor responsible
> > for any delay in receipt.
> >
> > > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> >
> > > Dutch
> >
> > > Airlines) is registered in Amstelveen, The Netherlands, with
> >
> > > registered number 33014286
> >
> > > ********************************************************
> >
> > >
> >
> > ********************************************************
> >
> > For information, services and offers, please visit our web site:
> > https://urldefense.proofpoint.com/v2/url?
> >
> u=http-3A__www.klm.com&d=AwIGaQ&c=SgMrq23dbjbGX6e0ZsSHgEZX6A4IAf
> 1SO3AJ2bNrHlk&r=dOGCMY197NTNH1k_wcsrWS3_fxedKW4rpKJ8cHCD2L8&m=
> vUuUnchIk8qp8ANX9ecD5HSZje8iCRgiNUPhmahQWTQ&s=
> eE7XROkp9Iv02y6CJDM84muZgTbKNhpt7nAgYPrCs-0&e=
> > . This e-mail and any attachment may contain confidential and
> > privileged material intended for the addressee only. If you are not
> > the addressee, you are notified that no part of the e-mail or any
> > attachment may be disclosed, copied or distributed, and that any
> > other action related to this e-mail or attachment is strictly
> > prohibited, and may be unlawful. If you have received this e-mail by
> > error, please notify the sender immediately by return e-mail, and
> > delete this message.
> >
> >
> >
> > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/
> > or its employees shall not be liable for the incorrect or incomplete
> > transmission of this e-mail or any attachments, nor responsible for
> > any delay in receipt.
> >
> > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> > Dutch Airlines) is registered in Amstelveen, The Netherlands, with
> > registered number 33014286
> >
> > ********************************************************
> >
> >
> >
> > ********************************************************
> > For information, services and offers, please visit our web site:
> > https://urldefense.proofpoint.com/v2/url?u=http-3A__www.klm.com&d=AwIFaQ&c=SgMrq23dbjbGX6e0ZsSHgEZX6A4IAf1SO3AJ2bNrHlk&r=dOGCMY197NTNH1k_wcsrWS3_fxedKW4rpKJ8cHCD2L8&m=2khjm5kIUk4HAMhbAmSErW4psymX49EvvcPlonm458c&s=KERJhiJfWrogTNbgmZOr6_HGAgfaXIQntXRwL231Di4&e=
> >  . This e-mail and any attachment may contain
> > confidential and privileged material intended for the addressee
> > only. If you are not the addressee, you are notified that no part of
> > the e-mail or any attachment may be disclosed, copied or
> > distributed, and that any other action related to this e-mail or
> > attachment is strictly prohibited, and may be unlawful. If you have
> > received this e-mail by error, please notify the sender immediately
> > by return e-mail, and delete this message.
> >
> > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/
> > or its employees shall not be liable for the incorrect or incomplete
> > transmission of this e-mail or any attachments, nor responsible for
> > any delay in receipt.
> > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal
> > Dutch Airlines) is registered in Amstelveen, The Netherlands, with
> > registered number 33014286
> > ********************************************************
> >
>
>
>

________________________________

Notice: The information and attachment(s) contained in this communication are 
intended for the addressee only, and may be confidential and/or legally 
privileged. If you have received this communication in error, please contact 
the sender immediately, and delete this communication from any computer or 
network system. Any interception, review, printing, copying, re-transmission, 
dissemination, or other use of, or taking of any action upon this information 
by persons or entities other than the intended recipient is strictly prohibited 
by law and may subject them to criminal or civil liability. Carilion Clinic 
shall not be liable for the improper and/or incomplete transmission of the 
information contained in this communication or for any delay in its receipt.