You're welcome, happy to help.
Deleting objects is very database and active log intensive, but it also
hits the CPU, that said, I've never seen a 16 core machine really struggle
on CPU within the blueprint specs, even with compression enabled on the
containerpool running maximum backup performance there was still some room
at 10Gb/s line-speed backup performance.
The biggest and most noticeable upgrade I have given Spectrum Protect
servers is a swap to SSD, it changes everything, even admin commands become
more responsive.
Most of the time it starts out fine, but then, as the system gets more load
and the database grows it slowly gets slower when later (it seems pretty
suddenly) it becomes unworkable.
I think the memory upgrade is most likely just a coincidence timing wise.
But again, run the benchmarks and nmon and check the data. :-)
On Tue, Jul 25, 2017 at 9:20 PM, Zoltan Forray <zforray AT vcu DOT edu> wrote:
> Thanks for the suggestions. I am looking at the "blueprint" stuff and it
> looks pretty heavy-duty. I will look into running nmon. Things seem to
> have gotten worse since I upgraded the memory. DB backups to NFS/ISILON
> are now running 15+ hours with very little load (stopped all replications
> since they were becoming never-ending). Deleting of lots of objects
> (20-million is one example) is running into many days if not a week.
>
> On Tue, Jul 25, 2017 at 2:57 PM, Stefan Folkerts <
> stefan.folkerts AT gmail DOT com>
> wrote:
>
> > Another thing you could do besides the benchmark is run nmon in batch
> mode
> > on the Linux Spectrum Protect servers and analyze that, run it for an
> hour
> > or 2 when the load is heavy, I could help you with the output if you
> could
> > use some help, no problem.
> >
> > On Tue, Jul 25, 2017 at 8:49 PM, Stefan Folkerts <
> > stefan.folkerts AT gmail DOT com>
> > wrote:
> >
> > > How many drives in what kind of a raid setup? what kinds of performance
> > > are you getting from them, do you have any idea?
> > >
> > > I have had nothing but issues with performance on deduplicating setups,
> > > especially with replication in play when you use 15K disks.
> > > I've had setups with 24x15K drives in raid 10 in a V7000 and I still
> was
> > > having a hard time getting all the work done, as soon as you drop in
> some
> > > SSD's you set, but I suppose they can work on small setups.
> > > I would never use 15K drives again for any kinds of disk based
> > > deduplicating setup however, SSD's are not that expensive anymore, you
> > can
> > > use read intensive SSD's in most cases and be fine.
> > >
> > > Really, the blueprint benchmark tool works fine and gives a good
> > > indication of your base disk performance, if it's far below what's
> > > described for the blueprint that's probably the problem.
> > > I would put my money on the 15K drives for the database and my
> suggestion
> > > (in the case of insufficient database performance) would be place the
> > > database and the active log on SSD's
> > >
> > >
> > >
> > > On Tue, Jul 25, 2017 at 7:58 PM, Zoltan Forray <zforray AT vcu DOT edu>
> wrote:
> > >
> > >> The two database filesystems (1TB each) are on internal, 15K SAS
> drives.
> > >>
> > >> On Tue, Jul 25, 2017 at 1:34 PM, Stefan Folkerts <
> > >> stefan.folkerts AT gmail DOT com>
> > >> wrote:
> > >>
> > >> > My question would be on what type of storage is the Spectrum Protect
> > >> > database located.
> > >> > Second question, have you run the IBM blueprint benchmark tool on
> the
> > >> > storagepool and database storage, and if so, what were the results?
> > >> >
> > >> > On Mon, Jul 24, 2017 at 3:55 PM, Sasa Drnjevic <
> Sasa.Drnjevic AT srce DOT hr
> > >
> > >> > wrote:
> > >> >
> > >> > > Not sure of course...But, I would blame NFS
> > >> > >
> > >> > > Did you check the negotiated speed of your NFS eth 10G ifaces?
> > >> > > And that network?
> > >> > >
> > >> > > Regards,
> > >> > >
> > >> > > --
> > >> > > Sasa Drnjevic
> > >> > > www.srce.unizg.hr
> > >> > >
> > >> > >
> > >> > > On 24.7.2017. 15:49, Zoltan Forray wrote:
> > >> > > > 8-cores/16-threads. It wasn't bad when it was replicating from
> > >> > 4-SP/TSM
> > >> > > > servers. We had to stop all replication due to running out of
> > space
> > >> > and
> > >> > > > until I finish this cleanup, I have been holding off
> replication.
> > >> So,
> > >> > > the
> > >> > > > deletion has been running standalone.
> > >> > > >
> > >> > > > I forgot to mention that DB backups are also running very long.
> > >> 1.5TB
> > >> > DB
> > >> > > > backup runs 8+hours to NFS storage. These are connected via
> 10G.
> > >> > > >
> > >> > > > On Mon, Jul 24, 2017 at 9:41 AM, Sasa Drnjevic <
> > >> Sasa.Drnjevic AT srce DOT hr>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> On 24.7.2017. 15:25, Zoltan Forray wrote:
> > >> > > >>> Due to lack of resources, we have had to stop replication on
> one
> > >> of
> > >> > our
> > >> > > >> SP
> > >> > > >>> servers. The replication target server is 7.1.6.3 RHEL 7, Dell
> > >> T710
> > >> > > with
> > >> > > >>> 192GB RAM. NFS/ISILON storage.
> > >> > > >>>
> > >> > > >>> After removing replication from the nodes on source server, I
> > have
> > >> > been
> > >> > > >>> cleaning up the replication server by deleting the filespaces
> > for
> > >> the
> > >> > > >> nodes
> > >> > > >>> we are no longer replicating.
> > >> > > >>>
> > >> > > >>> My issue is the delete filespaces on the replication server is
> > >> taking
> > >> > > >>> forever. It took over a week to delete one filespace with
> > >> 31-million
> > >> > > >>> objects?
> > >> > > >>
> > >> > > >>
> > >> > > >> That is definitely tooooo loooong :-(
> > >> > > >>
> > >> > > >> It would take 6-8 hrs max, in my environment even under
> > "standard"
> > >> > > load...
> > >> > > >>
> > >> > > >> How many CPU cores does it have?
> > >> > > >>
> > >> > > >> And how is/was it performing the role of a target repl. server
> > >> > > >> performance wise?
> > >> > > >>
> > >> > > >> Regards,
> > >> > > >>
> > >> > > >> --
> > >> > > >> Sasa Drnjevic
> > >> > > >> www.srce.unizg.hr
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>
> > >> > > >>>
> > >> > > >>> To me it is highly unusual to take this long. Your thoughts on
> > >> this?
> > >> > > >>>
> > >> > > >>> --
> > >> > > >>> *Zoltan Forray*
> > >> > > >>> Spectrum Protect (p.k.a. TSM) Software & Hardware
> Administrator
> > >> > > >>> Xymon Monitor Administrator
> > >> > > >>> VMware Administrator
> > >> > > >>> Virginia Commonwealth University
> > >> > > >>> UCC/Office of Technology Services
> > >> > > >>> www.ucc.vcu.edu
> > >> > > >>> zforray AT vcu DOT edu - 804-828-4807
> > >> > > >>> Don't be a phishing victim - VCU and other reputable
> > organizations
> > >> > will
> > >> > > >>> never use email to request that you reply with your password,
> > >> social
> > >> > > >>> security number or confidential personal information. For more
> > >> > details
> > >> > > >>> visit http://infosecurity.vcu.edu/phishing.html
> > >> > > >>>
> > >> > > >>
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > --
> > >> > > > *Zoltan Forray*
> > >> > > > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > >> > > > Xymon Monitor Administrator
> > >> > > > VMware Administrator
> > >> > > > Virginia Commonwealth University
> > >> > > > UCC/Office of Technology Services
> > >> > > > www.ucc.vcu.edu
> > >> > > > zforray AT vcu DOT edu - 804-828-4807
> > >> > > > Don't be a phishing victim - VCU and other reputable
> organizations
> > >> will
> > >> > > > never use email to request that you reply with your password,
> > social
> > >> > > > security number or confidential personal information. For more
> > >> details
> > >> > > > visit http://infosecurity.vcu.edu/phishing.html
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> *Zoltan Forray*
> > >> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> > >> Xymon Monitor Administrator
> > >> VMware Administrator
> > >> Virginia Commonwealth University
> > >> UCC/Office of Technology Services
> > >> www.ucc.vcu.edu
> > >> zforray AT vcu DOT edu - 804-828-4807
> > >> Don't be a phishing victim - VCU and other reputable organizations
> will
> > >> never use email to request that you reply with your password, social
> > >> security number or confidential personal information. For more details
> > >> visit http://infosecurity.vcu.edu/phishing.html
> > >>
> > >
> > >
> >
>
>
>
> --
> *Zoltan Forray*
> Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator
> Xymon Monitor Administrator
> VMware Administrator
> Virginia Commonwealth University
> UCC/Office of Technology Services
> www.ucc.vcu.edu
> zforray AT vcu DOT edu - 804-828-4807
> Don't be a phishing victim - VCU and other reputable organizations will
> never use email to request that you reply with your password, social
> security number or confidential personal information. For more details
> visit http://infosecurity.vcu.edu/phishing.html
>
|
