• Please help support our sponsors by considering their products and services.
    Our sponsors enable us to serve you with this high-speed Internet connection and fast webservers you are currently using at ADSM.ORG.
    They support this free flow of information and knowledge exchange service at no cost to you.

    Please welcome our latest sponsor Tectrade . We can show our appreciation by learning more about Tectrade Solutions
  • Community Tip: Please Give Thanks to Those Sharing Their Knowledge.

    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.

  • Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)

    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Slow backup stg to AWS

gernthefish

ADSM.ORG Member
#1
I created a 10TB volumes in AWS, mounted it as an ext4 filesystem on our linux TSM server, and created a seq file stg copypool w/ 35 200gb volumes. For some reason, the initial backup stg ran at a constant ~14mb/sec. We are copying from 1 LTO6 tape at a time, but this still seems VERY slow. Based on several metrics we're looked at, I'm suspecting TSM as the bottleneck, but can't pinpoint anything. Any thoughts?
 

marclant

ADSM.ORG Moderator
#2
I suggest you try to manually copy a large amount of date to AWS outside TSM and see what kind of throughput you are getting. That will help you determine if the bottleneck is outside or inside TSM.
 

gernthefish

ADSM.ORG Member
#3
Thanks marclant, we have tried copying data from outside TSM and got much higher throughput. Any ideas on what could be limiting this within TSM?
 

marclant

ADSM.ORG Moderator
#4
No. Probably need a deep performance analysis to identify the bottleneck. It's never as easy as enabling the option: go_fast=yes
 

TykneeTym

ADSM.ORG Member
#5
By AWS do you mean S3? I don't think AWS provides block or file level storage out to non-EC2 systems. If this is the case you're doing a translation from File to Object (S3 is object storage) that may be part of your problem. What hardware/software are you using to do this translation?
 

gernthefish

ADSM.ORG Member
#6
Yeah, we came to the same conclusion on the object stg, but AWS now has an S3 file gateway that appears to be a much better solution. This allows us to mount a bucket as an NFS share. Testing has shown improved but inconsistent performance #s so far. Still trying to get a better handle on this.
 
#7
Hi Did you ever perform the deep performance analysis? If so what was the conclusion? Also, what improvement did you see using the NFS gateway? you say performance was inconsistent but was the improvement consistent writing to NFS.
 

gernthefish

ADSM.ORG Member
#8
No, we haven't gotten to the deep analysis yet. Amazon claims file gateway max rates for upload is 125 mb/sec and 20mb/sec for download, but we're still seeing problems. On a linux NFS client host, we are getting about 60+ mb/sec uploading a single file using the AWS CLI (bypasses the gateway), but much slower results when we use standard cp copy of the same file (uses gateway). With cp, smaller files typically get 20-30 mb/sec, but larger files (>20gb or so) are VERY slow. Like 1 mb/sec or something. Amazon is claiming a fix is in the works, so we'll see what happens.
 

Advertise at ADSM.ORG

If you are reading this, so are your potential customer. Advertise at ADSM.ORG right now.

UpCloud high performance VPS at $5/month

Get started with $25 in credits on Cloud Servers. You must use link below to receive the credit. Use the promo to get upto 5 month of FREE Linux VPS.

The Spectrum Protect TLA (Three-Letter Acronym): ISP or something else?

  • Every product needs a TLA, Let's call it ISP (IBM Spectrum Protect).

    Votes: 7 23.3%
  • Keep using TSM for Spectrum Protect.

    Votes: 16 53.3%
  • Let's be formal and just say Spectrum Protect

    Votes: 4 13.3%
  • Other (please comement)

    Votes: 3 10.0%

Forum statistics

Threads
30,889
Messages
131,420
Members
21,194
Latest member
jamesmacd40