Slow backup stg to AWS

[gernthefish]

I created a 10TB volumes in AWS, mounted it as an ext4 filesystem on our linux TSM server, and created a seq file stg copypool w/ 35 200gb volumes. For some reason, the initial backup stg ran at a constant ~14mb/sec. We are copying from 1 LTO6 tape at a time, but this still seems VERY slow. Based on several metrics we're looked at, I'm suspecting TSM as the bottleneck, but can't pinpoint anything. Any thoughts?

 

[marclant]

I suggest you try to manually copy a large amount of date to AWS outside TSM and see what kind of throughput you are getting. That will help you determine if the bottleneck is outside or inside TSM.

 

[gernthefish]

Thanks marclant, we have tried copying data from outside TSM and got much higher throughput. Any ideas on what could be limiting this within TSM?

 

[marclant]

No. Probably need a deep performance analysis to identify the bottleneck. It's never as easy as enabling the option: go_fast=yes

 

[TykneeTym]

By AWS do you mean S3? I don't think AWS provides block or file level storage out to non-EC2 systems. If this is the case you're doing a translation from File to Object (S3 is object storage) that may be part of your problem. What hardware/software are you using to do this translation?

 

[gernthefish]

Yeah, we came to the same conclusion on the object stg, but AWS now has an S3 file gateway that appears to be a much better solution. This allows us to mount a bucket as an NFS share. Testing has shown improved but inconsistent performance #s so far. Still trying to get a better handle on this.

 

[ch02all]

Hi Did you ever perform the deep performance analysis? If so what was the conclusion? Also, what improvement did you see using the NFS gateway? you say performance was inconsistent but was the improvement consistent writing to NFS.

 

[gernthefish]

No, we haven't gotten to the deep analysis yet. Amazon claims file gateway max rates for upload is 125 mb/sec and 20mb/sec for download, but we're still seeing problems. On a linux NFS client host, we are getting about 60+ mb/sec uploading a single file using the AWS CLI (bypasses the gateway), but much slower results when we use standard cp copy of the same file (uses gateway). With cp, smaller files typically get 20-30 mb/sec, but larger files (>20gb or so) are VERY slow. Like 1 mb/sec or something. Amazon is claiming a fix is in the works, so we'll see what happens.

 

[gernthefish]

Correction: the Amazon file gateway max throughput rates I listed in the previous post were incorrect. They do not have max rates listed for file gateway yet - only for cached volumes, stored volumes and tape gateway. Just wanted to set the record straight. See link for more info:

http://docs.aws.amazon.com/storagegateway/latest/userguide/resource-gateway-limits.html