• Please help support our sponsors by considering their products and services.
    Our sponsors enable us to serve you with this high-speed Internet connection and fast webservers you are currently using at ADSM.ORG.
    They support this free flow of information and knowledge exchange service at no cost to you.

    Please welcome our latest sponsor Tectrade . We can show our appreciation by learning more about Tectrade Solutions
  • Community Tip: Please Give Thanks to Those Sharing Their Knowledge.

    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.

  • Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)

    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

IBM Spectrum Protect Tivoli Storage Manager (TSM) Spectrum Protect to Google Cloud Platform via rclone

BackupPlan

Newcomer
Joined
Nov 15, 2022
Messages
1
Reaction score
0
Points
0
Location
Here
Website
www.tannerwilliamson.com
I'm new here, and have mostly come to learn from those with significant TSM experience. I'm fairly new to TSM, and can't contribute much on that regard here yet, but wanted to take the opportunity to contribute where I can, especially as seeing this Google Cloud Platform sub forum doesn't have a single thread yet.

I'll share some ideas for anyone who was looking for a method of getting TSM setup with Google Drive via rclone via GCP service accounts and APIs.

My boss was initially hesitant to try out the native cloud storage integration with the version of TSM we have (8.1.x), but was willing to meet me halfway by allowing backup to filesystem on disk, where I used rclone to present a remote storage location from GCP to the local TSM instance as a filesystem mount point. Rclone is free and open source software available for linux, windows, and more.

There are two ways you can go about using rclone with cloud storage. One is to define your remote storage configuration and then to mount it as a live mount point, and the other is to define the remote storage configuration but only use it for rclone specific on-demand calls. I'll explain why this is relevant later*.

Google Cloud Platform allows you to use many of their APIs for free, while paying for the cost of storage. There is a way to use Google Drive storage that comes with Google Workspace accounts (formerly known as Google apps and or Google G-suite), without having to incur any additional costs beyond what your business may already be paying for Google user accounts. The idea is that you create a shared team drive in drive.google.com. You then go into Google Cloud Platform console, and enable the Google Drive API. Next in GCP, create at least one service account. The service account will have a user identity given to you like [email protected]*google-something*.com. You copy that [email protected], and then go back to your Team Drive on drive.google.com, and choose to "share the drive with specific people", and invite the user to have full read and write access to the shared drive, while putting in the [email protected] user identity given to you for the google service account.

Next you configure rclone on your TSM host where you want the files to be accessible, and you can now choose to either mount the storage live, or script around rclone and synchronize with a utility such as rsync, or use rclone's copy, sync, and move files commands as your heart desires.

*Note that each Google User account is limited to uploading 750GB per day, and downloading 10TB per day. If you need to upload more than 750GB per day, you'll need to create multiple service accounts, and simply rotate through them. I did this by creating a list of archive files locally, then comparing it to a list of files already stored remotely. I then get the delta of new files needing to be uploaded, and put it into a list. That list is then used to generate a new file with the commands for uploading the files. I rotate through a list of service accounts for each file. For example, if I were to have 10 files to upload and 5 service accounts to use, my prepared command list would look something like this:

Note that these example commands are not syntactically correct, but are provided in a simplified form for the sake of demonstrating the concept of rotating through service accounts. You'll need to ensure you use the correct command syntax for your rclone operations.

Code:
rclone upload /opt/local-archives/file-A.bin [email protected]
rclone upload /opt/local-archives/file-B.bin [email protected]
rclone upload /opt/local-archives/file-C.bin servic[email protected]
rclone upload /opt/local-archives/file-D.bin [email protected]
rclone upload /opt/local-archives/file-E.bin [email protected]
rclone upload /opt/local-archives/file-F.bin [email protected]
rclone upload /opt/local-archives/file-G.bin [email protected]
rclone upload /opt/local-archives/file-H.bin [email protected]
rclone upload /opt/local-archives/file-I.bin [email protected]
rclone upload /opt/local-archives/file-J.bin [email protected]
rclone upload /opt/local-archives/file-K.bin [email protected]
rclone upload /opt/local-archives/file-L.bin [email protected]
rclone upload /opt/local-archives/file-M.bin [email protected]
rclone upload /opt/local-archives/file-N.bin [email protected]
rclone upload /opt/local-archives/file-O.bin [email protected]

Note how the service accounts are rotated through for each upload. This gives me (5) x 750GB of bandwidth to use for this upload task.

You can create up to 100 service accounts on a GCP account, so you have a theoretical maximum of uploading up to 100 x 750GB or 750,000 GB aka 75TB in a single rolling 24 hour period.

You could always improve on my methodology by implementing a way to spawn multiple upload processes at a time, and do simultaneous uploads to make your upload job go faster.


Some resources for learning about rclone:

Storage Systems Supported by RCLONE:

General Documentation

rclone commands:
 

The Spectrum Protect TLA (Three-Letter Acronym): ISP or something else?

  • Every product needs a TLA, Let's call it ISP (IBM Spectrum Protect).

    Votes: 22 19.5%
  • Keep using TSM for Spectrum Protect.

    Votes: 67 59.3%
  • Let's be formal and just say Spectrum Protect

    Votes: 15 13.3%
  • Other (please comement)

    Votes: 9 8.0%

Forum statistics

Threads
31,842
Messages
134,917
Members
21,814
Latest member
DavideGrossiTSM
Top