• Please help support our sponsors by considering their products and services.
    Our sponsors enable us to serve you with this high-speed Internet connection and fast webservers you are currently using at ADSM.ORG.
    They support this free flow of information and knowledge exchange service at no cost to you.

    Please welcome our latest sponsor Tectrade . We can show our appreciation by learning more about Tectrade Solutions
  • Community Tip: Please Give Thanks to Those Sharing Their Knowledge.

    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.

  • Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)

    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Number of migration processes drops due to tape drive preemption

alexp36

ADSM.ORG Member
Joined
Jun 14, 2018
Messages
15
Reaction score
0
Points
0
We have our SP server configured to run 6 migration processes. There are 8 tape drives.

What generally happens is that the migration from disk to tape starts with 6 processes, but then gradually loses processes to preemption from backups or restores which need a tape drive.
After while it ends up with only about 2 processes, and eventually only 1.

There is a lot of data to migrate (the disk pool is 17TB), and once it's down to 1 or 2 processes, the migration takes a very long time to complete(24 hours or more), and it often doesn't keep up with the data coming into the disk pool.

Is there some way of forcing a migration to either re-spawn processes as tape drives become available, or automatically exit and restart, once it gets down to only 1 or 2 processes?
 

RecoveryOne

ADSM.ORG Senior Member
Joined
Mar 15, 2017
Messages
347
Reaction score
81
Points
0
I know of no way of having it restart, unless you use external scripting to watch your processes. A rough idea could be a ksh script to run 'q pr' and count the number of migrations along with their pid. Case statements if pidCount > $x cancel pr $1, cancel pr $2.
Downside to that, is migration will naturally reduce the number of streams it's using as it finishes up moving data.

Sounds like your best bet is to get more drives, and/or review your schedules and eliminate resource contention.

Just my two cents.
 

alexp36

ADSM.ORG Member
Joined
Jun 14, 2018
Messages
15
Reaction score
0
Points
0
I know of no way of having it restart, unless you use external scripting to watch your processes. A rough idea could be a ksh script to run 'q pr' and count the number of migrations along with their pid. Case statements if pidCount > $x cancel pr $1, cancel pr $2.
Downside to that, is migration will naturally reduce the number of streams it's using as it finishes up moving data.

Sounds like your best bet is to get more drives, and/or review your schedules and eliminate resource contention.

Just my two cents.

Thanks, was probably what I was expecting, from what I've read so far. Good point about the downside - like you say, it's not that easy to know if it's down to one migration process because it's nearly finished, or if it's because of contention.
 

RecoveryOne

ADSM.ORG Senior Member
Joined
Mar 15, 2017
Messages
347
Reaction score
81
Points
0
Without knowing what to what you are moving, you might be able to look at free space of source pool for a rough gauge as to when a migration process is finishing up. And then just put in some arbitrary water marks in your script that if less than 20% free restart migration process else end.

But that could cause unforeseen issues elsewhere. Not saying it couldn't be done, but then you might get into a situation where migration never catches up, and you'd have to pause items to get cought up before you ran out of stgpool space.

Guess it comes down to whatever is easier for you:
More landing stgpool space, with scripts that take action upon logic you build.
More drives/hba/adapters to drive those tape drives.
Try to tune schedules.

For reference, I just took a peek at one of my migration processes, 4 drives. File stgpool on IBM 5030 to LTO6 tape. 7.1TB or 7.3TB moved from 0622 to 1418 one day, and then the next run took from 0645 to 0132. Just wildly depended on my server load at the time. Interesting thing, the first 3 threads on the 0645 to 0132 run finished in roughly the same time frame as the first run, but the last thread moved roughly equal bytes and files didn't finish until 0132.

I'm not concerned in the processing delay, as I am set in resources at this moment in time.
 

marclant

ADSM.ORG Moderator
Joined
Jun 16, 2006
Messages
3,742
Reaction score
619
Points
0
Location
Canada
Website
www.ibm.com
Migration from random disk to tape is done at the node and filespace level. So if a node has much larger daily backups than others, you might see the number of drives slowly go down as there's only a handful of large nodes/filespaces left to process.

Migration from file device class in contrast is done file volume by file volume, so it's easier to keep more concurrent threads going until the end. File pools also perform better than random disk pool because of the larger block size. File pool are sequential and use 256K block size, while random only 4K (I believe, don't quote me on that). If you decide to switch to a file pool, make sure to follow the recommendations here:
 

alexp36

ADSM.ORG Member
Joined
Jun 14, 2018
Messages
15
Reaction score
0
Points
0
Migration from random disk to tape is done at the node and filespace level. So if a node has much larger daily backups than others, you might see the number of drives slowly go down as there's only a handful of large nodes/filespaces left to process.

Migration from file device class in contrast is done file volume by file volume, so it's easier to keep more concurrent threads going until the end. File pools also perform better than random disk pool because of the larger block size. File pool are sequential and use 256K block size, while random only 4K (I believe, don't quote me on that). If you decide to switch to a file pool, make sure to follow the recommendations here:
Thanks. I think it is actually a file based pool. The device class which the pool uses has a device type of "file", so assume that's what you mean. Cheers.
 

Advertise at ADSM.ORG

If you are reading this, so are your potential customer. Advertise at ADSM.ORG right now.

DigitalOcean $100 Credit

Support ADSM.ORG and get DigitalOcean FREE credit. DigitalOcean currently offer a $100, 60-day Free Credit for new accounts. Sign-up here:

DigitalOcean Referral Badge

The Spectrum Protect TLA (Three-Letter Acronym): ISP or something else?

  • Every product needs a TLA, Let's call it ISP (IBM Spectrum Protect).

    Votes: 20 18.7%
  • Keep using TSM for Spectrum Protect.

    Votes: 65 60.7%
  • Let's be formal and just say Spectrum Protect

    Votes: 13 12.1%
  • Other (please comement)

    Votes: 9 8.4%

Forum statistics

Threads
31,898
Messages
135,992
Members
21,783
Latest member
london
Top