Replicate/export node constantly juggles tapes

[Lars-Owe]

In order to balance the load across our servers we're trying to move a medium size node (about 20TB) from one server to another. Initially we tried to use replication, and achieved some 5TB in a months time. Then we changed strategy to export. The first couple of TB from the diskpool went over in less than two days, but now the process is reading from tape. First it reads a few files from a primary pool tape, then a few from a copy tape, then back to the primary, waiting for mounts most of the time. It acts pretty much the same way as replication, except that it moves disk data first. All data is spread across 16-17 primary tapes for this node, most of them holding over 1TB. We're not aware of any tape in error state, missing, or being read only. Please advise.

Best regards,
lars-owe

 

[moon-buddy]

Sounds and looks normal to me.

Data cannot be all on disk so it takes it from tapes. If the collocation is OFF, then expect this behavior.

 

[marclant]

If the collocation is OFF, then expect this behavior.

To expand on that, it's because export is done by node, then filespaces. So, each time a new filespace is exported, it's highly possible it will need a tape previously used for another filespace.

Since you are only exporting one node, if you have enough space in a diskpool, do a move nodedata to move it all to disk. However, you are not really saving anything, the time you spend moving the data is time you could be spending exporting.

 

[marclant]

First it reads a few files from a primary pool tape, then a few from a copy tape, then back to the primary

If both the primary and copypool are in the library, then the TSM Server can pick the files from any.

This explains how it happens during a restore: http://www-01.ibm.com/support/docview.wss?uid=swg21380647 But it's essentially the same approach during an export as far as volume selection is concerned between primary and copy volumes.

 

[Lars-Owe]

Since you are only exporting one node, if you have enough space in a diskpool, do a move nodedata to move it all to disk. However, you are not really saving anything, the time you spend moving the data is time you could be spending exporting.

There are only five file spaces with about 25TB of data spread across some 145000000 files for this node. A colocation group is set. Less than 20 tapes are used in the primary pool, but well over 100 for the copy pool, so it seems unlikely that the data would be more concentrated on copy tapes. I'm afraid that we only have 12TB in our disk pools, most of which is needed for the daily operation. Besides we're using TSM 7.1.1 with a bug regarding moving data back to a pool where it already resided.

To alleviate the drawback of switching tapes every few seconds or possibly minute, I kicked up the mount retention time, and thus were able to save the wait for quite a few mounts but not all.

Now I'd like to sync the last files that have changed. As I changed strategy from replication to export I need to get the replication going again with sync receive/sync send. Unfortunately the sending tsm server dumps core after a few hours before actually exporting anything, regardless of whether only one file system or all are being replicated :-(