stgrule actiontype=copy for the tsm for virtual machine restore from tape copy

[naresh4u2007]

Hello ,

Any one tried stgrule with actiontype=copy for the virtual environment to perform the restore from tapes.
primary stgpool is containers and we are running daily stgrule with actiontype=copy to the tapes and when we make primary stgpool unavailable to perform the restore from tapes it is too slow.
Any one experience these issue please suggest how to proceed.
Regards,
Naresh

 

[dietmar]

Hi,

this does not work. Mainly because during the copy to Tape the blocks (very small) are not in sequense on Tape. We had a call open with IBM. No solution. Works as designed (not) . Currently i am not aware of a Solution to get a VM Backup snapshot dehydrated to Tape for direct restore . As alternative we baclient image backup and dehydrate it to tape. This works as expected (it restores 1 fat file ) . Streams what the Tape could manage . For DR try to make more Containers pools on more different Servers maybe , then the repair stgpool Container runs in a reasonable time. A pity, but thats what it is.

This was some 2 years ago, maybe i missed something and this changed.

Br, Dietmar

 

[Trident]

Hi,

You need to keep the ctl files on disk only. Create a small stgpool (devt=file or disk).

Have a look at this:

https://www.ibm.com/docs/en/storage-protect/8.1.20?topic=reference-vmctlmc

Good luck,

 

[dietmar]

Hi, spare your time , if you think how the stgrule works - copy all dehydrated vm blocks to tape which is missing in pool. It does not collocate or something ( even if it would collocate+filespace by node it data blocks are not in sequence ) - it will never stream of be fast at all.

The only thing (in my opionion ) which would work if there would be a feature like syntetic full , and it produces large files on tape in sequence for the restore procedure .

For a fast DR , have a very fast server , good Disks on the Containers with low latency and repair the Container from Tape from a not too large Pool. Special Containers with TDP Data has a very very small chunk size in the Container ( arround 60k), there a hell of a lot IO and DB2 updates have to be done during repair of the stgpool from Tape. As faster the Server (DB2) + low IO Latency is ( small writes ) the faster the rebuild will be.

I went thru (DR) test and production repairs, and the performance differes a lot depending on the Data on the Container.(chunk size) (between 20-40 TB repair / Day on the same hw ) . Sadly i had no chance to use faster Disks to compare if it would change or if the DB2 was the bottleneck . ( had a big v5000 with 1PB Raid10 6TBSATA ) . Has anyone their Containers on SSD ?

The DR Plan now :
We ended up with doing 10 Servers with baclient image backup dehydrate to Tape for the most important Servers to restore first . ( 150MB/s - LTO8 )
future: use a 2nd replication target Server with Library sharing . ( rebild of Server,container faster , restore from 2 Servers )

All not perfect, but a DR restore from 100% Tape is also not a solution .

Br, Dietmar

ps. I also work with a different Backup Tool with an S3 as DR - not better . 60MB/s for a single session - the backpu tools generates one big file in backend , so a 10TB+ VM is boring to store/retrieve from S3 ... Huge HW needed just for DR copy, a lot of Maintenance ( software defined ) . very expensive .

 

[Trident]

Hi, spare your time , if you think how the stgrule works - copy all dehydrated vm blocks to tape which is missing in pool. It does not collocate or something ( even if it would collocate+filespace by node it data blocks are not in sequence ) - it will never stream of be fast at all.

This needs to be in place before you do the backup. So, if you add it now, then it will be for future restore. Secondly, for better performance, you can collocate per filespace on the copy pool (maybe have a dedicated copy pool for your vmware env.). It will still need be a lot of io and the small chunks will still be there, but you should be able to get better performance.

Added:
If you want to test with all local storage unavailable, then restore the ctl files first to a new stgpool. This would be a fairly small amount of data, but will speed up the restore process afterwards a lot.

Rgds,

 

[dietmar]

Hi,

Even then the Full ( eg 10k Blocks ) and the Inc by Day ( eg 1k blocks) are not in sequense on Tape . They are "simple" new blocks which are copied over random . ( even if each VM is on one 1 Tape exclusivly ) . The stgrule does not know which blocks the tdp client(baclient) will retrieve in sequense from Tape, it will just append all missing blocks randomly to Tape.

It is not much better if the Control Files are on Disk and restored first. Some time ago, but i also went thru this and it did not solve the problem that the Data Files Blocks are not in sequense on Tape. To be able to test you need to have the Control Files anyway on Disk + copypool )

anyway i would be very happy if this would work and i am wrong . Spend 10 Tapes for the 10 critical VM's would be fine, but not that i am aware that this is usefull. ( it works but in "bytes" Speed ) .

I would real like to dehydrate vm to tape with direct restore possiblity, so any solution is very welcome.

Best regards, Dietmar

 

[fredric]

Hi,
I strugle with the same problems, very slow restore from tape(lto9).
I have primary a directory container pool -replicated to a target server. Even a copypool(stgrule copy) and I have also done some retention set backups.

I have tested both restore from a copypool(stgrule copy) and Retention sets. I set the disk pools unavailable before testing the restores.

I was a suprised that the retention sets restore wanted to use a tape used for vmctl, but even with the vmctl (vmware control files) on disk, the restore took very long time.

The retention sets are a "synthetic full backup" and hydrated copies of the vm. But the files seems to be spread/mixed with other filespaces on the tape. How can I get the filespace to be stored sequel on the tape?

 

[dietmar]

Hi Fredric,

You should have read this before and trust me ( a bit ) . This does not work.

As said above, we are now migrating ( for a customer ) to multible replication target servers where we migrate from 1 120 TB VMWARE(1k vm's) Dedicated Container Pool on one Server, to 2 Servers with 3-4 Pools. In Addition we use new Servers with 4Ghz+ and Flash for those Containers ( because of the tiny block size ) . For DR we have then 2 Servers to rebuild in paralell with smaller Containers .

Rebuilding the Containers will be the time to count on. The Faster the total System , the smaller the Container Pool brings availability for restore operations .

If Randsomware happens u need to take care which versions are able to be restored, so restoring all version ( the container) could start immediately , and if this does not take tooo long this is fine .

So we (will hopefully) end up with following @Customer:

* Criestie Image Backup for very critical Systems ( stgrule copy to Tape )
* Some big File Export ( encrypted )migrate from disk to LTO directly.
* 2 Replication Target Servers with 4 tapes each
* new HW for Replication Targets CPU/Disk ( need to be faster than primary ... )
* Flag VM TAG for importants ( MC ) , use different Container Pools

As said i have seen 15-35TB / Day repair times on Containers, so it depends a LOT on Server HW + chuck size.

2 Server in paralell:

* Rebuild Server - restore DBB
* Rebuild 1 Container ( eg. VMWARE MC Gold Systems )
* Restore Image Backups and Exported Big files could start immediately
* Rebuild 1 Container done .
* Restore Gold VM's
* Rebuild 2 Container ...

=> prepare a lot of LOG Space, and you will need to do DBB in between .

So if prepared and divided by importance, first restore procedures could start at day1 when the HW is available again. ( OS,SP and DBB Restore if documented and tested could be done within 5-8 h )

If everything is gone u need to bring the vCenter, Datamover for hotadd back also . Restore vCenter via ESXi (nbd) was always very slow and it was also huge with a lot of disks . I am not an expert in vCenter - but i think building new and restore configuration would be faster .

The worst case i was "in" , that arround 250 TB ( one Pool) took us about 10+ days to repair from Tape. So this is nothing u want at all again.

Best regards, Dietmar

 

[fredric]

Hi Dietmar,

thank you very much for your respond.

I thought "Retention sets" would save my day. Have you tested Retention sets?

I must probably re-think how I do my backups to tape and set up smaller container pools and protect them to tape (only replicating to target server today).

So my stgrule copy to tape is waste of time?
I have no experience of Criestie Image Backups.

I appreciate your help!

Best regards, Fredric

 

[dietmar]

Hi,

From my opinion for TDP VMWARE Data there is no option for dehydrate . Criestie uses/could use a baclient image Backup . It helps with the ISO Boot File + dissimilar hw restore , but you could create them by yourself as well. ( a bit of work/pain if you are not used to this Windows PE+baclient). You have then for example a C Image file which could be dehydrated and restored well from Tape .

VMTags are SP MC Classes those point to the Container, if you use not the same classes on our replikation Target u could define 2 Pools on the Target and devide them.

You should setup a local Protect to Tape anyway if you have the resouces ( now ! ) . AirGap is quite important from my point of you .

good luck , Dietmar

System Backups :
* Do not use System State -> It is a pain in the A. on Tape .
* Image Backup could be restored to image (only balcient cli support ) in one piece and opend with tools like 7zip.
* Restore on C is very rare anyway ....

 

[fredric]

Hi,

When you do a local protect to tape and bring the tapes offline/offsite(once a week), will the backup data not be spread out on many tapes after some months?
I would like to have a full copy every time/week, is that possible?

//Fredric

 

[dietmar]

If you going to use DRM ( offsite Tapes) you will more Tapes and offsite reclaiming to get the Tapes free again. The retention set to Disk (pit restore) gives u that in logic. Restored/Repaired in case of a DR in the hole container pool.

 

[dietmar]

Still do not understand why IBM has no SP HW Boxes with max. 100TB Capacity which could be stacked and mirrored/replicated. Build something GUI nice arround. The Backend is real good. Attach SAN Tapes on the protect them. Harden the SW u HW as much as possible and voila Keep baclient+api interface, and do something nicer and better for MS and Virtualisation backups.

whishful thinking

 

[fredric]

Yes, sounds like a plan. Have you told it to IBM?

I think there are very little documentation about restores, specially restores involving tapes.

//Fredric

 

[dietmar]

have been @IBM for 15 years (tivoli sw group) , now a IBM BP for 10 years . Too small to be so important to have any influance how "i" would like to have it *lol* .

 

[fredric]

Oh great, I have only worked with IBM products for 17years (HP/SUN earlier)

 

[fredric]

Any idea of a way to "move" filespaces to another(new) container-stgpool?

If not, I'm planing to create new copygroup policies/tags in vCenter to store new backup data to new directory container stgpools. But it would be nice if it was possible to "move" the filespaces directly.

 

[dietmar]

I am not that far, i would suggest to tag 1 vm with the new stgpool and make a full - set the other stgpool to unavailable and try to restore . It should work , after the cycle of eg 30 days/versions - the old data should disapear on the old pool. messing arround with node and filespace names does not work as the are referenced in the control files for tdp vmware data ...

 

[dietmar]

keep in mind that you will need additional Space and backend license if it is the case - as dedup is only working per stgpool container , and during the 30 days even more ... dedup vmware data is up to 1:5

 

[dietmar]

my inital design will be to keep the primary sp server with 1 pool for all data , and use 2 repl target servers to devide it on 2 hosts with 3-4 pools . So if the dedup is not so good because of multible pools on copy this does not change the license, as only primary Data is counted .