Offsite Backups

[srp]

Hi,



I hope I am in the right forum, else please tell me otherwise!



1. Being a newbie I was wondering if I could accomplish the following without "DRM".



- Back up our Disk(Primary) storage pool every night.



- Perform a full backup of the database every night and run incremental backups Mon - Fri and copy data to tape - can I use copypool or ltopool?.



- Run Archive backups every weekend and month by backing up my primary storage pool(diskpool) to my Copypool(tapepool) and storing offsite and retaining that data for 1 month for weekly's tapes and 1 year for the monthly tape backups.



- Ship the database and copy storage pool volumes to an offsite location every day and be able to RESTORE them for up to a year?



2. From the link below I have been reading up on offsite storage and TSM:

http://www.redbooks.ibm.com/redbooks/pdfs/sg245416.pdf



The steps that you need to perform to send data offsite are:

1. Back up your vital data to the copy storage pools.

2. List all volumes that must be sent to offsite.

3. Check out volumes from the library.

4. Update the volume location to offsite.



Backup your vital data

Use the backup stgpool command to create copies of files that reside in a primary

storage pool, and store them in a copy storage pool. This is shown in

Example 13-35. File versions are only copied if they do not already exist in the

copy storage pool — if a file is already present in the specified copy storage pool,

then it is not recopied. However, if a copy of a physical file already exists in the

copy storage pool, but the copy is marked damaged, a new copy will be created,

provided that the primary physical file is not also marked damaged. The following

command backs up onsite data from the DISKDATA and DISKDIRS storage pool

to the OFFDATA and OFFDIRS storage pools respectively. (This task can also

be automated using a server script...



Offsite tape management to onsite

In this section we show how to move offsite volumes to the onsite location. These

offsite volumes either have no active data on them (status of EMPTY) or are

being returned to recreate damaged data in a primary storage pool.

Offsite tapes can be seen as your second good copy of critical data. As you

perform backup and archive operations, all data is saved in storage pools, which

are the basic unit for onsite operations. When you perform a backup stgpool

command, IBM Tivoli Storage Manager verifies which data has not previously

been sent to offsite yet and then performs the copy operation of any new data.

This means the operation is incremental and hence minimizes the amount of

time required to execute it.

As backup and archive data age and expire, the offsite tapes contain less and

less active data. Offsite tape reclamation consolidates these tape volumes,

replacing the partially filled tapes with full and empty tapes. The newly emptied

tapes remain in a pending state until after the reuse delay period has expired. At

this time the status of the tape changes to EMPTY. Empty tapes can be brought

back onsite and returned to the scratch pool.



3. And do I understand this correctly about Copypool's from above ?

Yesterday, I could NOT restore data from an "offiste Copypool tape because a volume was deemed "Empty" and either because the REUSE DELAY=0 or ?

I was unaware my Copypool tapes were expiring (expire inventory? process?)

So then, I set the access=readonly and it WAS deleted!!

What can I do about this if we don't run DRM and I want to use Copypool?

Should I just use Ltopool(tapepool)? Perhaps migrate from diskpool to the ltopool(tapepool)?

Or could I use the command: "backup stg diskpool ltopool"?

Would that preserve my data if my setting are correct?

I am unsure what parmaters to set to keep my data onsite(available) & offsite(available for restores up to a yr?) and not have TSM delete it periodically.



sory for the load of information here, hope somone can decipher and give some feedback.



Thanks for any help on the subject of "offsite storage".



srp

 

[srp]

Hi,



I looked over my TSM server config.

Still working on keeping my ARCHIVE & INCR data copies on my COPYPOOL tapes offsite and intact for restoration.



Operating without DRM is tough and now it seems I can't use Data Protection for Archiving either...



tsm:>set archiveretentionprotection on

ANR2799W Server operation is in violation of server license terms - server is not licensed for Tivoli Storage

Manager for Data Retention.

ANR0117E The server contains stored data. Archive data retention protection cannot be changed.

ANS8001I Return code 27.





My CopyPool says I have 365 retention, will TSM listen to thsi without Data Protection?



tsm: WILMSNTCH5>q co type=archive f=d



Policy Domain Name: STANDARD

Policy Set Name: ACTIVE

Mgmt Class Name: STANDARD

Copy Group Name: STANDARD

Copy Group Type: Archive

Retain Version: 365

Retention Initiation: Creation

Retain Minimum Days:

Copy Serialization: Shared Static

Copy Frequency: CMD

Copy Mode: Absolute

Copy Destination: ARCHIVEPOOL

Last Update by (administrator): ADMIN

Last Update Date/Time: 07/18/2005 15:54:36

Managing profile:



Policy Domain Name: STANDARD

Policy Set Name: STANDARD

Mgmt Class Name: STANDARD

Copy Group Name: STANDARD

Copy Group Type: Archive

Retain Version: 365

Retention Initiation: Creation

Retain Minimum Days:

Copy Serialization: Shared Static

Copy Frequency: CMD

Copy Mode: Absolute

Copy Destination: ARCHIVEPOOL

Last Update by (administrator): ADMIN

Last Update Date/Time: 07/18/2005 15:54:36

Managing profile:







So, if some of my CopyPool tapes show an "Empty" status and the others as "full" or "filling" some of my data is intact and being kept so far...



TSM1 COPYPOOL LTOCLASS1 381,468.0 45.2 Filling

TSM12 LTOPOOL1 LTOCLASS1 316,412.8 99.9 Full

TSM13 LTOPOOL1 LTOCLASS1 287,754.6 3.4 Full

TSM14 LTOPOOL1 LTOCLASS1 381,468.0 1.6 Filling

TSM15 COPYPOOL LTOCLASS1 381,468.0 0.3 Filling

TSM16 LTOPOOL1 LTOCLASS1 293,249.6 1.1 Full

TSM17 COPYPOOL LTOCLASS1 381,468.0 2.1 Filling

TSM18 LTOPOOL1 LTOCLASS1 278,145.3 76.6 Full

TSM19 COPYPOOL LTOCLASS1 381,468.0 51.0 Filling

TSM2 COPYPOOL LTOCLASS1 381,468.0 16.2 Filling

TSM21 COPYPOOL LTOCLASS1 336,320.0 82.1 Full

TSM23 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM26 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM27 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM28 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM30 LTOPOOL1 LTOCLASS1 314,045.2 100.0 Full

TSM31 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM33 LTOPOOL1 LTOCLASS1 313,894.6 93.2 Full

TSM34 LTOPOOL1 LTOCLASS1 381,468.0 39.6 Filling

TSM35 LTOPOOL1 LTOCLASS1 298,958.2 100.0 Full

TSM38 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM4 COPYPOOL LTOCLASS1 381,468.0 30.2 Filling

TSM41 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM43 LTOPOOL1 LTOCLASS1 381,468.0 0.0 Filling

TSM44 COPYPOOL LTOCLASS1 381,468.0 32.7 Filling

TSM6 LTOPOOL1 LTOCLASS1 283,434.2 4.8 Full

TSM7 COPYPOOL LTOCLASS1 0.0 0.0 Empty

TSM9 LTOPOOL1 LTOCLASS1 288,561.6 22.1 Full



Does my EXPINTERVAL relate to just my incrementals?



tsm:>q opt



Server Option Option Setting Server Option Option Setting

----------------- -------------------- ----------------- --------------------

CommTimeOut 60 IdleTimeOut 15

BufPoolSize 65536 LogPoolSize 512

DateFormat 1 (mm/dd/yyyy) TimeFormat 1 (hh:mm:ss)

NumberFormat 1 (1,000.00) MessageFormat 1

Language AMENG Alias Halt HALT

MaxSessions 25 ExpInterval 24



How will this relate to my ARCHIVES, if any?



I updated these params for my copypool storage pool:



tsm: WILMSNTCH5>upd stg copypool maxscratch=100

ANR2202I Storage pool COPYPOOL updated.



tsm: WILMSNTCH5>upd stg copypool reusedelay=60

ANR2202I Storage pool COPYPOOL updated.



Did I screw this up or is this kinda' on the right track?



Thanks.



SRP

 

[srp]

Hi again,



Looks like I created a monster with what I've written so far...sorry about blathering on...and on...sigh.



Point blank, I realize I cannot use the old fashioned way of "rotational" backups with TSM.

I need to have incremental and archive backups for DR purposes.

I know backups to COPYPOOL provide this but reclamation and copying tapes are tough for me with one tape drive in my library.



I'm looking into virtual nodes for my setup, anyone out there configure before?



Please respond if you have any resources or can assist in any way, thanks!



SRP

 

[Harry_Redl]

Hi,

lets have sort this out ...

I have went through your post and have few basic questions:

1) your environment - TSM server version, library, drive, disk space for TSM

2) TSM pool configuration - place "q stg" output here

3) WHAT do you want to do - simple answer please (for example: backup XGB of data a day and take it daily offsite or backup daily, have data onsite and weekly take offsite)





Answer these and we can then move on.



Hope it helps



Harry

 

[srp]

Hi Harry,



Thanks for responding.



Win 2003 TSM server 5.3

3582-LTO2 Tape Library

1 LTO2 drive

SAN space for DISKPOOL set at 522 GB.

Daily INCR = Around 175-250 GB

Weekly/Monthly ARCH = Around 475 - 515GB.



Storage Device Estimated Pct Pct High Low Next Stora-

Pool Name Class Name Capacity Util Migr Mig Mig ge Pool

Pct Pct

----------- ---------- ---------- ----- ----- ---- --- -----------

ARCHIVEPOOL DISK 0.0 M 0.0 0.0 90 70 DISKPOOL

BACKUPPOOL DISK 0.0 M 0.0 0.0 90 70 FILPOOL1

COPYPOOL LTOCLASS1 37,645 G 2.8

DISKPOOL DISK 522 G 0.0 0.0 0 0 LTOPOOL1

FILPOOL1 FILEDEV1 0.0 M 0.0 0.0 90 70

LTOPOOL1 LTOCLASS1 157,502 G 1.5 2.8 90 70

RECLAIMPOOL FILEDEV1 0.0 M 0.0 0.0 90 70 LTOPOOL1

SPACEMGPOOL DISK 0.0 M 0.0 0.0 90 70





1. I would like to continue Daily incr backups to my TAPEPOOL - LTOPOOL1 - onsite.(leave in library)

This is incremental forever AFAIK, right?

And I have 1 "latest" copy of each backed up file from the LTOPOOL tapes for DR that are backed up to COPYPOOL tapes(offsite).



2. Next, I want to figure out how I can backup data and save a version/copy of a file for 1 yr.(365) - can I do this with my onsite TAPEPOOL --> LTOPOOL1?

I was told I cannot archive backups and store them on my COPYPOOL tapes?

So can I use archiving or just the backup method w/ options to keep multiple copies of my data for 1 yr?

We unfortuanately do not have DRM.



Hope that was clear, I appreciate the help.



SRP

 

[Harry_Redl]

Hi,



0) your main problem is 1 drive - if it is possible to get a second one, it would ease things a lot - you would be able to copy data tape-to-tape as well as doing tape reclamations easily.





So lets get dig it





Change your pool structure to:

Primary: Backuppol (size 0) -> Diskpool -> LTO1POOL

Copy: Copypool





1) your backups go to diskpool and your diskpool is big enough to hold a daily backup- that is good

2) before migration occurs, schedule "backup stg diskpool copypool" - it will copy daily backups to offsite media

3) AFTER finishing of this process update volumes which were written to to access "offsite" - using command "upd vol <volume name> acc=offsite" or "upd vol * wherestg=copypool whereacc=readw access=offsite", check them out from the library and take them offsite

3a) DO NOT FORGET TO BACKUP YOUR DB AND TAKE IT OFFSITE AS WELL

4) migrate data from diskpool to LTO1POOL (it moves daily backups to onsite tapes) - this can be done using "upd stg diskpool highmig=1 lowmig=0" command - change these back to 90/70 after the end of reclamation



Now you have two copies on tape onsite and offsite. These tapes hold BOTH active and inactive versions of your backups- identical copies.



To achieve the second goal (archiving the data monthly for 1 year period) there are more ways how to do it



1) schedule archives instead of backups at the end of month ad follow previous scheme - the archives will be kept onsite as well as offsite - can be "space consuming"



2) define new primary pool on LTO1CLASS - for example ArchiveTape, update storage pool archive so its nextpool is ArchiveTape (or define a new disk-class pool between them - so the archive goes to disk first) and then run archives monthly - after successfull archive (and migration to the tape if you use disk-class pool between) update corresponding volume to "unavailable" access - so the TSM does not try to mount it and take them offsite with appropriate DB backup - this one is not "clean" (taking primary tapes offsite)but it should work



It can be done in more ways - this is the first shot to point you in the possible direction. However with a single drive you are very limited - persuade the responsible ones that you need another - for tape reclamations and tape-to-tape operations. When I was on TSM course there was a big poster on the wall saying "NEVER EVER IMPLEMENT A TSM WITH A SINGLE DRIVE"



Another thing - DRM is a usefull tool but basically it offers "volume tracking" (and yes, some more ......) - it tells you what tapes you need to take offsite and what return back - you can live without it (you are going to be scripting guru after few months )



If you have further questions, do not hesitate to contact me





Hope it helps



Harry

 

[srp]

Harry,



Excellent info. Thank-you!

I have 3 primary questions below.



Yep, I agree about the "one drive" issue and should be slapped on the nose with a newspaper for implementing TSM with only one drive - past budgetary concerns...sigh.

I have created a budget request and am waiting for approval for a second LTO2 or 3 drive and DRM next year.



First, changed my backupool to this:



BACKUPPOOL DISK 0.0 M 0.0 0.0 90 70 DISKPOOL



Then, I have implemented thru scripting the process you described from 1 - 4, this works great.



1. How do I schedule reclamation after "migration"? (How/when will I know it is finished?)

In my logs it started right when I set my diskpool to high=100 low=99 so I could run my daily backups without the data migrating to my tapepool.(ltopool1)



<snippet>



08/15/2005 17:30:01 ANR2561I Schedule prompter contacting WILMSN1 (session 3)

to start a scheduled operation. (SESSION: 3)

08/15/2005 17:30:02 ANR0403I Session 3 ended for node WILMSN1 (). (SESSION: 3)

08/15/2005 17:30:03 ANR0406I Session 4 started for node WILMSN1 (Linux86)

(Tcp/Ip wilmsn1.us.war.weirgroup.net(57082)). (SESSION:

4)

08/15/2005 17:30:04 ANR0407I Session 5 started for administrator ADMIN

(Linux86) (Tcp/Ip wilmsn1.us.war.weirgroup.net(57084)).

(SESSION: 5)

08/15/2005 17:30:04 ANR2017I Administrator ADMIN issued command: UPDATE

STGPOOL DISKPOOL highmig=100 lowmig=99 next=LTOPOOL1

(SESSION: 5)

08/15/2005 17:30:04 ANR2202I Storage pool DISKPOOL updated. (SESSION: 5)

08/15/2005 17:30:04 ANR0405I Session 5 ended for administrator ADMIN

(Linux86). (SESSION: 5)

08/15/2005 17:49:47 ANR0984I Process 3 for SPACE RECLAMATION started in the

BACKGROUND at 17:49:47. (PROCESS: 3)

08/15/2005 17:49:47 ANR4930I Reclamation process 3 started for primary storage

pool LTOPOOL1 automatically, threshold=60, duration=None.

(PROCESS: 3)





<snippet>







2. Your advice:



a) schedule archives instead of backups at the end of month ad follow previous scheme - the archives will be kept onsite as well as offsite - can be "space consuming"



I was told not to copy archive data to a copypool.

Because of the nature - incremental changes only?

Would we not get the full data, only changes? Or is that just from LTOPOOL1 copies?

What don't understand about how COPYPOOL's work?



3. I setup an ARCHIVE Tape & disk stg Pools:



ARCHIVEDISK DISK 0.0 M 0.0 0.0 90 70 ARCHIVETAPE

ARCHIVEPOOL DISK 0.0 M 0.0 0.0 90 70 ARCHIVEDISK

ARCHIVETAPE LTOCLASS1 0.0 M 0.0 0.0 90 70



I suppose it wouldn't hurt to add space to the ARCHIVEDISK pool?

I would have to take it away from the primary diskpool!



DISKPOOL DISK 522 G 0.0 0.0 0 0 LTOPOOL1



Or can I leave it as is?



I'll stop there, hope to hear from you.

You have been a world of help!



SRP

 

[Harry_Redl]

Hi,



1) there is no TSM way how to find that migration is over. Migration is automated process which occurs when the pool occupancy reaches a highmig threshold and ends when it gets below lowmig threshold. So to end the migration raise the thresholds above the occupancy - but unfortunatelly the migration does have to end immediatelly

Possible way is to check this at the OS level (use dsmadmc command for getting process info and parse it).

reclamation occurs when the reclaim threshold (percentage of WASTED space on the volume) of the full onsite or offsite volume is reached - it is not highmig or lowmig threshold related - so it is started automatically as well



2) archives are someting like snapshots of the current state - it copies the currrent versions from the client. Archives are independent on backups so they do not mix. Take them as a full backup which does not interfere with your "incremental forever" backups

There is no technical reason for not having them in the same copypool as backups



3)if you are going to use archives, your ARCHIVEPOOL should be large enough to hold the full backup of your nodes





I recommend you to download and read some TSM manuals - it seems to me you are a bit confused in the TSM concepts (no problem - we all were beginners once) - it may bring more light in your problem.



You can find them here

http://publib.boulder.ibm.com/infocenter/tivihelp/v1r1/index.jsp



Hope it helps



Harry

 

[srp]

Hi Harry,



Thanks again for the quick response.



I actually can predict migration by how much data is backed up.

So, at about 50 GB hr for the diskpool -> tapepool process, I have set my reclamtion to start during the early day after a night's backup.

I have plenty of time to run it durong the day hours and can give breaks for check out/in of tape vols.

Yep, and I could get proc info from one of my Linux bash shell scripts and grep, cut, or awk out any info I need to check status!

Maybe use a perl script if needed, eh?

Have you written any perl or shell scripts for TSM?



I am creating a new large pool for my archivepool/archive tape vols as you suggested.

I chose that option for the "needed" space for my onsite(primary tapepool)tape vols.



I do use the TSM manuals and find blogs or psots in google and have referred to them whenever possible.

Of course, I should always check the literature available to me before bugging a nice chap like yourself.

I think adsm.org is great and hope the website will always be available for rookies like myself when "we" feel like we are at the end of our ropes and panicking. <smile>

Individuals like like yourself help calm the ripples of stress...



I really appreciate your insight and experience of TSM.

Thank-you very much for all of the advice.



SRP

 

[Harry_Redl]

Hi,



I have just proven myself being wrong in my last post

In TSM 5.3 there is a new command MIGRATE STGPOOL which can run migration process without changing stgpool threshold. Moreover it lets you to use wait=yes parameter so it can be used in scripts to run consequent processes



See "help migr" to learn about all new options



.. and the same for reclamation - new command RECLAIM STGPOOL



You can see that nobody is perfect



Hope it helps and sorry for that



Harry