Steve, Kelly,
I inherited this ADSM installation and this is the customer's requirements. To
be able to fit into the customer's requirements, this is the only way to do it.
The first reason they are doing it this way around is cost. There is five huge
Oracle Databases of 300GB each that is backed up using SQL Backtrack. These
backups are full online backups every day, passing through a disk storage pool
before been migrated to StoragTek Redwood Tapes. These tapes can hold a lot of
data, but is expensive. In order to make use of copypools, they need double
the amount of tapes currently in use. There is no budget for this. It is
difficult to convince them to buy 10 more tapes. That is for the oracle
database backups.
The customer have a 18 node SP installed. The system backups (user,
application, etc. data), also goes to a disk storage pool and then gets
migrated to StorageTek Timberline tapes. Again, to cut down on cost, no budget
for more tapes to be able to do copypool.
To cut down on the backup window, these full "system" backups is spread over
the full week e.g.. Node1, 5, 7 on Monday, Node2, 6, 8 on Tuesday etc. etc.
Also, we only do full backups on some filesystems, not all of them (script to
do this), otherwise Incremental backups for the rest.
Hope this explains the scenario.
Kelly, I think you are missing the point. The original problem we had is that
ADSM backup/archive changes the access time of files in a HSM controlled
filesystem, and thus void the ADSM Automignouse functionality of ADSM HSM. I
really don't think that is the way ADSM is suppose to work. ADSM HSM is part
of ADSM, and thus should co-exist and compliment ADSM. Don't you think so?
Thanks and regards
Gerrit
Steve Harris wrote:
> Gerrit,
>
> Unless you are backing up a huge amount of "temporary" data, wouldn't you
> be better off to still use a copypool but only run the backup stg command
> on a weekly basis?
>
> With your method you are backing up stuff that you already have, and will
> need a sizeable backup window. The copypool approach will only copy
> changed data and will not impact the client system.
>
> I'm sure you have your reasons for doing it your way - but I'd really like
> to know what they are.
>
> Steve.
>
> ADSM etc Guy
> The Wesley Hospital, Brisbane Australia
>
> Gerrit van Zyl <gvanzyl AT FARITEC.CO DOT ZA>@VM.MARIST.EDU> on 10/04/2000
> 22:29:12
>
> Please respond to "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
> To: ADSM-L AT VM.MARIST DOT EDU
> cc:
> Fax to:
> Subject: Re: HSM not migrating
>
> Reinhard,
>
> Because of the huge amount of backups we are doing, we cannot use copy stg
> pools to create offsite copies. So we develop out own scripts to determine
> which "live" tapes to take offsite and which ones to return to the robot.
> To guarentee we always have a copy of any file offsite, we need to once a
> week do a full selective backup.
>
> In any case here is the selective backup of a file : istat iostat.25nov
>
> File istat before selective backup :
>
> 02n09sw /bgs/best1 # istat iostat.25nov
> Inode 2060 on device 39/1 File
> Protection: rw-r--r--
> Owner: 500(best1) Group: 1(staff)
> Link count: 1 Length 71731 bytes
>
> Last updated: Thu Nov 25 10:34:54 1999
> Last modified: Thu Nov 25 10:34:54 1999
> Last accessed: Wed Apr 05 12:12:53 2000
>
> Doing selective backup with dsmnode -t=p:
>
> /bgs/best1 # dsmmode -t=p dsmc sel iostat.25nov
> ADSTAR Distributed Storage Manager
> space management Interface - Version 3, Release 1, Level 0.7
> (C) Copyright IBM Corporation, 1990, 1999. All Rights Reserved.
> ADSTAR Distributed Storage Manager
> Command Line Backup Client Interface - Version 3, Release 1, Level 0.7
> (C) Copyright IBM Corporation, 1990, 1999, All Rights Reserved.
>
> Selective Backup function invoked.
>
> Node Name: F02N09SW-N25
> Session established with server ADSM: AIX-RS/6000
> Server Version 3, Release 1, Level 2.55
> Server date/time: 04/10/00 14:15:47 Last access: 04/10/00 10:59:57
>
> Normal File--> 71,731 /bgs/best1/iostat.25nov [Sent]
> Selective Backup processing of 'iostat.25nov' finished without failure.
>
> Total number of objects inspected: 1
> Total number of objects backed up: 1
> Total number of objects updated: 0
> Total number of objects rebound: 0
> Total number of objects deleted: 0
> Total number of objects failed: 0
> Total number of bytes transferred: 70.07 KB
> Data transfer time: 0.00 sec
> Network data transfer rate: 10,909.58 KB/sec
> Aggregate data transfer rate: 47.64 KB/sec
> Objects compressed by: 0%
> Elapsed processing time: 00:00:01
>
> File istat after selective backup :
>
> /bgs/best1 # istat iostat.25nov
> Inode 2060 on device 39/1 File
> Protection: rw-r--r--
> Owner: 500(best1) Group: 1(staff)
> Link count: 1 Length 71731 bytes
>
> Last updated: Thu Nov 25 10:34:54 1999
> Last modified: Thu Nov 25 10:34:54 1999
> Last accessed: Mon Apr 10 14:15:55 2000
>
> As you can see, the access time has change.
>
> Regards
> Gerrit
>
> Reinhard Mersch wrote: Gerrit,
>
> I can not reproduce that. In my environments, a "dsmmode -t p dsmc sel
> file"
> leaves the file's atime unchanged.
>
> But even if the backup changes the atime: A file that has not been accessed
> for 30 days should not have been backed up for 30 days because there was no
> need to. So the atime should nevertheless be old. It looks quite strange to
> me. What are you doing there?
>
> Gerrit van Zyl writes:
> > Reinhard,
> >
> > Even using the dsmmode -t=p option when doing a selective backup,
> changes the atime of the file.
> > I've just tested it. Using "istat filename", one can see the access
> time for that file. The dsmmode
> > -t=p option does not preserve the access time as they claim. So this is
> a huge problem for me. We can
> > only migrate files on our /R3_data filesystem if the files have not been
> accessed for at least 30 days.
> > Any other ideas as on how to solve this problem?
> >
> > Regards
> > Gerrit
> >
> >
> > Reinhard Mersch wrote:
> >
> > > Gerrit van Zyl writes:
> > > > Reinhard,
> > > >
> > > > I had a look:
> > > >
> > > > 1. No errors in the AIX errorlog.
> > > > 2. No, I do not catch HSM error messages with the ERRORPROG
> option. How do you use this?
> > >
> > > What I am doing is: In dsm.sys:
> > > ERRORPROG /var/adsm/errorprog
> > > /var/adsm/errorprog is a shell script essentially containing one line:
> > > mail -s "Mail from HSM" backup
> > > This sends the HSM error message to "backup" (which is me). Don't
> forget
> > > to make that script executable.
> > >
> > > > 3. Dsmmonitord is running.
> > > > 4. This is a huge filesystem, and no, it is impossible that all of
> the files have been accessed.
> > >
> > > You are backing them up, don't you? (See below.)
> > >
> > > > 5. Dsmreconcile gives the following:
> > > > f03n13sw / # dsmreconcile -c /R3_data
> > > > ADSTAR Distributed Storage Manager
> > > > space management Interface - Version 3, Release 1, Level 0.7
> > > > (C) Copyright IBM Corporation, 1990, 1999. All Rights Reserved.
> > > >
> > > >
> > > > Reconciling '/R3_data' file system:
> > > > Querying the ADSM server for a list of backed up files...
> > > > Received 705752 entries
> > >
> > > Boy, this IS huge. There seem to be a lot of small files within the
> > > filesystem. Don't forget that small files (4KB or less) will not be
> > > migrated! Does this explain your problem?
> > >
> > > > Now traversing the file system...
> > > > Writing the new migration candidates list...
> > > > Note: unable to find any candidates in the file system.
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > Obviously, this is the problem. I can only guess about the reason. All
> > > possible candidates smaller than 4KB? To repeat myself: I would set
> > > AUTOMIGNOUSE to 0. APAR IX89650 might apply to you (text appended
> below).
> > >
> > > > ANS9250I File system '/R3_data' reconciliation completed.
> > > >
> > > > 6. I am able to manually migrate files.
> > > > 7. Set the quota (e.g. "dsmmigfs update /R3_data -Quota=1024G"
> > > > you do allow 1 TB from this filesystem to be migrated, don't
> you?)
> > > > What do you mean with this?
> > >
> > > The quota specifies the amount of data that may be migrated. You did
> not
> > > specify a quota, I would definitely do so. (The default is quite
> small.)
> > >
> > > APAR IX89650:
> > >
> > > APAR NUMBER: IX89650 RESOLVED AS: DOCUMENTATION ERROR
> > >
> > > ABSTRACT:
> > > IX89650: UNABLE TO SPACE MANAGE FILES BASED ON AUTOMIGNONUSE=1 AS
> ADSM
> > > INCREMENTAL AND SELECTIVE BACKUPS ALTER THE FILES ATIME VALUE.
> > >
> > > ORIGINATING DETAILS:
> > > The ADSM backup/archive client alters files atime value during
> > > selective or incremental backups. This results in HSM being
> > > unable to properly migrate files based on the automignonuse
> > > value set in the mgmtclass. This was tested on AIX 4.2.1 with
> > > the 3.1.0.6 client code.
> > >
> > > LOCAL FIX AS REPORTED BY ORIGINATOR:
> > > Run the selective or incremental backup as a command parameter
> > > of the dsmmode -t=p command. Example:
> > > 'dsmmode -t=p dsmc i'
> > >
> > > RESPONDER SUMMARY:
> > > The BA client in ADSM v2 preserved the atime
> > > of files for backup and restore. Preserving the atime
> > > of files for reading has been disabled 11/97. Reason:
> > > when preserving the atime (using syscall utime(), the ctime
> > > of the file was indirectly updated. HSM uses ctime as part of
> > > the key for the premigration database. The ba client uses the
> > > ctime (besides other file attributes) to compare the state
> > > of files. Unfortunately, this modification has never been
> > > reflected in the documentation, since it limits the use of the
> > > 'non-usage' option of the ADSM HSM client.
> > >
> > > RESPONDER CONCLUSION:
> > > Trying to preserve the time stamp of a file,
> > > using syscall utime(), indirectly changes the ctime time stamp,
> > > which leads to undesired side affects.
> > >
> > > --
> > > Reinhard Mersch Westfaelische
> Wilhelms-Universitaet
> > > Zentrum fuer Informationsverarbeitung - ehemals
> Universitaetsrechenzentrum
> > > Roentgenstrasse 9-13, D-48149 Muenster, Germany Tel:
> +49(251)83-31583
> > > E-Mail: mersch AT uni-muenster DOT de Fax:
> +49(251)83-31653
> > <!doctype html public "-//w3c//dtd html 4.0 transitional//en">
> > <html>
> > Reinhard,
> > <p>Even using the dsmmode -t=p option when doing a selective backup,
> changes
> > the atime of the file.
> > <br>I've just tested it. Using "istat <i>filename", </i>one can
> see
> > the access time for that file. The dsmmode -t=p option does not
> preserve
> > the access time as they claim. So this is a huge problem for me.
>
> > We can only migrate files on our /R3_data filesystem if the files have
> > not been accessed for at least 30 days. Any other ideas as on how
> > to solve this problem?
> > <p>Regards
> > <br>Gerrit
> > <br>
> > <p>Reinhard Mersch wrote:
> > <blockquote TYPE=CITE>Gerrit van Zyl writes:
> > <br> > Reinhard,
> > <br> >
> > <br> > I had a look:
> > <br> >
> > <br> > 1. No errors in the AIX errorlog.
> > <br> > 2. No, I do not catch HSM error messages with the
> ERRORPROG
> > option. How do you use this?
> > <p>What I am doing is: In dsm.sys:
> > <br> ERRORPROG
>
> > /var/adsm/errorprog
> > <br>/var/adsm/errorprog is a shell script essentially containing one
> line:
> > <br> mail -s "Mail from HSM" backup
> > <br>This sends the HSM error message to "backup" (which is me). Don't
> forget
> > <br>to make that script executable.
> > <p> > 3. Dsmmonitord is running.
> > <br> > 4. This is a huge filesystem, and no, it is impossible
> > that all of the files have been accessed.
> > <p>You are backing them up, don't you? (See below.)
> > <p> > 5. Dsmreconcile gives the following:
> > <br> > f03n13sw / # dsmreconcile -c /R3_data
> > <br> > ADSTAR Distributed Storage Manager
> > <br> > space management Interface - Version 3, Release 1, Level 0.7
> > <br> > (C) Copyright IBM Corporation, 1990, 1999. All Rights
> Reserved.
> > <br> >
> > <br> >
> > <br> > Reconciling '/R3_data' file system:
> > <br> > Querying the ADSM server for a list of backed up
> > files...
> > <br> >
> Received
> > 705752 entries
> > <p>Boy, this IS huge. There seem to be a lot of small files within the
> > <br>filesystem. Don't forget that small files (4KB or less) will not be
> > <br>migrated! Does this explain your problem?
> > <p> > Now traversing the file system...
> > <br> > Writing the new migration candidates list...
> > <br> > Note:
> > unable to find any candidates in the file system.
> > <br>
> > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > <br>Obviously, this is the problem. I can only guess about the reason.
> > All
> > <br>possible candidates smaller than 4KB? To repeat myself: I would set
> > <br>AUTOMIGNOUSE to 0. APAR IX89650 might apply to you (text appended
> below).
> > <p> > ANS9250I File system '/R3_data' reconciliation completed.
> > <br> >
> > <br> > 6. I am able to manually migrate files.
> > <br> > 7. Set the quota (e.g. "dsmmigfs update /R3_data
> -Quota=1024G"
> > <br> > you do allow 1 TB from this
> filesystem
> > to be migrated, don't you?)
> > <br> > What do you mean with this?
> > <p>The quota specifies the amount of data that may be migrated. You did
> > not
> > <br>specify a quota, I would definitely do so. (The default is quite
> small.)
> > <p>APAR IX89650:
> > <p>APAR NUMBER: IX89650
>
> > RESOLVED AS: DOCUMENTATION ERROR
> > <p> ABSTRACT:
> > <br> IX89650: UNABLE TO SPACE MANAGE FILES BASED ON
> AUTOMIGNONUSE=1
> > AS ADSM
> > <br> INCREMENTAL AND SELECTIVE BACKUPS ALTER THE FILES ATIME VALUE.
> > <p> ORIGINATING DETAILS:
> > <br> The ADSM backup/archive client alters files atime value during
> > <br> selective or incremental backups. This results in HSM
> being
> > <br> unable to properly migrate files based on the automignonuse
> > <br> value set in the mgmtclass. This was tested on AIX 4.2.1 with
> > <br> the 3.1.0.6 client code.
> > <p> LOCAL FIX AS REPORTED BY ORIGINATOR:
> > <br> Run the selective or incremental backup as a command parameter
> > <br> of the dsmmode -t=p command. Example:
> > <br> 'dsmmode -t=p dsmc i'
> > <p> RESPONDER SUMMARY:
> > <br> The BA client in ADSM v2 preserved the atime
> > <br> of files for backup and restore. Preserving the atime
> > <br> of files for reading has been disabled 11/97. Reason:
> > <br> when preserving the atime (using syscall utime(), the ctime
> > <br> of the file was indirectly updated. HSM uses ctime as part of
> > <br> the key for the premigration database. The ba client uses the
> > <br> ctime (besides other file attributes) to compare the state
> > <br> of files. Unfortunately, this modification has never been
> > <br> reflected in the documentation, since it limits the use
> > of the
> > <br> 'non-usage' option of the ADSM HSM client.
> > <p> RESPONDER CONCLUSION:
> > <br> Trying to preserve the time stamp of a file,
> > <br> using syscall utime(), indirectly changes the ctime time
> stamp,
> > <br> which leads to undesired side affects.
> > <p>--
> > <br>Reinhard Mersch
>
>
> > Westfaelische Wilhelms-Universitaet
> > <br>Zentrum fuer Informationsverarbeitung - ehemals
> Universitaetsrechenzentrum
> > <br>Roentgenstrasse 9-13, D-48149 Muenster, Germany
>
> > Tel: +49(251)83-31583
> > <br>E-Mail: mersch AT uni-muenster DOT
> de
>
>
> > Fax: +49(251)83-31653</blockquote>
> > </html>
> > begin:vcard
> > n:;
> > x-mozilla-html:FALSE
> > adr:;;;;;;
> > version:2.1
> > note;quoted-printable: ///=0D=0A
> (o o)=0D=0A+----------oOO--(_)
> ------------+=0D=0A| |=0D=0A| Gerrit
> van Zyl |=0D=0A| Snr Consultant |=0D=0A|
> Faritec (Pty) Ltd |=0D=0A| gvanzyl AT faritec.co DOT za |=0D=0A|
> 0825704266 |=0D=0A|
> |=0D=0A+---------------------oOO-------+=0D=0A
> |_|_|=0D=0A /| |\=0D=0A______
> _ooO_Ooo___________=0D=0A=0D=0A
> > end:vcard
>
> --
> Reinhard Mersch Westfaelische Wilhelms-Universitaet
> Zentrum fuer Informationsverarbeitung - ehemals Universitaetsrechenzentrum
> Roentgenstrasse 9-13, D-48149 Muenster, Germany Tel: +49(251)83-31583
> E-Mail: mersch AT uni-muenster DOT de Fax:
> +49(251)83-31653
>
> ===========================================
> This email message has been swept by
> MIMESweeper at The Wesley Hospital - Australia
> ===========================================
>
> (See attached file: gvanzyl.vcf)
|
