Curtis,
Yes, that is correct. I am remembering all of the PAIN associated with having
NetBackup Vault make copies to physical tape. It went something like this:
1) do the backup to VTL
2) vault images from VTL to physical tape
3) keep track of the stuff that has been vaulted and then expire images
4) once all images on a tape have been expired, then do a "bpexpdate
-deassignempty" on a virtual tape (which, by the way, Symantec told us there
was a rare, but fatal bug with this method).
5) try to keep Vault running ahead of the backups so that there would be enough
virtual tapes available to use for backups
6) rerun failed vault jobs to allow for proper image expiration
7) repeat steps 1 through 6 several times
Arrrrgghhh the pain!!!
-----Original Message-----
From: Curtis Preston [mailto:cpreston AT glasshouse DOT com]
Sent: Monday, September 24, 2007 3:56 PM
To: Liddle, Stuart; Paul Keating; Clem Kruger; VERITAS-BU AT mailman.eng.auburn
DOT edu
Subject: RE: Re: [Veritas-bu] Script to label expired tapes in a VTL
I think you nailed it, Stu. I remember your previous posts on this topic, and
that you said you went to this method because the NBU Vault method wasn't
duping the tapes fast enough for you. As I recall, it was because your backups
had millions of files in them, and this was slowing down your dupe process.
You went to the tape-out functionality of your VTL because it made the copies
faster (significantly so), and were willing to live with any downsides because
it made the copies possible.
You are correct. Most people do not use their VTLs this way. Part of the
reason is a good amount of FUD put out by the backup vendors. Another reason
is that it does come with some major downsides. In your case, you had to
choose which downsides were worse, and in your case, the downside of not
getting the copies done at all was unacceptable, so you decided to deal with
the downsides of the other method.
---
W. Curtis Preston
Backup Blog @ www.backupcentral.com
VP Data Protection, GlassHouse Technologies
-----Original Message-----
From: Liddle, Stuart [mailto:liddles AT amgen DOT com]
Sent: Monday, September 24, 2007 4:55 PM
To: Curtis Preston; Paul Keating; Clem Kruger; VERITAS-BU AT mailman.eng.auburn
DOT edu
Subject: RE: Re: [Veritas-bu] Script to label expired tapes in a VTL
I think I'm beginning to understand my confusion to some of the earlier
comments and now see from the conversations that are taking place that the way
we are using our VTL's is vastly different from the way that most of the
respondents to this thread are using theirs.
Correct me if I'm wrong, but most of you have been describing the method for
using your VTL's as one of what I'll call "self-contained" VTL's and that there
is no real connection to the physical tapes. In other words, in order to get
data from the virtual tapes you have to use Vault or some other method of
duplication to get the data off-site. So, the tapes are always in the VTL and
are re-cycled just as physical tapes are in a physical tape library. Hence the
need to delete and/or re-label.
We abandoned that method in favor of having our VTL's manage a partition of the
physical tape library and have a direct link to the physical tapes. In other
words, a barcode in the virtual library is the same as a barcode on a physical
tape.
So, for us, when a tape is full or we want to eject it and send it offsite, it
gets cloned to physical tape and it's no longer "visible" to NetBackup in the
virtual library. But, we have set up our VTL to use a "shadow" pool where the
virtual tape is still available until the VTL needs the space. In some
instances, the data can be available for up to 10 days.
If we need a tape for a restore, we just import it read-only from the "shadow"
pool and inventory the virtual library in NetBackup and off we go. When we are
done, we just eject it from the virtual library and since it's already in sync
with the physical tape, nothing more is required.
Now, this might sound "messy" from the standpoint that you don't know where the
physical tape really is according to NetBackup....as far as it knows, it's just
not in the library and since you didn't use Vault, it's whereabouts are
unknown. In our experience this doesn't matter because we have a separate app
to track our off-site tapes.
So, the whole discussion about re-labeling the virtual tapes is just an
interesting discussion to me because we don't do that method.
--stuart
-----Original Message-----
From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Curtis
Preston
Sent: Monday, September 24, 2007 12:15 PM
To: Paul Keating; Clem Kruger; VERITAS-BU AT mailman.eng.auburn DOT edu
Subject: Re: [Veritas-bu] Script to label expired tapes in a VTL
The problem is not de-dupe; the problem is thin provisioning and
oversubscription. There are a lot of VTLs that allow both (with or w/o
de-dupe), and if you define more tapes than you actually have disk, you will
have this problem. I'll concede that oversubscription is a natural state in a
de-dupe VTL, as you define probably 20 times more tapes than you have "real"
storage for.
The problem I see you're describing is this:
1. You define more tapes than you really have capacity for (again, this is
normal in the de-dupe world)
2. You have a bunch that are partially full
3. You have a bunch that are in scratch, but have not been relabeled.
4. You may even have some that are brand new tapes that haven't been used at
all.
5. You're out of real free space
6. Since NBU prefers to append rather than use a new tape, it will grab the #2
tapes before the #3 tapes. Had it grabbed the #3 tapes, it would have cleared
the space they're taking up and you wouldn't have the problem, but that's not
how NBU (or any other backup s/w) works.
Here's a thought. Isn't there a way to tell NBU to mark tapes
full/frozen/something after a backup? If you did that, additional backups
wouldn't append to those tapes, and you wouldn't have to worry about labeling.
Would that work?
Of course, this whole labeling thing really isn't that big of a deal. A simple
shell script could handle it easy enough.
---
W. Curtis Preston
Backup Blog @ www.backupcentral.com
VP Data Protection, GlassHouse Technologies
-----Original Message-----
From: Paul Keating [mailto:pkeating AT bank-banque-canada DOT ca]
Sent: Monday, September 24, 2007 9:34 AM
To: Curtis Preston; Clem Kruger; VERITAS-BU AT mailman.eng.auburn DOT edu
Subject: RE: [Veritas-bu] Script to label expired tapes in a VTL
Not entirely true, Curtis.
When your virutal tape expires, the VTL has no way of knowing this
untill the tape is written to again.
Depending on the VTL, this may be too late.
I've got about 2TB of "free" space on my VTL, and about 1000 scratch VTs
(2800 total).
After a couple of weeks of Netbackup using and expiring VTs, The VTs are
going back to Scratch, but untill they're re-written, the pointers in
the repository still exist, therefore the VTL thinks the space is still
in use....so if you're using a de-duping VTL (or DSU to a de-duping FS)
and you're only doing "real time" freeing of space (ie, re-labelling a
VT only when you re-write it, or letting NBU delete images from the DSU
only as the DSU approaches a high disk utilization threshold), then your
target will need to do some high-perf defragging in order to provide you
with sufficient "free" space to write more images.
Non-de-dup TLs such as Quantums, which "hard" allocate a fixed chunk of
disk for each cart will not have this problem, nor will a DSU on a
standard FS, but with my current VTL, it's definitely a requirement to
occasionally kick off a script to "bplabel" recently expired tapes, so
that the defrag process can run at a higher priority (when IO to the TL
is lower.)
Paul
--
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf
> Of Curtis Preston
> Sent: September 22, 2007 5:15 AM
> To: Clem Kruger; VERITAS-BU AT mailman.eng.auburn DOT edu
> Subject: Re: [Veritas-bu] Script to label expired tapes in a VTL
>
> Oversubscription aside, once his tapes are expired, the space taken up
> by those tapes is immediately available for reuse. The next time the
> tape gets written to, it will delete all pointers to the
> space taken up
> by that tape.
>
> As to the VTL vs disk debate, I still think you should bring
> in all disk
> devices and let them duke it out before excluding an entire
> category of
> them. You're going to exclude a lot of really good products
> if you just
> "no VTLs."
>
> Remember that saying "I don't want a VTL but I do want de-dupe" means
> that you're going to use NAS. While that will meet a whole
> lot of needs
> for a whole lot of people, there's also some really big backups that
> need a lot more than you can push over IP. For those backups, you're
> going to want a block transfer protocol (i.e. SCSI), and for that,
> you're currently going to be buying a VTL. (Unless you're
> just going to
> buy a non-deduped disk in which case I'd say you're REALLY
> wasting your
> money.)
>
> ---
> W. Curtis Preston
> Backup Blog @ www.backupcentral.com
> VP Data Protection, GlassHouse Technologies
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf Of Clem
> Kruger
> Sent: Saturday, September 22, 2007 4:24 AM
> To: VERITAS-BU AT mailman.eng.auburn DOT edu
> Subject: Re: [Veritas-bu] Script to label expired tapes in a VTL
>
> Hi Steve,
>
> This is the downer on VTL's. You do not get your "tape" space back
> automatically. It is for these reasons I recommend that one never go
> VTL's. NetBackup 6.0 and 6.5 allow disk to disk backups; the
> images are
> easily replicated to an offsite facility.
>
> The time for all "tape" has come and gone. The de-duplication facility
> in 6.5 makes life even easier. Why VTL's (which does SCSI emulation)
> when you and use disk which is faster and has more protection?
>
> Clem.
>
> -----Original Message-----
> From: veritas-bu-bounces AT mailman.eng.auburn DOT edu
> [mailto:veritas-bu-bounces AT mailman.eng.auburn DOT edu] On Behalf
> Of swaltner
> Sent: 21 September 2007 17:32 PM
> To: VERITAS-BU AT mailman.eng.auburn DOT edu
> Subject: [Veritas-bu] Script to label expired tapes in a VTL
>
>
> We deployed a VTL last month, which has been working very nicely. This
> is in a NetBackup 5.1 environment with the VTL attached to our Solaris
> based master server as well as to our NAS server for local
> NDMP backups.
> One thing I'd like to do is over-subscribe on the back-end
> storage, but
> before I do that I'd like to automate the process of freeing
> up the disk
> space used in the VTL when a NetBackup tape is expired. Just
> curious if
> anyone has already written such a beast and would like to
> share with me
> as a starting point.
>
> If not, I suspect I'll use the following logic:
>
> - Every day (at noon??), query the robots defined in the VTL
> and keep a
> record of tapes that are allocated.
> - When a tape goes from allocated to non-allocated from one day to the
> next, use a command like the following to erase the tape's contents:
> bplabel -erase -o -d dlt -m VTL123
>
> This would write a small label at the beginning of the virtual tape,
> causing the VTL to drop all the other data that had been stored on the
> tape.
>
> Any reason this wouldn't work? Any gotchas with writing this
> script that
> I should look out for?
>
> Steve
>
> +-------------------------------------------------------------
> ---------
> |This was sent by steve.waltner AT lsi DOT com via Backup Central.
> |Forward SPAM to abuse AT backupcentral DOT com.
> +-------------------------------------------------------------
> ---------
>
>
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
> _______________________________________________
> Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
====================================================================================
La version française suit le texte anglais.
------------------------------------------------------------------------------------
This email may contain privileged and/or confidential information, and the Bank
of
Canada does not waive any related rights. Any distribution, use, or copying of
this
email or the information it contains by other than the intended recipient is
unauthorized. If you received this email in error please delete it immediately
from
your system and notify the sender promptly by email that you have done so.
------------------------------------------------------------------------------------
Le présent courriel peut contenir de l'information privilégiée ou
confidentielle.
La Banque du Canada ne renonce pas aux droits qui s'y rapportent. Toute
diffusion,
utilisation ou copie de ce courriel ou des renseignements qu'il contient par une
personne autre que le ou les destinataires désignés est interdite. Si vous
recevez
ce courriel par erreur, veuillez le supprimer immédiatement et envoyer sans
délai à
l'expéditeur un message électronique pour l'aviser que vous avez éliminé de
votre
ordinateur toute copie du courriel reçu.
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
_______________________________________________
Veritas-bu maillist - Veritas-bu AT mailman.eng.auburn DOT edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
|
