Veritas-bu

[Veritas-bu] Checking to see if millions of files are backed up?

2007-03-27 00:05:51
Subject: [Veritas-bu] Checking to see if millions of files are backed up?
From: Anderson.Mccammont at morganstanley.com (McCammont, Anderson (IT))
Date: Tue, 27 Mar 2007 05:05:51 +0100
As a colleague of mine once put it: 'BANANA - Backups Are Not Archives,
NOT ARCHIVES'.  Practise this mantra.  You should really think about
rearchitecting.

That said, if your data isn't important to your firm and you're
confident that you can restore from backups (you're making multiple
copies, probably not using removable media but if you are you're doing
full media verifies and sending them out the building to different
offsite facilities in different directions using different carriers
right?), and you're looking at this as an interim poor man's one-way HSM
(you know your data access patterns and want to keep things around on
disk as long as possible but manage your primary storage), then you
could 'trust' your backup app, find the last successful backup date at
the appropriate retlevel contains the file you want, and the script that
you trigger when your filesystem utilisation gets to a certain point is
free to remove anything before that date.  No, I wouldn't do that
either.  

The point is that you're either comfortable in your backup app or you're
not, and if you're not you can't get out of doing anything other than
reading the tape.  If you trust your backup app but not bparchive you
can get your backup app to dump a filelist.  If that's millions of files
and impacts on other uses of your backup plant then that's something you
need to deal with.

Alternatively, if you're not prepared to trust bparchive, or it's not
timely enough for you (I've never had occasion to use it so can't
comment on either point), you could always generate your own filelist
that you use to take the regular backup, check the success of the backup
and if it worked 'bless' the filelist.  Then at some point in the future
when you get into space pressure you can purge files in the blessed
filelists.  Be careful though to only bless fully successful backups
(not partials - think locked files etc), but this really falls into the
category of trusting the backup application, and hence becomes a
circular argument.  I'd be more inclined to test bparchive rather than
engineer my own, but remember, BANANA.


Separate solutions for separate problems:
Stretched HA for hardware failure/Disaster Recovery
CDP/Snapshots/Backups for User Error
Archives for Data Retention


-----Original Message-----
From: veritas-bu-bounces at mailman.eng.auburn.edu
[mailto:veritas-bu-bounces at mailman.eng.auburn.edu] On Behalf Of Justin
Piszcz
Sent: 27 March 2007 06:17
To: Whelan, Patrick
Cc: Veritas-bu at mailman.eng.auburn.edu
Subject: Re: [Veritas-bu] Checking to see if millions of files are
backed up?

A nice idea; however, this is actually part of a much larger and
complicated system in which certain files have to kept for certain
retentions, both on disk and backed up to tape, think tape archival
with different retention rates.  If I were to change the entire
architecture behind it, this may be a good solution and is something
on my plate for the future.  However, in the interim, I just need a
solution to verify files have been backed up to tape and remove them
if they are older than N days if we are running low on space on any
particular server.

On 3/26/07, Whelan, Patrick <Patrick.Whelan at colt.net> wrote:
> Why not have a script that runs a backup followed by an archive. Check
> the error code of the backup, if is not 0 then don't run the archive.
If
> it is 0 then run the archive which will automatically delete the files
> when it completes successfully. It will not delete antything if it
fails
> even with a 1.
>
> Regards,
>
> Patrick Whelan
>
> -----Original Message-----
> From: veritas-bu-bounces at mailman.eng.auburn.edu
> [mailto:veritas-bu-bounces at mailman.eng.auburn.edu] On Behalf Of Justin
> Piszcz
> Sent: 26 March 2007 22:36
> To: bobbyrjw at comcast.net
> Cc: Veritas-bu at mailman.eng.auburn.edu
> Subject: Re: [Veritas-bu] Checking to see if millions of files are
> backed up?
>
>
> The problem with that is two-fold:
>
> 1. We backup multiple copies of the data, therefore, the archive
option
> will not work. 2. What if a tape has an I/O error half way through the
> archive process? Yikes.
>
> Justin.
>
> On 3/26/07, Bobby Williams <bobbyrjw at comcast.net> wrote:
> > Why not set up an archive schedule?  That way, the files can be
> > archived and NetBackup will ensure that they are on tape before
> > removing.
> >
> >
> >
> >
> > Bobby Williams
> > 2205 Peterson Drive
> > Chattanooga, Tennessee  37421
> > 423-296-8200
> >
> > -----Original Message-----
> > From: veritas-bu-bounces at mailman.eng.auburn.edu
> > [mailto:veritas-bu-bounces at mailman.eng.auburn.edu] On Behalf Of
Justin
>
> > Piszcz
> > Sent: Monday, March 26, 2007 4:27 PM
> > To: Veritas-bu at mailman.eng.auburn.edu
> > Subject: [Veritas-bu] Checking to see if millions of files are
backed
> > up?
> >
> > If one is to create a script to ensure that the files on the
> > filesystem are backed upon before removing them, what is the best
> > data-store model for doing so?
> >
> > Obviously, if you have > 1,000,000 files in the catalog and you need
> > to check each of those, you do not want to bplist -B -C -R 999999
> > /path/to/file/1.txt for each file.  However, you do not want to grep
> > "1" one_gigabyte_catalog.txt either as there is really too much
> > overhead in either case.
> >
> > I have a few ideas that involves neither of these, but I was
wondering
>
> > if anyone out there had already done something similar to this that
> > was high performance?
> >
> > Justin.
> > _______________________________________________
> > Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
> > http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
> >
> >
> _______________________________________________
> Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
> http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
>
>
>
************************************************************************
*************
> The message is intended for the named addressee only and may not be
disclosed to or used by anyone else, nor may it be copied in any way.
>
> The contents of this message and its attachments are confidential and
may also be subject to legal privilege.  If you are not the named
addressee and/or have received this message in error, please advise us
by e-mailing security at colt.net and delete the message and any
attachments without retaining any copies.
>
> Internet communications are not secure and COLT does not accept
responsibility for this message, its contents nor responsibility for any
viruses.
>
> No contracts can be created or varied on behalf of COLT
Telecommunications, its subsidiaries or affiliates ("COLT") and any
other party by email Communications unless expressly agreed in writing
with such other party.
>
> Please note that incoming emails will be automatically scanned to
eliminate potential viruses and unsolicited promotional emails. For more
information refer to www.colt.net or contact us on +44(0)20 7390 3900.
>
>
_______________________________________________
Veritas-bu maillist  -  Veritas-bu at mailman.eng.auburn.edu
http://mailman.eng.auburn.edu/mailman/listinfo/veritas-bu
--------------------------------------------------------

NOTICE: If received in error, please destroy and notify sender. Sender does not 
intend to waive confidentiality or privilege. Use of this email is prohibited 
when received in error.