Hello,
@Ryan Novosielski:
Yes you are right! I did miss that post. I don't appear to have
received that bacula mailing list digest mail, and obviously depend on
them too much! I'll monitor the mailing list directly on sourceforge
also in future. Thank you.
@Timo Neuvonen:
Thanks for replying. Sorry, I did not see your post, as I never
received the mailing list digest. I hadn't spotted the difference in
Dan's post re which retention he was referring to, and I've spent the
weekend re-reading and considering my options. I will now be
redesigning the retention periods, but will need to be careful of the
database size, as it's already quite large.
Seems to achieve what I need, I'll need the file/job/volume retentions
all matching for my Daily, weekly and monthly backups, and then rely
on bscan for the Yearly one, (as I cannot keep the job/file records in
the db for up to 10 years), so that I can get individual files when I
need them no matter how old. The idea of having to restore the entire
job (which in some cases is over 300Gb) to get a single file from a
few months ago is too much, or am I missing the point there? Might be
acceptable for a file from years ago, but I presume bscan avoids the
need for this too?
I clearly run the risk of having a massive database though, so I'll
have to monitor this closely.
@Kleber Leal:
I don't think I can have larger retention periods for Files and Jobs
than Volumes, as the manual states:
"The durations inter-depend a bit because if Bacula prunes a Volume,
it automatically removes all the Job records, and all the File
records. Also when a Job record is pruned, all the File records for
that Job are also pruned (deleted) from the catalog."
I think in my case, I have not specified a Job or File retention at
all, and hence the default is applying. I need to set both of these
to the same as the Volume, so they are all available for each other
and then all expire at the same time.
Re the scratch pools, I have done exactly the same as you for the
disks Volumes, one large LVM volume, auto create labels, and limit the
volumes.
However, the issue is that the different pools will not recycle each
others disk volumes. If the Daily incremental backup is unable to
complete, as it needs more volumes than the limit I've set, and I
can't let it create more as I'm at the max of my disk space (delicate
balancing act), my problem is the Weekly and Monthly pools have 100's
of expired volumes, but the Daily backups can't recycle those and use
that space.
I think the new feature in 5.0.1 (Actiononpurge = truncate) will solve
this. The weekly and monthly pools should truncate themselves thus
leaving a lot more space in the file system for me to greatly increase
the Daily backup volume limit. Again, the balance will be that the
weekly/monthly volumes are copied/migrated and then purged from disk
before the Daily Disks volumes needs them.
@Dan Langille:
Sorry for misinterpreting your original advice. It makes more sense now!
_____
I plan on doing the following starting today:
1. Upgrade bacula 3.0.2 to 5.0.3 to take advantage of the following
new features:
a) ActionOnPurge=Truncate - stops weekly/monthly/yearly disk pools
hogging space even when the retention is only 2 days (i.e. time to get
them to tape)
b) Use the new FileRetention/JobRetention directives for Pools to
avoid doing it on each client, as they are all the same
c)Use the new speed command available in the btape program - to help
benchmark my tape drive performance. Monthly tape copies are taking
days to complete, and the subsequent daily backups are queuing up.
Need to adjust priorities also.
2. Probably switch from copy jobs to migrate jobs - Don't mind going
to tape for a file really, and migrate jobs seem better in that the
original gets purged on successful migrate and thus releases the disk
space. Also avoids the issue of some disk volumes expiring before the
copy happens if it's delayed for any reason.
3. Increase the LVM volume holding my database, as my planned changes
to the File and Job volumes are going to greatly increase it.
I'm most concerned with the upgrade part, as I'm not sure what the
best rollback would be it doesn't work for me. I'm sure it's well
documented, so I'll check the forums for warnings and advice.
If anyone has a good link to upgrading from 3.0.2 to 5.0.3 I'd be grateful.
Thank you everyone.
Regards,
Dermot.
On 14 November 2010 01:47, Kleber Leal <kleber.leal AT gmail DOT com> wrote:
>
>
> 2010/11/12 Dermot Beirne <dermot.beirne AT dpd DOT ie>
>>
>> Hello,
>> I got no response to my last mail below, so I'll try and summarise my
>> questions a little:
>> (Bacula 3.0.2 on Ubuntu 9.04)
>>
>> 1. If I copy a backup job, is the retention period of the original
>> backup job or the copy backup job used by bacula. I ask because my
>> originals (disk) have a 2 day retention, but their copies (tape) have
>> up to 12 months, but are still not kept in the catalogue and I need to
>> use bscan to restore them. What do I need to check/change?
>
> You can use job and file retention period bigger than volume retention
> period.
> This should keep jobs and files information into catalog and the director
> will search for files to restore in both file and tapes volumes.
>
>>
>> 2. Is there a way Bacula can recycle tape volumes back into Scratch
>> before the pool is empty. Without purging volumes, the Scratch pool
>> runs out.
>>
>> 3. I currently have Daily, Weekly and Monthly disk volumes, which are
>> copied to tape within a few hours of backup, and have a retention of
>> just 2 days to keep space from running out on disk. The Daily disk
>> backup now needs more space, and can't recycle the weekly or monthly
>> disk volumes as they are in different pools in different directories.
>> Hence over 2 TB of disk is inaccessible to the Daily jobs, even though
>> those volumes have expired. Is it possible to have all 3 backup types
>> share and recycle the same disk volumes, in one directory with the
>> same naming scheme, but for the copy job to still know which is a
>> daily, weekly, monthly disk volume so it can apply the required
>> retention to the tape volume? This would allow much better use of
>> disk space.
>
> I think you define a scratch pool you can do this, but I used LVM to group
> all disks into a unique volume and actived automatic creation of volume
> files and limit the number of volumes. I have three 1.5TB disks groups to
> make a volume of 4TB.
>
>>
>> 4. The writing from disk to tape appears very slow. Is there a
>> document/thread somewhere that will help me troubleshoot or benchmark
>> the autoloader throughput (Dell P124T)
>>
>> Thank you for any info.
>> Dermot.
>>
>> On 9 November 2010 10:07, Dermot Beirne <dermot.beirne AT dpd DOT ie> wrote:
>> > Thanks for replying.
>> >
>> > On my first point below, where you suggest the volume retention is too
>> > low, the volume retention for the monthly tape backups to 12 months,
>> > yet if I attempt to restore a file from a jobid of, say, last March
>> > monthly backup, it is no longer in the catalogue, and I need to bscan
>> > that monthly tape volume to repopulate the catalogue, so I can restore
>> > a file. Is the fact that these are "copy" jobs overriding the
>> > retention on the tape pools with the shorter disk pools retentions, or
>> > can you expand on your suggestion that the retention is the problem.
>> >
>> > On the second point, I understand that purging is the last resort, but
>> > could not find an alternative. We have a tape autoloader with 8
>> > slots. Every day we remove the tapes used that day and replace with
>> > Scratch tapes. The problem is that Bacula never moves tapes back into
>> > Scratch until it has none left. If it is writing the backups to tape,
>> > and uses the last Scratch tape, it will then recycle an old tape
>> > volume (presumably) and ask for manual intervention for someone to
>> > insert this tape, hence the backups stop. Is there a way it can
>> > recycle the volumes back into Scratch before the pool is empty, so the
>> > admin has enough to refill the autoloader each day?
>> >
>> > On the third point, I do use limits on the disk volumes, which is
>> > forcing the recycling as you say, and this works well to stop bacula
>> > causing the backup server to run out of space before it tries to
>> > recycle, however, how do I do this for Tape volumes? If I set such a
>> > limit on the tape pools, its hard to predict which ones it will
>> > recycle, and hence it will stop and wait for manual intervention to
>> > load the tape it has just decided to recycle. I think it needs to do
>> > recycling earlier in the process, so that the Scratch pool is self
>> > replenishing.
>> >
>> > Appreciate your advice.
>> > Dermot.
>> >
>> >
>> > On 6 November 2010 23:30, Dan Langille <dan AT langille DOT org> wrote:
>> >> On 11/5/2010 12:00 PM, Dermot Beirne wrote:
>> >>>
>> >>> Hello,
>> >>> I have been using bacula now for 1 year this month, and it's been a
>> >>> fairly smooth ride, but I have a number of questions that i'd be
>> >>> grateful for some direction on:
>> >>>
>> >>> Bacula version 3.0.2 on Ubuntu 9.04
>> >>>
>> >>> 1. I am concerned about my retention policy. I have 4 disk pools and
>> >>> 4 corresponding tape pools.
>> >>>
>> >>> All the disk pools (Daily, Weekly, Monthly, Yearly) have a volume
>> >>> retention of 2 days and Volume Use Duration also 2 days
>> >>>
>> >>> The Tape Pools are as follows:
>> >>> Daily - 12 days
>> >>> Weekly - 32 days
>> >>> Monthly - 12 months
>> >>> Yearly - never recycle
>> >>>
>> >>> I am finding that when needing to restore from an old job, I have to
>> >>> run bscan on the tape as Bacula says it has no records for that job in
>> >>> it's catalog.
>> >>> This is even for monthly backups. Why can this be?
>> >>
>> >> Your Job Retention is too low.
>> >>
>> >>> 2. Use of scratch pools.
>> >>> Every few weeks we have to print a list of the volumes (from the daily
>> >>> and sometimes weekly tape pools), and manually purge (purge jobs
>> >>> volume) anything older than the above retention times in order to
>> >>> force them back into the Scratch pool, as the Scratch pool runs out of
>> >>> volumes because Bacula is not automatically purging once the retention
>> >>> policy expires.
>> >>> Is there a fix for this? Could this be causing/affecting point 1
>> >>> above (i.e. doing manual purges)?
>> >>
>> >> Purging is the last resort. If the scratch pool contains Volumes, they
>> >> will
>> >> be used.
>> >>
>> >>> 3. Disk pool structure.
>> >>> I have 4 directories on my backup server for the Disk pools, Daily,
>> >>> Weekly, Monthly and Yearly. The Daily, Weekly and Monthly directories
>> >>> are all in the region of 1.4Tb in size. The problem is that the
>> >>> server is near it's capacity of disk space, and I can't add any more
>> >>> clients until I address this. I feel the current structure is not
>> >>> making the best use of the space. The retention policy of 2 days on
>> >>> the disk volumes is to allow a tape copy job to run the morning after
>> >>> a nightly backup, and these tapes are then moved to a safe and
>> >>> replaced with Scratch tapes in the autoloader. It also allows us to
>> >>> do a quick restore of a file that may have been deleted the previous
>> >>> day, which is the most common restore request.
>> >>
>> >> It sound like you need to set a limit on the Pool so Bacula starts to
>> >> recycle Volumes. Bacula will not recycle Volumes if it is permitted to
>> >> create new Volumes.
>> >>
>> >>>
>> >>> The space taken up by the weekly and monthly volumes is wasted, as
>> >>> they have expired 2 days after use, but the space is still not
>> >>> available to the Daily volumes. I suspect I'd get a lot more clients
>> >>> covered if I could change this.
>> >>>
>> >>> Is it possible to have a single directory shared by all pools, and all
>> >>> pools share and recycle the same volumes. What are the pros and cons
>> >>> of doing this, or is there a better way.
>> >>>
>> >>> Is there any benefit in upgrading bacula to address any of these
>> >>> items.
>> >>>
>> >>> I'm sorry for the long mail, and appreciate anyone taking the time to
>> >>> read
>> >>> it.
>> >>> Advice welcome on how to proceed with Bacula from here.
>> >>
>> >>
>> >> --
>> >> Dan Langille - http://langille.org/
>> >>
>> >>
>> >
>>
>>
>> ------------------------------------------------------------------------------
>> Centralized Desktop Delivery: Dell and VMware Reference Architecture
>> Simplifying enterprise desktop deployment and management using
>> Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
>> client virtualization framework. Read more!
>> http://p.sf.net/sfu/dell-eql-dev2dev
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users AT lists.sourceforge DOT net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
------------------------------------------------------------------------------
Centralized Desktop Delivery: Dell and VMware Reference Architecture
Simplifying enterprise desktop deployment and management using
Dell EqualLogic storage and VMware View: A highly scalable, end-to-end
client virtualization framework. Read more!
http://p.sf.net/sfu/dell-eql-dev2dev
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
