Just a thought... If I understand you correctly, the files never change once 
they are created? In that case, your best bet might be to use a copy-based 
scheme for backup.

I'd create two directories on your main server.

/current
/to_backup

Both on the same file system.

/current (or whatever you want to call it) holds all the files.
/to_backup holds hard links to only those files in current that haven't been 
backed up yet. You can use a script to identify those current files, for 
instance by time stamp, a checksum or the like.

The actual backup can be done several different ways. You could use bacula. Or 
you could simply use cp, scp, rsync or similar. After the backup has completed, 
you delete the links in /to_backup.

That way, you only back up files that have changed, and still have a full set 
of files on the backup.

> -----Original Message-----
> From: Ralf Gross [mailto:Ralf-Lists AT ralfgross DOT de]
> Sent: Friday, November 27, 2009 4:24 AM
> To: bacula-users AT lists.sourceforge DOT net
> Cc: bacula-devel AT lists.sourceforge DOT net
> Subject: [Bacula-users] RFC: backing up hundreds of TB
> 
> [crosspost to -users and -devel list]
> 
> Hi,
> 
> we are happily using bacula since a few years and already backing up
> some dozens of TB (large video files) to tape.
> 
> In the next 2-3 years the amount of data will be growing to 300+ TB.
> We are looking for some very pricy solutions for the primary storage
> at the moment (NetApp etc). But we (I) are also looking if it is
> possible to go on with the way we store the data right now. Which is
> just some large raid arrays and backup2tape.
> 
> I've no problem with the primary storage and 10 very big raid boxes
> (high availability is not needed).  What frightens me is to backup all
> the data. Most files will be written once and maybe never been
> accessed again. But the data need to be online and there is a
> requirement for backup and the ability to restore deleted files
> (retention time can be different, going from 6 months to a couple of
> years).
> 
> The snapshot feature of some commerical products is a very nice
> feature for taking backups and it's a huge benefit that only the
> deltas are stored.
> 
> Backup2tape means that with the classic Full/Diff/Incr setup we'll
> need many tapes. Even if the data on the primary storage won't change.
> 
> Backup2disk has the disadvantage that a corrupt filesystem (been
> there, seen that...) can ruin TB's of backed up data. And we will need
> file storage that is much bigger than the primary storage (keeping x
> versions of a file....).
> 
> Anyone else here with the same problem? Anyone (maybe Kern or Eric)
> here that can tell if one of the upcoming new bacula features (dedup?)
> could help to solve the problem with the massive amount of tapes
> needed and the growing time windows and bandwidth for backups?
> 
> Any thought?
> 
> Thanks, Ralf
> 
> -----------------------------------------------------------------------
> -------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008
> 30-Day
> trial. Simplify your report design, integration and deployment - and
> focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users