BackupPC-users

Re: [BackupPC-users] experiences with very large pools?

2010-02-19 11:23:58
Subject: Re: [BackupPC-users] experiences with very large pools?
From: Ralf Gross <Ralf-Lists AT ralfgross DOT de>
To: backuppc-users AT lists.sourceforge DOT net
Date: Fri, 19 Feb 2010 17:22:02 +0100
Les Mikesell schrieb:
> On 2/19/2010 9:42 AM, Ralf Gross wrote:
> > Les Mikesell schrieb:
> >> Ralf Gross wrote:
> >>>
> >>> I think I've to look for a different solution, I just can't imagine a
> >>> pool with>  10 TB.
> >>
> >> Backuppc's usual scaling issues are with the number of files/links more 
> >> than
> >> total size, so the problems may be different when you work with huge 
> >> files.  I
> >> thought someone had posted here about using nfs with a common archive and
> >> several servers running the backups but I've forgotten the details about 
> >> how he
> >> avoided conflicts and managed it.  Maybe this would be the place to look at
> >> opensolaris with zfs's new block-level de-dup and a simpler rsync copy.
> >
> > ZFS sounds nice, but we have no experience with opensolaris or ZFS.
> 
> That's something that could be fixed.

sure, but it's something I can't estimate right now.

 
> > And I heard in the past that not all of ZFS's features are ready for
> > production.
> 
> In the past, nothing worked on any OS.

that a bit hard...
 
> > bit off topic:
> > Right now I'm looking for a cheap storage solution that is based
> > on supermicro chassis with 36 drive bays (server) or 45 drive bays
> > (expansion unit) in 4 HU. Frightening, that would be 810 TB in one
> > Rack (36 + 45 HDDs x 5 x 2 TB, 40 HU) with 5 servers. Only problem is
> > power, cooling and.... backup.
> 
> What's generating that kind of data?  Can you make whatever it is write 
> copies to 2 different places so you don't have to deal with finding the 
> differences in something that size for incrementals?  Or perhaps store 
> it in time-slice volumes so you know where the changes you need to back 
> up each day will be?

The data is mainly uncompressed raw video data (AFAIK HDF, I don't
work with the data). Users come with external HDD's and copy the data
on the samba files servers. 

Right now we backup 70 TB to tape, but I would like to get rid of
tapes. For the large RAID volumes there is also no regular backup. We
make it by acclamation. But this should change now...



Ralf

------------------------------------------------------------------------------
Download Intel&#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/