|
On Tuesday 09 September 2008 02:31:24 Michael Heim wrote:
> Kern Sibbald wrote:
> > On Sunday 07 September 2008 03:04:21 Michael Heim wrote:
> >> Hi Kern,
> >>
> >> can't you use something like data spooling for this?
> >>
> >> First step: Make a virtFull to a temp spool file on disk
> >> Second step: Spool this to the destination
> >
> > Yes, interesting idea. It could possibly help in some cases, but the
> > problem is that it doesn't scale well enough. Take for example an
> > extreme example: someone has a backup to tape amounting to 10TB. With
> > this, suggestion, it would be necessary to spool 10TB to disk then to
> > write it to tape. Someone with this size of data will require a solution
> > that is tape to tape ...
Hello Michael,
Yes, you are dealing with some pretty big iron. The last time I worked on mainframe type machines was in 1980, and in those days, big iron wasn't what it is today.
My example of the 10TB backup wasn't really meant to be something realistic but just to point out that once you reach a certain size, a step like spooling to disk becomes very expensive in terms of needing "idle" disk space and in terms of efficiency since it is far faster to go LTO3(4) directly to LT03(4) rather than passing via much slower disks.
Bacula can write out images too, but I am not convinced it is a very good way to go ...
Best regards,
Kern
>
> Hi Kern,
>
> I think a 10 TB backup should be no problem to spool today for a company
> with such a request. But first, such a big filesystem should split into
> several smaller parts to be more flexible for a restore. In the
> datacenter, where I worked for over 7 years, we have had a TSM cluster
> with two SUN L500 tapelibs, each equipped with 8 LTO3 FC drives and
> about 500 slots. For holding backup data first (spool + migration), the
> cluster was connected to two EMC Clariion CX3-40 with 32 TB SATA
> capacity each (64 TB total). I think in environments with 10 TB
> filesystems a 10-20 TB spool area cannot be a problem.
>
> But filesystems with several TBs are very rare. In my opinion only
> video/multimedia application can utilize them. For normal file data the
> average filesize is between 50kb and 300kb. This means about 3-20
> million files per TB.
> A normal filesystem with more then about 500-1000 GB data shouldn't be
> used, because the time to restore is too high. A real life example (from
> my experience):
> A restore of a NTFS filesystem (on a HP DL385G2 Windows 2003
> cluster, 2x2,4 GHz Opteron-DC, 8 GB RAM) with 400 GB and 10 million
> files, would take about 3 days on a EMC Clariion CX700 with 4 GB cache
> (measured on a 400 GB slice of a Raid5 with 8x300 GB FC 10.000 UPM
> harddisks - no other IO is using the same Raid) with TSM.
>
> For such big filesystems (>1 TB or several million files) a file by file
> restore isn't really possible for a company with a SLA or a max. restore
> time. To handle those filesystems, imagebackups (a TSM feature to backup
> a whole filesystem with a own snapshot method) are used, so the restore
> of the same filesystem (400 GB, 10 million files) will only take 2h,
> because the whole filesystem image is restored.
>
> A good strategy is to create only filesystems below one TB and with less
> then 3-5 million files, because bigger filesystems couldn't be handled
> properly with normal hardware. In a normal IT environment with several
> smaller filesystems a spool area shouldn't be any problem. The only
> requirement to such a spool area should be very fast concurrent reads
> and writes, so a Raid 10 (or perhaps SSDs) should be considered.
>
> > I think the best suggestion so far has been to add new code in Bacula
> > that uses the volume list (list of volumes to be read) to ensure that
> > none of those volumes are selected for writting. That will provide a
> > quite reasonable means of ensuring there are no deadlocks -- providing
> > the user has two drives available.
> >
> > Best regards,
> >
> > Kern
> >
> >> regards
> >> Michael
> >>
> >> Kern Sibbald wrote:
> >>> Hello,
> >>>
> >>> As many of you know Virtual Backup (consolidation, synthetic full, ...)
> >>> is a new feature that is implemented in the development trunk and
> >>> scheduled to be released later this year. It essentially copies what
> >>> would be a "full current" restore to a new Volume thus creating an
> >>> virtual backup that can serve as a Full backup. This has a lot of
> >>> advantages, particularly for sites with full backups that run long
> >>> times or for remote sites where the time to transmit a full backup is
> >>> excessive.
> >>>
> >>> The Virtual Backup feature works much like Migration and Copy. It
> >>> reads from the required Volumes and writes to a Volume specified in the
> >>> pool as "Next Pool". This ensures that the read and write Volumes are
> >>> different.
> >>>
> >>> Everything seems to work fine with the Virtual Backup. However,
> >>> thinking about longer term operations, it has occurred to me that when
> >>> you want to make a second Virtual Backup things will become very
> >>> complicated. First, the Virtual backup will want to read the previous
> >>> Virtual backup volume, and then if that Volume is not full, it will
> >>> want to write to the same Volume. Even if the volume is full, you will
> >>> be in a situation where the Job will want to read and write to volumes
> >>> in the same pool. In all those cases, there is no guarantee that there
> >>> will not be a deadlock situation (actually Bacula currently cancels any
> >>> job attempting to read and write from the same Storage device).
> >>>
> >>> I am not 100% sure what to do to resolve this issue. I suppose one
> >>> could run a Migration job to "move" the Virtual Backup back to the Pool
> >>> from which it originally came, then the next Virtual Backup would work
> >>> fine (the same as the first one), but that sounds a bit kludgie.
> >>>
> >>> If anyone has any suggestions, I would appreciate to hear them.
> >>> However, suggestions that require implementing significant amounts of
> >>> code or complex new algorithms such as deadlock detection won't be very
> >>> helpful since there is no time left to do such implementations between
> >>> now and release time. In addition, deadlock detection won't help, what
> >>> we really need is deadlock resolution, and that is an even more
> >>> difficult subject.
> >>>
> >>> Best regards,
> >>>
> >>> Kern
> >>>
> >>>
> >>> -----------------------------------------------------------------------
> >>>-- This SF.Net email is sponsored by the Moblin Your Move Developer's
> >>> challenge Build the coolest Linux based applications with Moblin SDK&
> >>> win great prizes Grand prize is a trip for two to an Open Source event
> >>> anywhere in the world
> >>> http://moblin-contest.org/redirect.php?banner_id=100&url="">
> >>> _______________________________________________
> >>> Bacula-users mailing list
> >>> Bacula-users AT lists.sourceforge DOT net
> >>> https://lists.sourceforge.net/lists/listinfo/bacula-users
> >>
> >> ------------------------------------------------------------------------
> >>- This SF.Net email is sponsored by the Moblin Your Move Developer's
> >> challenge Build the coolest Linux based applications with Moblin SDK&
> >> win great prizes Grand prize is a trip for two to an Open Source event
> >> anywhere in the world
> >> http://moblin-contest.org/redirect.php?banner_id=100&url="">
> >> _______________________________________________
> >> Bacula-devel mailing list
> >> Bacula-devel AT lists.sourceforge DOT net
> >> https://lists.sourceforge.net/lists/listinfo/bacula-devel
| 