BackupPC-users

Re: [BackupPC-users] Using rsync for blockdevice-level synchronisation of BackupPC pools

2009-09-11 02:29:24
Subject: Re: [BackupPC-users] Using rsync for blockdevice-level synchronisation of BackupPC pools
From: Timothy J Massey <tmassey AT obscorp DOT com>
To: "General list for user discussion, questions and support" <backuppc-users AT lists.sourceforge DOT net>
Date: Fri, 11 Sep 2009 02:33:33 -0400
Les Mikesell <lesmikesell AT gmail DOT com> wrote on 09/10/2009 10:38:56 PM:

> Maybe I'm not making this clear.  I want the physical backuppc host 
> to be able 
> to mirror its current 750 GB backuppc partition onto what appears at
> the time to 
> be a virtual partition, but which results in the updating of the files 
that 
> comprise a vmdk disk. Backuppc isn't running on a VM.  After this step 
> completes, I want to rsync those files to copies elsewhere.  At the 
other 
> location, I might want to access the vmdk from a VM if it is necessary 
to 
> restore something.

So you're attempting to convert a physical BackupPC server into a virtual 
image?  VMware has conversion tools that do this.  I've only used the 
Windows version of VMware Converter, but it has worked perfectly for 
converting a physical host into a virtual host.  There is a Linux version.

I would look into this, rather than trying to do a poor-man's version of 
it.  It's a very simple process, and after a somewhat long wait (but 
shorter than doing a RAID1 rebuild) you will have a brand new virtual 
clone of your physical box!

And if all you're doing is to try to capture a file-based version of your 
block device (a physical partition) that you want to mount using some 
other physical server (or even a virtual server, come to think of it), I 
think you'd be *far* better off just dd'ing the partition into a file and 
using a loopback mount to mount it someplace else.

In other words, the only time you should be dealing with VMDK files is if 
you're trying to create a new virtual guest.  And if you are doing this, 
the proper way of doing this is *not* by trying to use LVM/RAID weirdness, 
but using the VMware Converter tools to do this for you properly.

If you're *not* trying to create a new virtual guest, then don't mess with 
VMDK files.  They're an annoyance that should only be dealt with if you 
actually have to.

> > I think you need 
> > to really think long and hard about the hoops you're trying to jump 
> > through.  The logic that you'd use for a physical machine is obselete 
when 
> > you're dealing with VMware.  Just copy the native files as native 
files to 
> > another machine with VMware server, and start the host.  That simple.
> 
> I need to get the image of my working system on there in the first 
place.

OK, then what I explained above is probably correct.  The way you do this 
is by using the VMware Converter tools, not by trying to do it yourself! 
:)

> > Again, what is your logic here?  You don't move the contents of a 
block 
> > device from one guest to another.  YOU MOVE THE ENTIRE GUEST.  Period. 

> > This is so, very, very way better than what you're trying to do.
> 
> The point is to get the data which is on a physical host partition 
> into a vmdk 
> that can be copied as a set of smaller files - and used that way 
> directly if needed.

Yup.  Use the conversion tools.

> > But you can't get VMware for OpenSolaris!  :)  I hate to tell you, ZFS 

> > buys you *NOTHING* in this situation.  You'll be forced to use VMware 
> > snapshotting anyway, in which case you no longer *need* filesystem 
> > snapshotting.
> 
> Virtualbox seems to be a reasonable match for VMware these days.  Itcan 
even 
> use vmdk format disks directly.  With zfs I'd be able to use the 
incremental 
> send/receive function which would likely be even better than 
> rsync'ing the files 
> sitting on top of it.

VirtualBox compares fairly with the free VMware Server, but VMware server 
is about 10% of what you can do with VMware--with the paid-for tools.

When it comes to commercial tools, VMware is in a class by itself, though 
Citrix is trying hard with XenServer (still too cumbersome and unpolished 
compared to VMware, and requires VM hardware for Windows).  When it comes 
to free-as-in-beer, XenServer is the best.  It's still cumbersome, but 
they give you several of the items for free that VMware charges for.

VirtualBox is neither the best tool overall, nor the best tool for free. 
And unfortunately, the GPL'ed code is only a fraction of what you really 
need for a usable virtualization environment.  If you want GPL tools, KVM 
(especially in RHEL 5.4) is the best around.

The only advantage that VirtualBox has is that it runs on OpenSolaris (or 
OS/2...).  For me, that's a non-feature.  Obviously, YM *does* V...  :)


Interesting thoughts.  I've never been a fan of running BackupPC inside of 
a virtualized guest.  Basically, my philosophy is to put as little between 
my backups and the hardware as possible.  I don't even use compression on 
my backups!  The idea of putting my backups inside of a virtual disk on 
top of yet another filesystem is not overly appealing.  Now I've got two 
ways for EXT3 to screw me!  :)

But the ability to rsync collections of VMDK files to a remote host *is* 
appealing.  Interesting...

However, you could achieve the same thing in other ways without having to 
run BackupPC in a virtualized guest.  You could simply use an rsync-like 
process for copying the block device to a remote (physical) host.  Years 
ago someone posted a script that read a block device 64k (IIRC) at a time 
and did an MD5SUM on it and compared it with a remote block device (via 
netcat, again IIRC).  If the blocks matched, they were skipped.  If they 
didn't, the block was sent over.  You could do something like that to a 
physical block device to achieve largely the same thing.

Or, if you wanted to use actual rsync and you wanted to avoid block 
devices, you could do the same thing by using a (very large!) loopback 
file for your BackupPC pool partition on a physical server and a physical 
partition and rsync the file (after you stop BackupPC and unmount the 
partition).  In fact, you could create a partition for your pool and 
create a single file that fills the entire device, and use that file as a 
loopback partition.  In that case, you could very nicely use LVM tools to 
snapshot the outer partition and you could even restart BackupPC while the 
remote sync was taking place!

How much performance do you lose using a loopback mount?  It's *gotta* be 
less than the overhead of virtualization!  I like that idea even better. 
But all it buys you is being able to use rsync directly on a file instead 
of coming up with a way to copy a block device in an rsync-like manner... 
And, to me, that's the best way of all.

Of course, now we've come full circle:  how do you copy a physical block 
device in an rsync-like manner?  :)

Tim Massey


------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day 
trial. Simplify your report design, integration and deployment - and focus on 
what you do best, core application coding. Discover what's new with 
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
BackupPC-users mailing list
BackupPC-users AT lists.sourceforge DOT net
List:    https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:    http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

<Prev in Thread] Current Thread [Next in Thread>