Networker

Re: [Networker] Fulls versus incrementals and inodes?

2007-10-24 18:14:33
Subject: Re: [Networker] Fulls versus incrementals and inodes?
From: A Darren Dunham <ddunham AT TAOS DOT COM>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Wed, 24 Oct 2007 22:11:24 +0000
On Wed, Oct 24, 2007 at 05:56:44PM -0400, George Sinclair wrote:
> If a given save set has a large number of files, say many tiny files, 
> does this affect the time it will take a full to start running?

Not to my knowledge unless it's doing an esitmate first (-E I think).
You can invoke 'save' by hand (with several -v flags) and see what it
does.

> I guess where I'm going with this is that I'm trying to understand how 
> Legato is affected by a large number of inodes and in what way.

The client process (save) does a directory walk as it grabs the data, so
there shouldn't be any big up-front costs.  Even on a huge tree, it
doesn't take long to get to the first file and start sending data.

> For example, I have this directory that takes hours to run a 'du'
> command on because it has a huge number of files, but it's only about
> 700 GB, but I might have another path that's over 1 TB, and it only
> takes 5 minutes to run a 'du'.

It takes it that long to *finish* I presume.  Both of them should start
running immediately, just like a full backup would.

> Just curious what an incremental does before it actually starts
> backing up any data versus what a full would do.

Both the incr and full do a treewalk, but the incr does a save/nosave
decision based on timestamps as it hits each file.  So a large tree with
few updates may take a long time before any "data" is sent to the
server.

> Obviously, both will update the client's index while the data is being
> backed up, so if the save set has a lot of files then the full backup
> will force the index to grow even larger, but it seems that while the
> full will undoubtedly cause the index to grow larger than the
> incremental, I would think the incremental would take longer to figure
> out what, if anything, needs to be backed up since it has to read the
> index from the last backup and traverse the inode information on the
> data path to determine what to actually back up, right?

Um, sort of.  It doesn't look at the index of the backup, just the
date.  

> What about a full?

Similar, but there's no date comparison as it encounters files (just
directive/NSR file comparisons).

-- 
Darren Dunham                                           ddunham AT taos DOT com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the body of the email. Please write to 
networker-request AT listserv.temple DOT edu if you have any problems with this 
list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER