Networker

Re: [Networker] Large quantity of small text files

2004-05-17 01:25:28
Subject: Re: [Networker] Large quantity of small text files
From: Darren Dunham <ddunham AT TAOS DOT COM>
To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
Date: Sun, 16 May 2004 22:25:18 -0700
>
> Howdy,
>
> We are in the midst of a minor disaster-recovery and are finding out
> that it is taking a very long time to recover some directories that have
> a large quantity of small text files (small = ~700 bytes, large quantity
> = 40,000+).
>
> Networker version is 6.1.3, server is HP-UX 11i, client is Solaris 8,
> UFS filesystems.  We use client-side compression (to reduce network
> traffic).  Tape technology is AIT-3.

Do you have the latest patches and updates applied to Solaris on the
client?  The DNLC may be able to help with this.

> It took about 20 minutes to restore one directory that contained about
> 40,000 of these small text files, which totaled about 60 Mbyte - it
> normally takes about 3 minutes to backup the directory, if that.
> Normally we can recover data in the range of 4Mbytes - 8Mbytes per
> second.

This doesn't sound completely out of the ordinary.  I'd say it has
nothing to do with Networker and more to do with the overload of the
directory.  You potentially have to do a linear search of every file
already in the directory when you add a new file.  This becomes a n^2
operation.

I'd suggest just using 'tar' or something to create a local copy and try
a restore of it.  Don't get Networker involved at all and see how long
it takes.  I'll bet it's similar.

> Is this the generally accepted behavior of Networker, or should I be
> seeing faster recover times of a large quantity of small text files,
> when going the other way - backups - take much less time?

The name lookup cache may work better to cache directory contents during
the backup than the restore.


--
Darren Dunham                                           ddunham AT taos DOT com
Senior Technical Consultant         TAOS            http://www.taos.com/
Got some Dr Pepper?                           San Francisco, CA bay area
         < This line left intentionally blank to confuse you. >

--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=

<Prev in Thread] Current Thread [Next in Thread>