Networker

Re: [Networker] Slow Win2k backup with lots of files

2006-06-06 21:16:10
Subject: Re: [Networker] Slow Win2k backup with lots of files
From: Siobhán Ellis <siobhanellis AT HOTMAIL DOT COM>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Wed, 7 Jun 2006 11:13:54 +1000
Some good points Peter

They key issue here is that you have millions of files on one
filesystem...   You will find that the client kernel is consuming a lot
of cpu and memory in its efforts to walk down the filesystem tree.


We found that by running parallel streams we could get around this. I'm not sure of the memory/CPU config of the system.

So firstly, are you comfortable with the amount of CPU and memory
available?   If you can beef up the memory you might  see some
improvement, also having more than one cpu core should help.

I have also been told there are a few kernel adjustments you can make to
increase the heap size available, and some other buffer settings to
optimise filesystem priority... Of course it all depends on what
application needs to run on the system also.... Unfortunately I leave
that up to my windowsey colleagues and I don't know the settings needed.

Other things to consider:

Win2k3  - I believe has some filesystem performance improvements over
win2k.

Change Journal -   using the change journal feature will help with
incremental and differential backups because it obviates the need to
walk the directory trees. Otherwise I find that incrementals of complex
filesystems like this take almost as long as a full...

This comes with its own inherent issues. rename a directory, all the files contained within are now considered to have changed, so it backs them all up. Compress a directory - same problem. IMHO Windows File Journaling is a typical microsoft solution.... almost there!

If you have the resources on your backup server to do consolidated
backups (3 drives are needed) and you use the change journal your full
backup will be a level 1 followed by tape cloning on the server.

There are many issues with Consolidated backup. Read the documentation before you use it. It, particularly, has issues with renaming or deleting directories. It can't deal with it.

Backup to adv_file...    The problem with a slow client is it's unable
to stream data fast enough to keep your tape device spinnning, so you
get a double penalty because you spend a lot of time waiting for the
tape drive to retrain - backing up to disk can help.

Another filesystem type.....   No idea what would happen here, but
veritas do have a version of VxFS for windows... Maybe someone out there
has some experience.

NAS:    put your data on a NAS, and use NDMP to back it up.   This makes
the filesystem traversal the problem of the vendor, some vendors are
better than others in this regard.

But it doesn't necessarily solve the problem. Vendors often sell NAS as a way to solve a backup problem. My experience has been that it usually doesn't. If you have lots of data you now need to connect it to a tape device directly and so you are back to the streaming issue.

Celerra, by the way, is multi-threaded. It searches for files whilst a file is being backed up. I have no knowledge of Netapp in this regard, they may do it as well.

Siobhan

To sign off this list, send email to listserv AT listserv.temple DOT edu and type 
"signoff networker" in the
body of the email. Please write to networker-request AT listserv.temple DOT edu 
if you have any problems
wit this list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER