Networker

Re: [Networker] Backup speed drop with 7.3.3 (nsmmd problem?)

2007-09-02 13:41:26
Subject: Re: [Networker] Backup speed drop with 7.3.3 (nsmmd problem?)
From: "Brian O'Neill" <oneill AT OINC DOT NET>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Sun, 2 Sep 2007 13:38:01 -0400
Macina, Conrad wrote:
> Another possibility: What do you know about the file systems being
> backed up when the speed drops? NetWorker (and most other backup
> software) has problems with "millions of tiny files" save sets because
> each file must be recorded in the index database, resulting in
> considerable overhead on the host system. If that's the case, I'm afraid
> there's no easy solution, other than block-level backups (SnapImage).

No, nothing unusual about these filesystems - typical file distribution for the most part, and the Oracle dumps are very large files, so filesystem meta overhead didn't really apply here.

Yaron Zabary wrote:

My bet is that the file system of the adv_file (ext2/ext3) is having a problem with allocating new blocks to the saveset. What do you see with vmstat ? Try


I think this got truncated, but I didn't see anything particularly unusual. Filesystem is ext3, and was only about 11% full.

I have about 10 Netapp filesystems I will be backing up via NFS, which it is intended will happen on the Networker server. Should I be backing them up via other hosts instead?

No. you should invest some money in NDMP licenses and backup over the network. It works much better than NFS. You should have some processing power on the server for running this, but other than that, it is a better solution.

The client decided not to pursue using NDMP due to the added cost, and was not concerned with the performance issue of NFS - but it was not expected to run this poorly. And for reference I'm backing up the nightly.0 snapshots.

FYI, as I mentioned in my last e-mail, I backed out to 7.3.2 Build 11. The Oracle FS had no problem, but when another group kicked it, the nsrmmd kit 99% CPU again. Eventually I decided to kill the nsrmmd and let Networker restart it. It did so, but took a little while before it would use the adv_file volume again. Once it did, it kicked back to full speed again. But after a while, it slowed back into the sub-1KB/s range again, although nsrmmd was not using CPU. When the Oracle dump backups kicked in again, it went full speed and completed that pretty quickly, then slowed down again, with only two NFS filesystems being backed up.

At times it would get a speed burst, but it wouldn't last long. It did finally finish though.

-Brian

To sign off this list, send email to listserv AT listserv.temple DOT edu and type 
"signoff networker" in the body of the email. Please write to networker-request 
AT listserv.temple DOT edu if you have any problems with this list. You can access the 
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER