Networker

Re: [Networker] Performance issues with large file systems

2006-03-01 05:37:52
Subject: Re: [Networker] Performance issues with large file systems
From: Allan Nelson <an AT CEH.AC DOT UK>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Date: Wed, 1 Mar 2006 10:32:15 +0000
Hi
We've come across a similar problem to this when we upgraded a box from
Win2000 to Win2003.

If you search the archive you'll see the thread (Windows 2003 - poor
performance), but here's what fixed the issue for us.  Could be worth
checking any offload settings on your NIC.

Hope this helps... Allan.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hi
I thought some of you might be interested to hear what resolved this
issue.

Basically, the issue was that we were getting 3-4mBps backup rate on a
Windows 2003 client which had been backing up fine when on Windows2000
(same hardware).

The card in the Windows 2003 client is an Intel Pro 1000 MT dual port
adapter.  
If we set the card to 100mbps, we got a throughput on backups of around
10-12 mBps.

By unchecking the various 'Offload' parameters in the Advanced settings
on the adapter, and re-setting to gigabit, we now have throughput of 30+
mBps.  
As it's a fairly beefy box we're hoping it wont have an adverse effect
on day to day use (by putting the load back on the client CPU's as
oppsed to the card).

We're a bit concerned that we've just 'masked' the real issue (ie why
does this card behave so poorly under Windows2003 when it worked fine
under Windows2000), but at least we can get our backups done in a
reasonable time now!  (We did try an older driver, but that made no
difference).

As a test, I also made this change on a much less powerful Windows2000
server (HP) with an HP dual port card.  Performance did increase
slightly on this box too.

Anyone out there doing a similar thing with their gigabit cards?

Cheeers... Allan.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


>>> xy.0815 AT GMX DOT NET 01/03/06 08:27 >>>
Hi NetWorkers,

in our installation we run both online and offline
backups of our Notes (V6.5.2) servers.
While online backups work very well (most of the time)
(main reason for doing this is: parallelism)
we are facing some performance issues with the offline
< plain file system >  backup of these servers.

All notes data resides in one - quite large - filesystem,
directory structure is almost flat.

For ~450GB Notes file system it takes more than 15 hrs to finish the 
offline backup, which is much more it should.

Environment:
all data on CX700, FC-2Gbps attached to client and server,
 details: Meta-Lun of 2x RAID-5 Luns = 2x [4+1]
LTO-2 FC-2Gbps attached to server
1 Gbit Ethernet
no CPU shortages on client and server
client compression is OFF
everything is on Win2k+3

Fore even more details see savegroup completion below.

To speed up the backup we decided to parallelize the file system
backup
manually. 'save -M' would be the best choice, but does not realy
support restores (BTW, who cares about restores :-)
So we configured 4 groups with 4 different directives,
each directive skips some portion of the filesystem,
ie:
<< E: >>
 +null: [a-k]*.nsf [r-z]*.nsf  <-- backup l*.nsf ... t*.nsf
 +null: ... more file types here

This works well but has not improved backup speed significantly.
So I'm a bit clueless now:
- servers have enough resources
- all pipes (LAN, SAN) are fat enough
- drives (disk, tape) should have enough power
but still this is very slow :-(
(BTW, no problem of doing Oracle backups >50MB/sec in the same
environment)

Just before I start to rack "my" brain,
I'd like to ask for some hints where to start the investigation first.
How are you doing backups of large filesystems?
(sure we can do LANfree, Snapshot, etc. but have decided to speed up
 the filesystem backup 1st)

Tips are much appreciated.

Thanks -sg-
--
Steffen Gattert; VISIOPLANT Hamburg

-------- savegroup completion msg of one group ------
pls. note the backups of "E:\" on x01 & x02 & x03
backup of "E:\" is about 25% of whole file system
4 groups backing up "E:\" on x01 & x02 in parallel
 1st: [a-f]*.nsf, ...
 2nd: [g-k]*.nsf, ...
 3rd: [l-r]*.nsf, ...
 4th: [s-z]*.nsf, ...
with 5 min. timeshift between each group

NetWorker savegroup: (notice) Lotus_Notes_Server_A_F completed
Start time:   Fri Feb 17 00:33:03 2006
End time:     Sat Feb 18 06:48:09 2006
--- Successful Save Sets ---
  xxxxxxxx01: C:\    level=full,   1426 MB 01:49:19  17770 files
  xxxxxxxx01: D:\    level=full,   4238 MB 02:01:31    689 files
  xxxxxxxx01: E:\    level=full,    141 GB 16:04:51  10370 files
<--!!!
  xxxxxxxx01: SYSTEM_STATE:\ level=full, 13 MB 00:00:21     17 files
  xxxxxxxx01: SYSTEM_DB:\ level=full, 874 KB 00:00:13     11 files
  xxxxxxxx01: SYSTEM_FILES:\ level=full, 227 MB 00:09:20   1848 files
index:xxxxxxxx01 level=full, 32 MB 00:00:33    164 files
  xxxxxxxx02: C:\    level=full,   2468 MB 03:46:28  18397 files
  xxxxxxxx02: D:\    level=full,   4844 MB 02:20:18   1137 files
  xxxxxxxx02: E:\    level=full,    124 GB 22:19:15   5128 files
<--!!!
  xxxxxxxx02: SYSTEM_STATE:\ level=full, 13 MB 00:02:02     15 files
  xxxxxxxx02: SYSTEM_DB:\ level=full, 895 KB 00:02:23     11 files
  xxxxxxxx02: SYSTEM_FILES:\ level=full, 226 MB 00:10:50   1841 files
index:xxxxxxxx02 level=full, 57 MB 00:00:25    166 files
  xxxxxxxx03: C:\    level=full,   1389 MB 00:52:05  15325 files
  xxxxxxxx03: D:\    level=full,   8792 MB 03:11:47   1314 files
  xxxxxxxx03: E:\    level=full,    118 GB 22:13:57   4532 files
<--!!!
  xxxxxxxx03: SYSTEM_STATE:\ level=full, 12 MB 00:02:15     15 files
  xxxxxxxx03: SYSTEM_DB:\ level=full, 889 KB 00:02:06     11 files
  xxxxxxxx03: SYSTEM_FILES:\ level=full, 226 MB 00:11:00   1800 files
index:xxxxxxxx03 level=full, 49 MB 00:00:24    122 files
  xxxxxxxx04: C:\    level=full,    746 MB 00:42:11  11025 files
  xxxxxxxx04: D:\    level=full,    917 MB 00:32:32   9819 files
  xxxxxxxx04: E:\    level=full,     18 GB 04:40:51   1755 files
  xxxxxxxx04: SYSTEM_STATE:\ level=full, 12 MB 00:02:11     15 files
  xxxxxxxx04: SYSTEM_DB:\ level=full, 877 KB 00:02:06     11 files
  xxxxxxxx04: SYSTEM_FILES:\ level=full, 226 MB 00:10:36   1820 files
index:xxxxxxxx04 level=full, 11 MB 00:00:10     96 files

To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the
body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems
wit this list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER

-- 
This message (and any attachments) is for the recipient only. NERC
is subject to the Freedom of Information Act 2000 and the contents
of this email and any reply you make may be disclosed by NERC unless
it is exempt from release under the Act. Any material supplied to
NERC may be stored in an electronic records management system.

To sign off this list, send email to listserv AT listserv.temple DOT edu and 
type "signoff networker" in the
body of the email. Please write to networker-request AT listserv.temple DOT edu 
if you have any problems
wit this list. You can access the archives at 
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER