Hi Thomas,
Yea, I thought about the "load" issue. However, in this specific case, I'd
rather have the client push as much data regardless of load on the system -
it's a dedicated SAN storage node.
I did some investigation and discovered the following data path that backups
take:
- Data is sent to 'save/savepnpc' via a file descriptor from disk - this is
reasonable.
- Data is sent to 'nsrmmmd' via localhost:TCP from 'save' - this surprised me.
Why TCP and not UDP through the localhost interface?
- Data is sent to 'tape' via file descriptor - this is reasonable.
Or to put it another way: Disk -> Save -> localhost:TCP -> nsrmmd -> Tape
Using 'iperf' I measured the throughput of 'localhost:TCP' and I got ~55
MB/Sec. Not too bad since our 9840A's can _each_ get ~25+ MB/Sec w/our data
set.
I also tried renice'ing 'save' to a '-15' priority (just like nsrmmd) but it
didn't help any (or impact the system for that matter). I also tried enabling
'write buffering' at the 'st' driver level; but Solaris 8 has deprecated that
functionality - 'aioread' and 'aiowrite' need to be used instead. So that
means a source code change which probably doesn't exist.
I'll try the FTP trick you mentioned and see what comes of it. BTW, is there a
way to use '/dev/null' as a device rather than '/dev/rmt/XXXubn'? I'd like to
be able to take the drive and SAN out of the loop. Makes it easier to point
the finger.
As for disk storage. It's a Hitachi 9970. The _single_ filesystem is UFS @
~550 GB for Oracle "stuff". If you're not familier with the 9970, it's pretty
impressive hardware and we know it's got the HP to supply the data. As
evidenced by using multiple save streams.
I've broken up the subdirs into savesets (about ~15) but the u03 and u05 are
the biggies so they are always the reamaining savesets that take many more
hours to backup than the other subdirs.
> My problem is why I can't backup my Windows 2000 server with 150,000 small
> files at a speed over 20-40KB/second. Now that's a problem............;-)
We've got a bretty beefy W2K system here with ~1.2 Million small files and it
takes a lot of time /w 6.1.3. NW7 is supposed to come out with "block level"
support (if I understood correctly) to overcome this type of performance issue.
Thanks for you input,
/\/elson
On Mon, 12 May 2003, Thomas, Calvin wrote:
> I can't say for sure, but according to my testing, each save stream is
> limited to a certain percentage of the total machine. It makes sense since
> Legato wants a backup to run in the background without bringing the client
> to it's knees. The way to do a backup when you want speed without regard to
> client load is (as you discovered) to schedule multiple save streams.
> Clumsy but effective.
>
> Is this single save stream from a single hard drive, or an array? Try an
> FTP "put" from the save stream source location to a known fast file system.
> Then FTP "get" back to the same location. Compare this to the networker
> speeds you get. You may be surprised.
>
> Also bear in mind that 7MB/Sec is faster than many tape drives can handle.
>
>
> Calvin Thomas
> UNIX System Administrator
> NACA Logistics
>
>
>
>
> -----Original Message-----
> From: David E. Nelson [mailto:david.nelson AT NI DOT COM]
> Sent: Thursday, 08 May, 2003 07:39 AM
> To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
> Subject: [Networker] Why can't 'save' or 'savepnpc' write more ~7MB/Sec?
>
>
> Hi All,
>
> Through recent testing w/ a Hitachi 9970 and one FCAL 9840A, it was
> determined
> that a single save stream could only drive ~7MB/Sec. When multiple save
> streams were implemented on the same filesystem, we where able to achieve
> +25MB/Sec.
>
> Is there any tuning that can be done so that one save stream can achieve max
> throughput?
>
> Thanks,
> /\/elson
>
>
>
> --
> ~~ ** ~~ If you didn't learn anything when you broke it the 1st ~~ ** ~~
> time, then break it again.
>
> --
> Note: To sign off this list, send a "signoff networker" command via email
> to listserv AT listmail.temple DOT edu or visit the list's Web site at
> http://listmail.temple.edu/archives/networker.html where you can
> also view and post messages to the list.
> =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
>
--
~~ ** ~~ If you didn't learn anything when you broke it the 1st ~~ ** ~~
time, then break it again.
--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
|
