try using dd to create one really large file and back it up... that
should help indicate if the lots of small files are the problem...
dd if=/dev/zero of=/tmp/bigfile bs=1024k count=USE_SIZE_IN_MB_HERE
(using /dev/zero maybe cheating because zeros compress well?
you can use /dev/random, but you need to use a small block size
and patience--/dev/random is slow.)
steve
- - -
systems & network guy
high energy physics
university of wisconsin
> ---- Original Message ----
> From: David De Maeyer
> Hi all,
>
> First of all I would like to say thank you to all of you who answered.
> I didn't try all your suggestions yet, some of them were already
> applied with no success, some others need to be tried... I keep you
> informed.
>
> As a matter of fact, we deal with A LOT of small files which generally
> go along with a slower backup process. We have also tried another
> backup solution to see if we were too blind and... well looks like we
> all are here in our group as the other server just flies! And taking a
> closer look to the network didn't really show anything we didn't know.
>
> The problem for us is that we have no problems with other clients
> running the same OS (file service, web servers, calendar, etc.)
>
> It only affects the mail server. In the past, using a smaller
> workstation and another IMAP server, performances were from far better!
>
> Our preliminary conclusion: our backup server seems to be in trouble
> when dealing with a client hosting tons of small files.
>
> Thanks again for all your suggestions! We keep investigating.
> Regards,
> David
>
> ___________________________________________________
> David De Maeyer
> Roskilde University Center, Department of Computer Science
> Box 260, Hus 42.1, 4000 Roskilde, Denmark
> voice (+45) 46 74 38 29 / fax (+45) 46 74 30 72
>
>
>
>
> On 30 Apr 2004, at 03:14, Thomas, Calvin wrote:
>
> > To help find the bottleneck:
> >
> > Quickly find network problems by telneting to one of the servers,
> > opening
> > ftp, ftp "putting" a copy of any convienent 100MB file, then "getting"
> > the
> > same (copy) of the file back again. Takes seconds, tells the kbps/mbps
> > transfer speed between both machines in both directions. THIS IS
> > IMPORTANT
> > since I have had machines that can put at 10MBPS, but get at 15KBPS.
> >
> >
> >
> >
> > -----Original Message-----
> > From: Steve Rader [mailto:rader AT GINSENG.HEP.WISC DOT EDU]
> > Sent: Thursday, 29 April, 2004 01:23 PM
> > To: NETWORKER AT LISTMAIL.TEMPLE DOT EDU
> > Subject: Re: [Networker] backup mail server very slow (linux)
> >
> >
> > there's been some interesting talk about where the bottleneck may
> > be, but there really hasn't been any talk about how to find the
> > bottleneck...
> >
> > try "top", "vmstat 1" to verify that there aren't cpu and/or
> > memory bottlenecks
> >
> > try "bonnie++" and "time uasm ... > /dev/null" to verify
> > there isn't a disk read bottleneck
> >
> > try "iperf" to verify there isn't a network througput
> > bottleneck
> >
> > i'd be curious to hear other suggestions from folks on the list
> > that deal with NetWorker preformance troubleshooting.
> >
> > iperf is nice because it will show up fast ethernet auto-neg
> > problems almostly immediately (use -i 1).
> >
> > if you've got one of those MTAs that delivers each msg into it's own
> > file, you may never get good throughput. i routinely see backups of
> > huge source code trees drop into the ~ 60-80 KB/sec range (whereas
> > i usually get 2-4 MB/sec) going to DLT4K tapes.
> >
> > steve
> > - - -
> > systems & network guy
> > high energy physics
> > university of wisconsin
> >
> >> ---- Original Message ----
> >> From: David De Maeyer
> >> Hi all!
> >>
> >> We are facing a performance issue when taking a full backup of our
> >> mail
> >> server (>7000 users, 350 GB of data, running Debian linux).
> >>
> >> The backup server has no problem with the other linux clients. It is
> >> writing at around 100 KB/s... which is far from the 2 to 8 MB/s we are
> >> used to have on the other clients.
> >>
> >> It takes forever to backup all the user directories! And we simply
> >> can't find where the bottleneck is... just that there are a lot of I/O
> >> on the RAID on which these directories are stored.
> >>
> >> Would it help to increase/decrease the parallelism for the client
> >> and/or other clients (smaller filesystems)?
> >>
> >> Does somebody on this mailing-list once faced a similar problem? Is
> >> the
> >> upgrade of the client on the mail server a must?
> >>
> >> Regards,
> >> David
> >>
> >> server: NetWorker 7.1.1 (running Solaris 8)
> >> client: NetWorker 6.1.1 (running Debian linux, IMAP server, 350 GB on
> >> external RAID)
> >>
> >> ___________________________________________________
> >> David De Maeyer
> >> Roskilde University Center, Department of Computer Science
> >> Box 260, Hus 42.1, 4000 Roskilde, Denmark
> >> voice (+45) 46 74 38 29 / fax (+45) 46 74 30 72
> >>
> >> --
> >> Note: To sign off this list, send a "signoff networker" command via
> >> email
> >> to listserv AT listmail.temple DOT edu or visit the list's Web site at
> >> http://listmail.temple.edu/archives/networker.html where you can
> >> also view and post messages to the list.
> >> =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
> >
> > --
> > Note: To sign off this list, send a "signoff networker" command via
> > email
> > to listserv AT listmail.temple DOT edu or visit the list's Web site at
> > http://listmail.temple.edu/archives/networker.html where you can
> > also view and post messages to the list.
> > =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
> >
> > --
> > Note: To sign off this list, send a "signoff networker" command via
> > email
> > to listserv AT listmail.temple DOT edu or visit the list's Web site at
> > http://listmail.temple.edu/archives/networker.html where you can
> > also view and post messages to the list.
> > =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
>
> --
> Note: To sign off this list, send a "signoff networker" command via email
> to listserv AT listmail.temple DOT edu or visit the list's Web site at
> http://listmail.temple.edu/archives/networker.html where you can
> also view and post messages to the list.
> =*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
--
Note: To sign off this list, send a "signoff networker" command via email
to listserv AT listmail.temple DOT edu or visit the list's Web site at
http://listmail.temple.edu/archives/networker.html where you can
also view and post messages to the list.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
|
