Hello,
I misspoke, i did not realize that many files were in the saveset, i only
looked at saveset size, this issue regarding large number of small files
has been around for long time, the only reasonable option might be
snapshots and\or NDMP backup ( i believe you can buy and use NDMP licence
for server file systems-please check).
Thanks
Allan Nelson <an AT CEH.AC DOT UK>
Sent by: EMC NetWorker discussion <NETWORKER AT LISTSERV.TEMPLE DOT EDU>
03/19/2009 09:12 AM
Please respond to
EMC NetWorker discussion <NETWORKER AT LISTSERV.TEMPLE DOT EDU>; Please respond
to
Allan Nelson <an AT CEH.AC DOT UK>
To
NETWORKER AT LISTSERV.TEMPLE DOT EDU
cc
Subject
Re: How to speed up this backup (4GB -> 8 hours) ?
I'd echo what David Browning says below. Networker isn't good with
filesystems containing millions of files. (No disrepect to Networker - I
don't know if other vendors are or not).
As well as the number of files, you also have a very small average file
size there 127GB/9million is around 1.4k average file size.
I certainly have issues with some systems that have millions of small
files like this.
By all means check your network card/drivers/settings etc, but I think
this might be about as good as it gets with that filesystem.
Cheeers... Allan.
>>> DBrown AT LSUHSC DOT EDU 19/03/09 13:34:45 >>>
I think Networker just doesn't do large filesystem backups well, from a
standard backup to tape view.
In our case, we have 3 servers (out of over 300) that have huge
filesystems (user shares).
Our largest, which we divided into 2 separate drives/savesets (done to
help improve FULL backup speeds) have 6,772,755 and 4,495,494 files. Just
over 3.5 TB total.
Our FULL backup last month ran for 44 hours - yes that's 44 hours - I hope
every month that we don't have any issues during that weekend. For the
most part, we don't.
Anyway, I checked some of our incremental times, and they are similar to
yours - generally take about 6 hours every night (for each saveset). It
can be as short as 5 1/2, and as long as 7 hours. We seem to run a little
faster than you, but we still need at least 6 hours every night.
Some quick snapshots
5:39 44,232 files, 89GB
5:08 14,931 files, 58GB
5:29 13,387 files, 59GB
5:59 47,798 files, 118GB
6:47 45,564 files, 99GB
So, I don't think you are in that bad of shape. Do I wish my backups went
faster than this? Sure, I do. I can run a 3.5 TB exchange backup in 6 1/2
hours, but can't backup 100 GB of filesystem data in the same time. I
think it is just a limitation of disk drive, OS, and networker.
David M. Browning Jr.
IT Project Coordinator Enterprise Backups and Help Desk
-----Original Message-----
From: EMC NetWorker discussion [mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
Behalf Of Manel Rodero
Sent: Thursday, March 19, 2009 7:13 AM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: [Networker] How to speed up this backup (4GB -> 8 hours) ?
Hello,
We have a client that always seems to stall until its backups finishes
(at morning we see backup process at 99% and only one saveset for this
client running) and we would like to know where the problem is: client,
OS, filesystem, etc. so any idea would be very appreciated.
The client is 7.4.2 running in Linux Red Hat Enterprise Linux ES release
4 (Nahant Update 7). Our server is 7.4.2 but yesterday we upgraded to
7.4.4 with the same "problem".
The problematic saveset is the following:
/home2:
Used 277078076 1K-blocks
Available 723167204 1K-blocks
%occupied 28%
This saveset is mounted through a Fiber Switch Qlogic SAnSurfer attached
to a Sun Storedge 3510. The filesystem is GFS from RedHat.
Some fragments from the bootstrap log of last week:
a) For a full backup:
client1: /home2 level=full, 127 GB 12:19:56 9158073 files
b) For incremental (level) ones:
client1: /home2 level=1, 1580 KB 07:25:11 121 files
client1: /home2 level=2, 3391 MB 07:17:54 92124 files
client1: /home2 level=3, 4409 MB 07:56:02 136712 files
client1: /home2 level=4, 2748 MB 07:16:39 67574 files
We can notice in level logs that it seems the number of files doesn't
matter as the backup has, more or less, the same duration. So the
problem could be the total number of files and their comparison for
backing up only the changed ones.
Any idea about what can we look for? Any clues about which problem could
it be?
Thank you very much for any suggestions.
--
o o o Manel Rodero Blánquez | LCFIB - FIB - UPC
o o o IT System Manager | Campus Nord - Modul B6
o o o Barcelona School of Informatics | Jordi Girona, 1-3
U P C Technical University of Catalonia | 08034 Barcelona (Spain)
|
manel AT fib.upc DOT edu | Tel: +00 34 93 401 0847
http://www.fib.upc.edu/ | Fax: +00 34 93 401 7040
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type
"signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type
"signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
--
This message (and any attachments) is for the recipient only. NERC
is subject to the Freedom of Information Act 2000 and the contents
of this email and any reply you make may be disclosed by NERC unless
it is exempt from release under the Act. Any material supplied to
NERC may be stored in an electronic records management system.
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type
"signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
To sign off this list, send email to listserv AT listserv.temple DOT edu and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
