Veritas-bu

[Veritas-bu] Filesystem Backup Speed

2002-04-14 23:48:41
Subject: [Veritas-bu] Filesystem Backup Speed
From: john.martin AT redenterprise DOT com (John Martin)
Date: Mon, 15 Apr 2002 13:48:41 +1000
This is a multi-part message in MIME format.

------=_NextPart_000_0006_01C1E484.4085A180
Content-Type: text/plain;
        charset="iso-8859-1"
Content-Transfer-Encoding: 7bit

Cleaning.Colleagues,

I've been doing some research recently and I'd like to do a sanity check, so
I'd appreciate any feedback, anecdotal or otherwise.

>From my experience, recent research etc I've found the following

1. The average filesystem is made up roughly of

1/3 files below 50K

1/3 files between 50 and 500K

1/3 files 0.5-10 MB

2. The average backup speed for NTFS filesystems is about 5MB/sec for single
spindle, and 6-7MB/sec for striped RAID, or 10+ MB/sec for high end (e.g.
EMC with lots of cache)

3. Netware filesystem backup is approximately 30% slower than the equivalent
NTFS backup speed, possibly due to inefficiencies in SMDR, in fact I've
never seen a Netware machine backup faster than 6.5MB/sec regardless of
multiple streams, memory, cpu, filesize etc. This has to be a real worry for
those sites with TB sized NSS volumes (44+ hours to do a full backup - eeek
!!)

4. VxFS/UFS/Ext2fs is approximately 10% faster than the equivalent NTFS
backup speed

5. Backing up hundreds of thousands of small files (< 10K) can drop backup
performance to less than 500Kb/sec

6. Backing up badly fragmented large files (e.g. Lotus Notes Mail files),
can drop backup performance to less than 2MB/sec

7. Most changes in incremental backups are new/additional files rather than
modification of existing files (excluding mail files such as PST and Lotus
notes), and typically is about 2-5 files per user per day.

8. Most of the speed problems associated with filesystem backup is the
operating system overheads in the file open and file close system calls,
rather than fragmentation, which may explain the differences in the backup
speeds for various O/S's

I know that these are fairly large assumptions to make, however in the
absence of hard data which is rarely available on a site by site basis, I
was wondering if others feel that these are a reasonable basis for making
design decisions.

Regards

John Martin

Red Enterprise International Pty Ltd

Suite 125, National Innovation Centre

Australian Technology Park

Eveleigh NSW 1430

Australia

Ph: +61-2-9209-4540

Fax: +61-2-9310-7232

Mobile: +61-412-313-064

  -----Original Message-----
  From: veritas-bu-admin AT mailman.eng.auburn DOT edu
[mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]On Behalf Of Barda, Boaz
  Sent: Sunday, 14 April 2002 11:46 PM
  To: 'veritas-bu AT mailman.eng.auburn DOT edu'
  Subject: [Veritas-bu] Cleaning.


  Hi all,

  I just configured my new StorageTek L80 drives - Seagate LTO to be cleaned
every 100 hours.

  My first question is how many hours should I really use considering I
don't/cant use the TapeAlert feature?

  The second question... well problem is that It seems that I configured
everything correctly, one of my drives even say it needs cleaning...

  but nothing happens.
  Even if I initiate a manual command to clean the drive I get nothing... no
cleaning, no failure, no output at all even if /var/adm/mesages

  Thanks for any help you can provide me with.
  Best Regards,
  Barda Boaz, Texas Instruments - SDW - Israel.


------=_NextPart_000_0006_01C1E484.4085A180
Content-Type: text/html;
        charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Cleaning.</TITLE>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Diso-8859-1">
<META content=3D"MSHTML 6.00.2715.400" name=3DGENERATOR></HEAD>
<BODY>
<DIV>
<P><SPAN class=3D738224703-15042002><FONT face=3DArial color=3D#0000ff=20
size=3D2>Colleagues,</FONT></SPAN></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>I've been doing some =
research recently=20
and I'd like to do a sanity check, so I'd appreciate any feedback, =
anecdotal or=20
otherwise.</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>From my experience, =
recent research etc=20
I've found the following</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>1. The average filesystem =
is made up=20
roughly of </FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>1/3 files below 50K =
</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>1/3 files between 50 and=20
500K</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>1/3 files 0.5-10 MB =
</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>2. The average backup =
speed for NTFS=20
filesystems is about 5MB/sec for single spindle, and 6-7MB/sec for =
striped RAID,=20
or 10+ MB/sec for high end (e.g. EMC with lots of cache)</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>3. Netware filesystem =
backup is=20
approximately 30% slower than the equivalent NTFS backup speed, possibly =
due to=20
inefficiencies in SMDR, in fact I've never seen a Netware machine backup =
faster=20
than 6.5MB/sec regardless of multiple streams, memory, cpu, filesize =
etc. This=20
has to be a real worry for those sites with TB sized NSS volumes (44+ =
hours to=20
do a full backup - eeek !!)</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>4. VxFS/UFS/Ext2fs is =
approximately 10%=20
faster than the equivalent NTFS backup speed </FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>5. Backing up hundreds of =
thousands of=20
small files (&lt; 10K) can drop backup performance to less than=20
500Kb/sec</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>6. Backing up badly =
fragmented large=20
files (e.g. Lotus Notes Mail files), can drop backup performance to less =
than=20
2MB/sec</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>7. Most changes in =
incremental backups=20
are new/additional files rather than modification of existing files =
(excluding=20
mail files such as PST and Lotus notes), and typically is about 2-5 =
files per=20
user per day.</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>8. Most of the speed =
problems=20
associated with filesystem backup is the operating system overheads in =
the file=20
open and file close system calls, rather than fragmentation, which may =
explain=20
the differences in the backup speeds for various O/S's</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>I know that these are =
fairly large=20
assumptions to make, however in the absence of hard data which is rarely =

available on a site by site basis, I was wondering if others feel that =
these are=20
a reasonable basis for making design decisions.</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Regards</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>John Martin</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Red Enterprise =
International Pty=20
Ltd</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Suite 125, National =
Innovation=20
Centre</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Australian Technology =
Park</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Eveleigh NSW =
1430</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Australia</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Ph: =
+61-2-9209-4540</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Fax: =
+61-2-9310-7232</FONT></P>
<P><FONT face=3DArial color=3D#0000ff size=3D2>Mobile:=20
+61-412-313-064</FONT></P></DIV>
<BLOCKQUOTE dir=3Dltr style=3D"MARGIN-RIGHT: 0px">
  <DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft><FONT =
face=3DTahoma=20
  size=3D2>-----Original Message-----<BR><B>From:</B>=20
  veritas-bu-admin AT mailman.eng.auburn DOT edu=20
  [mailto:veritas-bu-admin AT mailman.eng.auburn DOT edu]<B>On Behalf Of =
</B>Barda,=20
  Boaz<BR><B>Sent:</B> Sunday, 14 April 2002 11:46 PM<BR><B>To:</B>=20
  'veritas-bu AT mailman.eng.auburn DOT edu'<BR><B>Subject:</B> [Veritas-bu]=20
  Cleaning.<BR><BR></FONT></DIV>
  <P><FONT face=3DArial size=3D2>Hi all,</FONT> </P>
  <P><FONT face=3DArial size=3D2>I just configured my new StorageTek L80 =
drives -=20
  Seagate LTO to be cleaned every 100 hours.</FONT> </P>
  <P><FONT face=3DArial size=3D2>My first question is how many hours =
should I really=20
  use considering I don't/cant use the TapeAlert feature?</FONT> </P>
  <P><FONT face=3DArial size=3D2>The second question... well problem is =
that It=20
  seems that I configured everything correctly, one of my drives even =
say it=20
  needs cleaning...</FONT></P>
  <P><FONT face=3DArial size=3D2>but nothing happens.</FONT> <BR><FONT =
face=3DArial=20
  size=3D2>Even if I initiate a manual command to clean the drive I get =
nothing...=20
  no cleaning, no failure, no output at all even if =
/var/adm/mesages</FONT></P>
  <P><FONT face=3DArial size=3D2>Thanks for any help you can provide me =
with.</FONT>=20
  <BR><FONT face=3DArial size=3D2>Best Regards,</FONT> <BR><FONT =
face=3DArial=20
  size=3D2>Barda Boaz, Texas Instruments - SDW - Israel.</FONT>=20
</P></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_0006_01C1E484.4085A180--


<Prev in Thread] Current Thread [Next in Thread>