Backup Performance - The 8GB/Min Goal
Our future storage space requirements were estimated to be 4 TB. Our
requirement is to complete the backups within an 8 hour window. To meet
this objective required a backup solution that could provide an 8 GB/Min
solution. We researched several technologies to obtain the required
level of performance. After months of research we decided to base our
solution around an article written by HP, "Getting the most performance
from your HP StorageWorks Ultrium 960 tape drive[1]". This white paper
discussed several technologies and methodologies to obtain the
performance levels desired. The article also provide information for
combining our backup solution of choice, Veritas NetBackup[2] and used
hardware that had excellent reviews[3]. During this process we decided
to use a DAS and NAS storage solution. We decided to avoid SAN storage
and disk to disk storage because of the overhead, complexity, and cost
associated with these technologies. In addition, we decided to implement
a single vendor solution to enhance support. Once our design was in
place with worked with multiple resellers and the manufacture, HP
directly to insure the equipment ordered with meet or exceed the
performance levels outlined in the HP performance article above. Once
the equipment was in place we were unable to obtain the results
documented. The article highlights some of our findings along the way. I
think our findings will raise an eyebrow or two and you will find it a
worthy read. After our ordeal we did identify a technique to obtain the
performance desired. After reading this article if you have experienced
the same challenges or are equally disappointed with HP's position we
would like to hear from you. Below is our tale.
Objective: Setup a simple, scalable, high performance single vendor DAS
backup solution.
Hardware:
DL580 G3
Dual Processor
4 GB RAM (2 GB per processor)
4 15K RPM drives
LSI Ultra 320 SCSI Controller for Tape Storage Unit
Max SCSI cache memory and battery to enable write back cache
MSA500 G2
Model with dual Ultra 320 SCSI channel
14 15K RPM, 72 GB Drives for performance
MSA30
14 15K RPM, 72 GB Drives for performance
Model with dual Ultra 320 SCSI channels
MSL6030
17 TB Backup storage capacity
Dual LTO 3 drives for performance and required backup
operations
Software Tools Utilized:
Intel Iometer
HP PAT (Performance Analysis Tool)
HP Read Data
Windows Explorer file copy
HP Library & Tape tool Prebackup test
Tests:
Over several months working with HP we tried a myriad of
configurations and tests.
These tests included but are not limited to the following.
Various Windows 2003 Server Service Packs & Hotfixes
BIOS updates for system, SCSI, Drives, Tape Lib, Tape Drive
etc.
Driver updates and versions for system, array controller,
and tape
Differing drivers to include both SCSI Mini-Port and
Non-Mini-Port drivers
Changing strip size on drive arrays
Changing RAID level from 0, 1, 5, and 10.
Changing Read/Write cache Levels
Changing SCSI driver parameters
Changing software buffer, memory, and block size parameters
Using dual data channels to MSA500
These tests were repeated using an MSA30.
Testing Results:
* Stripe size, Number of drives and RAID level made no
significance impact on the performance.
* MSA 500 G2 does not use two independent SCSI channels. The
second SCSI interface is only for failover and redundancy. With the same
configuration MSA 30 outperforms MSA 500 G2 using single SCSI channel.
* "Iometer" provides unusually high through put results. We
learned that this tool bypasses the Windows system drivers. Therefore
this tool is not an accurate measure of the performance.
* "HP PAT" also bypasses Windows system drivers and is not
recommended for benchmarking.
* The test data created by "HP Create Data" utility is euphoria
data and is not realistic. The data sizes created are perfect and no
performance loss is realized with SCSI data packets that range in sizes.
Expect your results to be approximately 50% or worse then the
performance numbers obtained using this data. "HP Read data", "HP
Prebackup test", "Windows Explorer" file copy and "VERITAS Netbackup"
Null test provided comparable performance results and were consistent
with 'real world' findings.
* Performance was never any better than it would have been with a
single drive. It runs as fast as the single drive.
* Performance was related to number of logical units (LUN).
Results obtained were 3 to 4 times better when LUNs were increased.
* HP SCSI drivers for Microsoft Windows have a 64K block size
limitation. All the test tools which could bypass this driver and set
communication parameters on the SCSI bus obtained increased performance.
All of the applications which utilized the system driver suffered.
*
Problem Identification:
These results lead us to the answer. The performance problem was with
the system driver. We learned that with the increase of performance of
the Ultra 160 and Ultra 320 SCSI controllers that the drivers were not
sufficient. With the release of Windows 2003 Microsoft release the
article, "Storport in Windows Server(tm) 2003: Improving Manageability
and Performance in Hardware RAID and Storage Area Networks[4]" that
addressed this problem and recommended the development of "Storport"
drivers.
The newest version of the drivers provided by HP are reported to be
"Storport" drivers. However, these drivers do not incorporate certain
performance enhancements. HP stated that additional development was
intended for these drivers. Future development was targeted towards the
serial attached SCSI drives.
Immediate Solution:
We learned that we could increase system performance, increase drive
space; provide fault tolerance and lower costs!
Our original design was to implement a RAID 10 solution using two 7 disk
RAID 0 arrays. This results in 490 GB of storage and approximately 1.8
GB/min data rate. Our solution was to exchange the MSA500 with an MSA30
and exchange the 72 GB 15 K RPM drives with 146 GB 15 K RPM drives. We
then setup two 4 disk RAID 5 Arrays, and two 3 disk RAID 5 Arrays. This
solution substantially increased disk space and provided almost 4 times
the performance. With this confirmation we were able to through put in
the neighborhood of 7 GB/Min. When combined with our NAS backup solution
we expect to bring our data rates up to our objective of 8 GB/Min.
Preliminary testing was promising.
Long Term Solution:
The long term solution would be for HP to provide the enhancements to
their system driver. During our conversations with HP no future
development is scheduled for these drivers. We were surprised by this
position as our server was only a few months old, and we purchase a care
pack to provide future enhancements. HP's response was that maintenance
updates would be provided to correct problems but no enhancements would
be forth coming. They encourage movement toward serial SCSI
implementations. We looked at the current technology available and their
product line does not support high speed drives, not does it provide any
migration paths to scale to larger systems. Even understanding this HP
held firm in their position. During one of conversations HP stated that
if they released drivers with the performance enhancements customers
would not need to buy newer systems!
We have a solution that will meet our needs. However, we are
disappointed in HP's position. We are interested in hearing your
opinions and experiences. If you would like any addition information
pertaining to our test or would like to share your own results please
feel free to contact us at Performance at vasc.com
________________________________
[1] Getting the most performance from your HP StorageWorks Ultrium 960
tape drive
[2] Tale of the Tape by Steven J. Schuchart Jr.
[3] HP Ultrium vs. IBM Ultrium vs. Seagate Ultrium vs. HP SuperDLT, by
Rothery Harris
[4] Storport in Windows Server(tm) 2003: Improving Manageability and
Performance in Hardware RAID and Storage Area Networks dated December
2003
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
http://mailman.eng.auburn.edu/pipermail/veritas-bu/attachments/20060701/6164d351/attachment-0001.html
|
