Re: [Bacula-users] slow copy job from disk to LTO-6 (speed issue)
2017-03-27 12:28:52
Thanks for the compliment!
I thought the 139MB/s was pretty good speed-wise as well since it
appeared to include some file seeking and other non-write time spent
positioning the tape.
The good speeds achieved confirmed my assumption that COPY jobs
don't do any transformations or processing on the data while copying
as this job's data was both compressed and encrypted in the source
SD (on disk).
~Caribe
On 03/27/2017 09:42 AM, Kern Sibbald
wrote:
Hello Caribe,
Yes, I guess old age is creping up on me, I did mean MB/sec. If you are getting 139MB/sec for a single client, you are doing very well.
Best regards,
Kern
On 03/27/2017 03:30 PM, Caribe
Schreiber wrote:
Thanks for all the good suggestions!
After enabling spooling, the speed increased to where it ought
to be:
backup-sd Spooling data ...
backup-sd Committing spooled data to Volume "P0001SL6". Despooling 54,455,614,705 bytes ...
backup-sd Despooling elapsed time = 00:06:31, Transfer rate = 139.2 M Bytes/second
For some reason I had it stuck in my head that because the copy
source was from a local disk attached to the same machine as the
tape, that spooling for the tape wouldn't be necessary. The
line I got hung up on from the spooling page in the manual (http://www.bacula.org/7.4.x-manuals/en/main/Data_Spooling.html)
is here:
"Bacula version 1.39.x and later supports writing a backup to
disk then later Migrating or moving it to a tape (or any
other medium). For details on this, please see the
MigrationMigrationChapter chapter of this manual for more
details."
It seems that the read side of the COPY is still slow because of
other overhead and needs to have spooling enabled on the write
side if it is tape. Hopefully this helps prevent someone else
from falling into the same logic trap that I did!
Kern: do you really see throughputs of 250-500GB/sec or did you
mean to say MB? I've done some I/O performance tuning projects
before, and 500GB/sec for anything is pretty amazing!
~Caribe
On 03/26/2017 09:33 AM, Kern
Sibbald wrote:
Hello,
As Laurent, I think, suggested you should not be doing Bacula compression if you are writing to an LTO-6 drive.
On a large, well tuned Bacula Enterprise installation, we see it getting backup speeds of 250 to 500GB/sec. You will probably never get close to that with a single backup job -- it requires running concurrent jobs and to have the right hardware -- a good part of it was outlined by Laurent. However to get speeds greater than say 150GB/sec, you either need to be a performance and hardware expert or have professional advice. High throughputs require knowledge of RAID, networking, fibre channel, tape drive characteristics, big hardware, and Bacula tuning experience (what directives to use and what values to set -- e.g. do not use Bacula compression, encryption, ... if you want the LTO-6 running full speed).
Best regards,
Kern
On 03/26/2017 05:24 AM, Caribe
Schreiber wrote:
*sigh*
Do you folks ever have one of those days where it seems like
you just can't win? I'm apparently having one of those
days.
Hopefully the mail client doesn't flub the line endings this
time.
Slow copy job output:
backup-sd JobId 64 Sending spooled attrs to the Director. Despooling 2,615 bytes ...
Elapsed time=02:03:24, Transfer rate=7.346 M Bytes/second
backup-dir JobId 64 No Files found to prune.
End auto prune.
Bacula backup-dir 7.4.4 (202Sep16):
Build OS: x86_64-pc-linux-gnu gentoo
JobId: 64
Job: BackupImage.2017-03-24_21.45.00_45
Backup Level: Full
Client: "primary-fd" 7.4.4 (202Sep16) x86_64-pc-linux-gnu,gentoo,
FileSet: "BackupFileSet" 2017-03-24 21:45:00
Pool: "file.full" (From Job resource)
Catalog: "MyCatalog" (From Client resource)
Storage: "file" (From Pool resource)
Scheduled time: 24-Mar-2017 21:45:00
Start time: 25-Mar-2017 00:12:38
End time: 25-Mar-2017 02:16:03
Elapsed time: 2 hours 3 mins 25 secs
Priority: 13
FD Files Written: 7
SD Files Written: 7
FD Bytes Written: 54,391,511,056 (54.39 GB)
SD Bytes Written: 54,391,515,157 (54.39 GB)
Rate: 7345.2 KB/s
Software Compression: 57.8% 2.4:1
Snapshot/VSS: no
Encryption: yes
Accurate: no
Volume name(s): full-Vol-0024|full-Vol-0029
Volume Session Id: 49
Volume Session Time: 1490104159
Last Volume Bytes: 22,911,377,270 (22.91 GB)
Non-fatal FD errors: 0
SD Errors: 0
FD termination status: OK
SD termination status: OK
Termination: Backup OK
Begin pruning Jobs older than 6 months .
No Jobs found to prune.
Begin pruning Files.
backup-dir Using Device "Drive-1" to write.
backup-sd Elapsed time=00:39:02, Transfer rate=23.22 M Bytes/second
backup-sd Alert: smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.4.39-gentoo] (local build)
Alert: Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
Alert: TapeAlert: OK
Alert: === START OF READ SMART DATA SECTION ===
Alert:
Alert: Error Counter logging not supported
backup-sd Sending spooled attrs to the Director. Despooling 2,615 bytes ...
Alert: Last n error events log page
Alert:
Alert:
dd read speed of one of the disk backup files in the job:
backup bacula # dd if=full-Vol-0029 of=/dev/null bs=4M
12799+1 records in
12799+1 records out
53687083817 bytes (54 GB, 50 GiB) copied, 111.637
s, 481 MB/s
Bonnie output:
backup ~ # bonnie++ -u root
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 1.97 ------Sequential Output------
--Sequential Input- --Random-
Concurrency 1 -Per Chr- --Block-- -Rewrite-
-Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP
K/sec %CP K/sec %CP /sec %CP
backup 126G 1211 97 601130 47 240240
17 5506 94 759982 22
698.1 10
Latency 9305us 710ms 646ms
15409us 276ms 519ms
Version 1.97 ------Sequential Create------
--------Random Create--------
backup -Create-- --Read--- -Delete--
-Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP
/sec %CP /sec %CP /sec %CP
16 16535 15 +++++ +++ +++++ +++
+++++ +++ +++++ +++ +++++ +++
Latency 385us 360us
714us 383us 370us 345us
1.97,1.97,backup,1,1490480328,126G,,1211,97,601130,47,240240,17,5506,94,759982,22,698.1,10,16,,,,,16535,15,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,9305us,710ms,646ms,15409us,276ms,519ms,385us,360us,714us,383us,370us,345us
SD config:
#
# A file changer for disk based backups
#
Autochanger {
Name = FileChgr1
Device = FileChgr1-Dev1
Changer Command = ""
Changer Device = /dev/null
}
Device {
Name = FileChgr1-Dev1
Media Type = file
Archive Device = /home/backups/bacula
LabelMedia = yes; # lets
Bacula label unlabeled media
Random Access = Yes;
AutomaticMount = yes; # when
device opened, read it
RemovableMedia = no;
AlwaysOpen = no;
Maximum Concurrent Jobs = 5
}
#
# An HP 1/8 autochanger device with one drive
#
Autochanger {
Name = AutochangerHP
Device = Drive-1
Changer Command =
"/usr/libexec/bacula/mtx-changer %c %o %S %a %d"
Changer Device = /dev/sg3
}
Device {
Name = Drive-1 #
Media Type = LTO-6
Archive Device = /dev/nst0
AutomaticMount = yes; # when
device opened, read it
AlwaysOpen = yes;
RemovableMedia = yes;
RandomAccess = no;
AutoChanger = yes
Maximum File Size = 10GB
Alert Command = "sh -c 'smartctl -H -l error
%c'"
}
I'm a little stumped about why the raw hardware read/write
speeds are so different from the bacula copy job speeds.
I'm hoping the collective wisdom of the list can give me
some ideas on other places to look for speed issues or
config changes so that I don't prematurely wear out this
drive head with shoe-shining (of course...faster backups are
always welcome too!).
Thanks in advance!
~Caribe
--
Caribe
Schreiber
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
Caribe
Schreiber
IT Engineer
Auction Harmony
Phone: (612) 605-7301 x 105
Fax: (612) 605-7302
www.AuctionHarmony.com
Join Us!
--
Caribe Schreiber
IT Engineer
Auction Harmony
Phone: (612) 605-7301 x 105
Fax: (612) 605-7302
www.AuctionHarmony.com
Join Us!
|
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
|
|