Re: [Bacula-users] Backup of many many files

> -----Ursprüngliche Nachricht-----
> Von: Christoph Litauer [mailto:litauer AT uni-koblenz DOT de]
> Gesendet: Dienstag, 2. September 2008 14:01
> An: Daniel Betz
> Cc: bacula-users AT lists.sourceforge DOT net
> Betreff: Re: [Bacula-users] Backup of many many files
>
> Daniel Betz schrieb:
> > Hi Jose,
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Jose Ildefonso Camargo Tolosa
> [mailto:ildefonso.camargo AT gmail DOT com]
> >> Gesendet: Dienstag, 2. September 2008 03:11
> >> An: Daniel Betz
> >> Cc: bacula-users AT lists.sourceforge DOT net
> >> Betreff: Re: [Bacula-users] Backup of many many files
> >>
> >> Hi!
> >>
> >> On Tue, Sep 2, 2008 at 3:15 AM, Daniel Betz <dbetz AT df DOT eu> wrote:
> >>> Hi all,
> >>>
> >>>
> >>>
> >>> can you give me some tips for backing up an filesystem with many
> many
> >> small
> >>> files on it ?
> >>>
> >>> The problem is, that i must backup mailservers with about 300 GB
> and
> >> more
> >>> than 1-2 million of many small files (Maildir).
> >> These are not *that* much files, I got a backup with 1.2M files, and
> >> 600GB on it.
> >>
> >
> > There are some servers with over 6 million of files in there.
> >
> >>> The File Table in the bacula database grows up to more than 3GB and
> >> the
> >>> backups takes more than 30 hours.
> >> What database server are you using?  more details on your backup
> >> (network speed, backup media, etc...).
> >>
> >>> Is this an good time, or is there an way to optimize this eg. to
> run
> >> more
> >>> cuncurrent jobs than one on the client ?
> >>>
> >>> When i turn on concurrent jobs must i enable data spooling ?
> >>>
> >>>
> >>>
> >>> This makes it impossible to restore single files "quick".
> >>>
> >>>
> >>>
> >>> When you need more information, please let me know.
> >>>
> >> Bacula version, which database (including version), and, off course,
> >> your platform.  Any extra info would be good.
> >>
> >
> > Media Type = Files.
> > Bacula Version 2.4.2
> > MySQL 5.0.45-7.el5 (latest vom CentOS 5.2)
> > CentOS 5.2 Final with Kernel 2.6.18-92.1.10.el5PAE on backupservers
> and 2.6.18-53.1.14.el5PAE on clients
> > 100 Mbit LAN
> > backupserver:
> > - Celeron 2.8 Ghz
> > - 1 GB RAM
> > - 4x SATA 500GB in RAID 0
> > clients:
> > - Quad Xeon 1,6Ghz
> > - 4 GB RAM
> > - 8x SAS 250 GB in RAID 10
> >
> > the speedup is only around 2 MB/s
> >
>
> Do you use compression? I had rates between 2-4 MB/s when using default
> compression. Switching to compression level 1 increased speed a litte.
> No compression should increase very much.
>

I use only GZIP1 for compression, that increases the speed a little bit,
but the backups are still horrible slow.
Heres an example from an server with not so many files:

+-------+----------+---------------------+------+-------+-----------+----------------+-----------+
| JobId | Name     | StartTime           | Type | Level | JobFiles  | JobBytes  
     | JobStatus |
+-------+----------+---------------------+------+-------+-----------+----------------+-----------+
|     1 | boreas   | 2008-09-02 01:00:00 | B    | F     | 2,187,452 | 
35,766,848,686 | T         |
+-------+----------+---------------------+------+-------+-----------+----------------+-----------+

02-Sep 12:22 backupserver JobId 1: Job write elapsed time = 11:22:02, Transfer 
rate = 882.8 K bytes/second
02-Sep 12:27 backupserver JobId 1: Bacula backupserver 2.4.2 (26Jul08): 
02-Sep-2008 12:25:21
  Build OS:               i686-pc-linux-gnu redhat
  JobId:                  1
  Job:                    boreas.2008-09-02_01.00.02
  Backup Level:           Full (upgraded from Incremental)
  Client:                 "boreas" 2.4.2 (26Jul08) i686-pc-linux-gnu,redhat,
  FileSet:                "Alles" 2008-09-02 01:00:00
  Pool:                   "boreas" (From Job resource)
  Storage:                "boreas" (From Job resource)
  Scheduled time:         02-Sep-2008 01:00:00
  Start time:             02-Sep-2008 01:00:00
  End time:               02-Sep-2008 12:25:21
  Elapsed time:           11 hours 25 mins 21 secs
  Priority:               10
  FD Files Written:       2,187,452
  SD Files Written:       2,187,452
  FD Bytes Written:       35,766,848,686 (35.76 GB)
  SD Bytes Written:       36,126,979,687 (36.12 GB)
  Rate:                   869.8 KB/s
  Software Compression:   40.8 %
  VSS:                    no
  Storage Encryption:     no
  Volume name(s):         boreas-0001
  Volume Session Id:      1
  Volume Session Time:    1220298214
  Last Volume Bytes:      36,232,078,544 (36.23 GB)
  Non-fatal FD errors:    0
  SD Errors:              0
  FD termination status:  OK
  SD termination status:  OK
  Termination:            Backup OK

the other server are still backing up :-(

Greetings,
Daniel

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users