Hi,
know i got all the necessary Information (bacula-director Version 1.38.11-8):
@Jesper (the timed tar)
server:~# time tar cf /dev/null /home/mailer4/
tar: Removing leading `/' from member names
real 96m25.390s
user 0m18.644s
sys 0m57.260s
@Arno
i'm not that good in mysql, but i already tuned mysql und set some new indexes,
see below:
mysql> show index from File;
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation
| Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| File | 0 | PRIMARY | 1 | FileId | A
| 15544061 | NULL | NULL | | BTREE | |
| File | 1 | JobId | 1 | JobId | A
| 74 | NULL | NULL | | BTREE | |
| File | 1 | PathId | 1 | PathId | A
| 1195697 | NULL | NULL | | BTREE | |
| File | 1 | FilenameId | 1 | FilenameId | A
| 5181353 | NULL | NULL | | BTREE | |
| File | 1 | FilenameId | 2 | PathId | A
| 7772030 | NULL | NULL | | BTREE | |
| File | 1 | JobId_2 | 1 | JobId | A
| 74 | NULL | NULL | | BTREE | |
| File | 1 | JobId_2 | 2 | PathId | A
| 1943007 | NULL | NULL | | BTREE | |
| File | 1 | JobId_2 | 3 | FilenameId | A
| 15544061 | NULL | NULL | | BTREE | |
| File | 1 | file_jobid_idx | 1 | JobId | A
| 74 | NULL | NULL | | BTREE | |
| File | 1 | file_jpf_idx | 1 | JobId | A
| 74 | NULL | NULL | | BTREE | |
| File | 1 | file_jpf_idx | 2 | FilenameId | A
| 7772030 | NULL | NULL | | BTREE | |
| File | 1 | file_jpf_idx | 3 | PathId | A
| 15544061 | NULL | NULL | | BTREE | |
+-------+------------+----------------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
12 rows in set (0.00 sec)
mysql> show index from Path;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation |
Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Path | 0 | PRIMARY | 1 | PathId | A |
1262216 | NULL | NULL | | BTREE | |
| Path | 1 | Path | 1 | Path | A |
NULL | 255 | NULL | | BTREE | |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)
mysql> show index from Filename;
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation |
Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
| Filename | 0 | PRIMARY | 1 | FilenameId | A |
4823581 | NULL | NULL | | BTREE | |
| Filename | 1 | Name | 1 | Name | A |
NULL | 255 | NULL | | BTREE | |
+----------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+
2 rows in set (0.00 sec)
mysql> show create table File;
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table
|
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| File | CREATE TABLE `File` (
`FileId` int(10) unsigned NOT NULL auto_increment,
`FileIndex` int(10) unsigned NOT NULL default '0',
`JobId` int(10) unsigned NOT NULL,
`PathId` int(10) unsigned NOT NULL,
`FilenameId` int(10) unsigned NOT NULL,
`MarkId` int(10) unsigned NOT NULL default '0',
`LStat` tinyblob NOT NULL,
`MD5` tinyblob NOT NULL,
PRIMARY KEY (`FileId`),
KEY `JobId` (`JobId`),
KEY `PathId` (`PathId`),
KEY `FilenameId` (`FilenameId`,`PathId`),
KEY `JobId_2` (`JobId`,`PathId`,`FilenameId`),
KEY `file_jobid_idx` (`JobId`),
KEY `file_jpf_idx` (`JobId`,`FilenameId`,`PathId`)
) ENGINE=MyISAM AUTO_INCREMENT=16691910 DEFAULT CHARSET=latin1 |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.02 sec)
--
--------------------------------------------------------------------------------------
onOffice Software GmbH
Feldstr. 40
52070 Aachen
Tel. +49 (0)241 44686-0
Fax. +49 (0)241 44686-250
Email: b.kunstleben AT onOffice DOT com
Web: www.onOffice.com
--------------------------------------------------------------------------------------
Registergericht: Amtsgericht Aachen, HRB 12123
Geschäftsleitung: Stefan Mantl, Torsten Kämper, Stefan Becker
--------------------------------------------------------------------------------------
----- Ursprüngliche Nachricht -----
Von: Arno Lehmann <al AT its-lehmann DOT de>
Gesendet: Donnerstag, 27. November 2008 11:54:56
An: <Bacula-users AT lists.sourceforge DOT net>
Betreff: Re: [Bacula-users] Large maildir backup
Hi,
27.11.2008 11:31, Boris Kunstleben wrote:
> Hi,
>
> i am doing exactly that since last Thursday.
> I have about 1.6TB in Maildirs and an huge number of small files. I have to
> say it is awfull slow. Backing up a directory with about 190GB of Maildirs
> took "Elapsed time: 1 day 14 hours 49 mins 34 secs".
> On the other hand i have a server with Documents and images (about 700GB)
> took much less time.
> All the Servers are virtuall Enviroments (Virtuozzo).
>
> Any Ideas would be appreciated.
Looks like the catalog database is the bottle-neck here. All the files
need to be added to it.
There are two rather simple solutions:
- Don't keep file information for this job in the catalog. This makes
restoring single mails difficult.
- Tune your catalog database for faster inserts. That can mean moving
it to a faster machine, assigning more memory for it, or dropping some
indexes (during inserts). If you're not yet using batch inserts, try
to recompile Bacula with batch-inserts enabled.
Arno
> Kind regards Boris Kunstleben
>
>
> Silver Salonen schrieb:
>> On Thursday 27 November 2008 11:07:41 James Cort wrote:
>>
>>> Silver Salonen wrote:
>>>
>>>> On Thursday 27 November 2008 09:50:14 Proskurin Kirill wrote:
>>>>
>>>>> Hello all!
>>>>>
>>>>> Soon I will deploy a large email server - it will use maildirs and will
>>>>> be about 1Tb of mail with really many small files.
>>>>>
>>>>> It is any hints to make a backup via bacula of this?
>>>>>
>>>>>
>>>> I think Bacula is quite good for backing up maildirs as they constist of
>>>> separate files as e-mail messages. I don't think small files are a
>>>>
>> problem.
>>
>>> I don't think they're a problem either and I also backup a maildir-based
>>> mail server.
>>>
>>> However, one thing you may want to be aware of - unless you take
>>> specific steps to avoid it, the maildirs on tape won't necessarily be in
>>> a consistent state. Obviously this won't affect your IMAP server - but
>>> it does mean that when you restore, metadata like whether or not emails
>>> have been read or replied to and recently received/sent email won't be a
>>> perfect snapshot of how the mailserver looked at any given point in time.
>>>
>> And when you have many incrementals in a row while restoring, you end up
>> seeing many duplicate messages, that have been deleted or moved during these
>> incrementals.
>>
>> It'll be better in Bacula 3.0, I guess :)
>>
>> --
>> Silver
>>
>> -------------------------------------------------------------------------
>> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
>> Build the coolest Linux based applications with Moblin SDK & win great prizes
>> Grand prize is a trip for two to an Open Source event anywhere in the world
>> http://moblin-contest.org/redirect.php?banner_id=100&url=/
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users AT lists.sourceforge DOT net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>
>
>
--
Arno Lehmann
IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück
www.its-lehmann.de
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
--
--------------------------------------------------------------------------------------
onOffice Software GmbH
Feldstr. 40
52070 Aachen
Tel. +49 (0)241 44686-0
Fax. +49 (0)241 44686-250
Email: b.kunstleben AT onOffice DOT com
Web: www.onOffice.com
--------------------------------------------------------------------------------------
Registergericht: Amtsgericht Aachen, HRB 12123
Geschäftsleitung: Stefan Mantl, Torsten Kämper, Stefan Becker
--------------------------------------------------------------------------------------
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
