Re: [Bacula-users] Mark files/directories take 28hours

Hi Heitor/All,

I found set "synchronous_commit = off" has very much improvement of rebuilding catalog using bscan command.

Do you have any idea of why mark a huge number of files take so long? After marked the files, I also found restoring data to local harddisk took a long time. I could see the speed took around 20MBps? It seems to be the bottle neck at bacula-fd and bacula-sd process since the harddisk (SATA) speed tested is greater than 100MBps. Many thanks!

Regards,

Keith

Show original message

On Wednesday, August 24, 2016 2:17 AM, Heitor Faria <heitor AT bacula.com DOT br> wrote:

Hi there,

Hello, Keith,

I was trying to restore a huge data to local harddisk from an offline backup harddisk. There are 36,225,746 files and the data size is 1.7TB.

Steps to restore data:
1. Rebuild catalog using bscan
2. bconsole > restore > option 3 "Enter list of comma separated JobIds to select" > enter {jobid} > mark folder_to_be_restored

After entered "mark folder_to_be_restored", it took about 28 hours to show $ prompt and let me proceed to enter "done". Not sure why took so long. Is it possible to do some tuning? I could see there is a process "bacula-dir" which occupied 4GB memory only. While postgres processes are below 2GB memory. There are 7.7GB is free and 3.5GB for cache.

System configuration:
OS: 2.6.35.14-106.fc14.x86_64
Mem: 16G
Partition1(for OS): RAID1
Partition2(for data restoration): RAID1
Bacula: 5.0.3
Postgresql: 8.4.9-1.fc14.x86_64

#postgresql.conf
shared:2GB
effective_cache_size = 8GB
maintenance_work_mem=512MB
work_mem:128MB
wal_buffers = 8MB
checkpoint_segments = 64
checkpoint_timeout = 20min
checkpoint_completion_target = 0.9
synchronous_commit = off

I would use Postgresql "synchronous_commit = on". It's the single option (I tested) that most improves T/s, but other people from this list have deep database knowledge and may suggest other modifications.

#sysctl.conf
kernel.shmmax = 6442450944
kernel.shmall = 4194304

Best,
Keith

Regards,

===========================================================================
Heitor Medrado de Faria - LPIC-III | ITIL-F | Bacula Systems Certified Administrator II

Do you need Bacula training? http://bacula.us/video-classes/

+55 61 8268-4220

Site: http://bacula.us FB: heitor.faria

===========================================================================

------------------------------------------------------------------------------

_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users