-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

mehma sarja wrote:
> MY SITUATION
> 
> I can take your Megabytes and shame you with my 9,868,868 mostly Maildir
> files and 690.8 GB space they take up. Take that! It took 25 hours to
> transfer and is currently "indexing." Before I ramble on, here is some
> confguration info:

Indexing?  Sorry, what do you mean?  Do you have some output of some
command?

> CONFIGURATION
> 
> dir Version: 2.4.2 (26 July 2008), amd64 freebsd7.1, PG version 8.2

I don't know if 2.4.4 helps, or if PostgreSQL 8.3.7 is better.

> The OS is a stripe of mirrors (2x2) running on FreeBSD ZFS filesystem
> and a spare (in a pear tree). Hey, it is still almost winter.

I would try plain old UFS as a comparison.

> WHAT?!
> 
> Yes, this humongo transfer has been done TWICE before and the indexing
> just runs for days till I get sick of it and kill PG.

What is PG doing?  Have you looked at that?

> WHAT HAVE YOU DONE?!
> 
> I have tweaked FreeBSD to run on all 4 CPU cores

Number of cores may not help if everything is running in serial.

Have you considered dividing the backup into four and running four
concurrent jobs?

Is PostgreSQL, bacula-sd, and bacula-dir running on the same box?

> and tweaked PG
> (postgresql.conf) thus:
> 
> # - Memory -
> 
> shared_buffers = 10MB # min 128kB or max_connections*16kB
> temp_buffers = 10MB # min 800kB
> work_mem = 10MB # min 64kB
> maintenance_work_mem = 40MB # min 1MB
> max_stack_depth = 525kB # min 100kB
> 
> # - Free Space Map -
> max_fsm_pages = 30000 # min max_fsm_relations*16, 6 bytes each
> max_fsm_relations = 1000 # min 100, ~70 bytes each

The experts there would be the PostgreSQL mailing lists.

> WHAT DID THIS GET YOU?
> 
> Well, the transfers for one finished 22% faster (25 instead of 32
> hours). And "top" does not show a high number for IVCSW - involuntary
> context switching. This means it is using more cores and maybe more
> memory. I was hoping it would also speed up indexing but I am now 20
> hours into indexing after the transfer completed.

vmstat and iostat output may help you diagnose.

> CAN SOMEONE HELP ME?
> 
> I suspect the PG tweaks are not right and I am dealing with a large
> number of files are my main problems. It could also have something to do
> with the two previous indexing which I stopped.

Perhaps.  Have you run any vacuums on PostgreSQL?

- --
Dan Langille

BSDCan - The Technical BSD Conference : http://www.bsdcan.org/
PGCon  - The PostgreSQL Conference:     http://www.pgcon.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (FreeBSD)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAknP9ZoACgkQCgsXFM/7nTw5BwCgyHG3s4odNe7ClQCw+Bgbqz5c
klIAoKnRt+0IO9+j05cS8HmgC/TBINtK
=PEgb
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users