Dne 24.3.2011 10:08, bacula-users-request AT lists.sourceforge DOT net
napsal(a):
> Send Bacula-users mailing list submissions to
> bacula-users AT lists.sourceforge DOT net
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://lists.sourceforge.net/lists/listinfo/bacula-users
> or, via email, send a message with subject or body 'help' to
> bacula-users-request AT lists.sourceforge DOT net
>
> You can reach the person managing the list at
> bacula-users-owner AT lists.sourceforge DOT net
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Bacula-users digest..."
>
>
> Today's Topics:
>
> 1. Re: Bacula 5.0.2 - Action On Purge = Truncate not working
> (Josh Fisher)
> 2. Re: waiting on max Storage jobs (Dan Langille)
> 3. Bacula jobs causing VM server reboot& consequent job fail
> (farmershort)
> 4. Re: bacula-dir.conf client config (Geert Stappers)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 23 Mar 2011 16:29:39 -0400
> From: Josh Fisher<jfisher AT pvct DOT com>
> Subject: Re: [Bacula-users] Bacula 5.0.2 - Action On Purge = Truncate
>
> not working
> To: bacula-users AT lists.sourceforge DOT net
> Message-ID:<4D8A5833.8000700 AT pvct DOT com>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
>
> On 3/23/2011 4:14 PM, Ondrej PLANKA (Ignum profile) wrote:
>> Ciao!
>>
>> same behavior in 5.0.3 version. File based Volumes are not truncated
>> after console command "purge volume action=all allpools storage=File "
>>
>> Any hints?
> Does it work with pool=poolname, rather than allpools?
Unfortunately this is also not working, same result .....
>> Thanks, Ondrej.
>>
>> Dne 21.3.2011 22:06, bacula-users-request AT lists.sourceforge DOT net
>> napsal(a):
>>> Send Bacula-users mailing list submissions to
>>> bacula-users AT lists.sourceforge DOT net
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>> or, via email, send a message with subject or body 'help' to
>>> bacula-users-request AT lists.sourceforge DOT net
>>>
>>> You can reach the person managing the list at
>>> bacula-users-owner AT lists.sourceforge DOT net
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Bacula-users digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Bacula 5.0.2 - Action On Purge = Truncate not working
>>> (Ondrej PLANKA (Ignum profile))
>>> 2. excluding files? (Mike Eggleston)
>>> 3. Troubles with bacula 2.4.4 (Wouter Verhelst)
>>> 4. Re: Troubles with bacula 2.4.4 (Dan Langille)
>>> 5. File Set configuration and excluding some files? (Mike Eggleston)
>>> 6. Re: 3 (korvus AT comcast DOT net)
>>> 7. Re: Bacula Director and SD crashing (Carlson, Mike)
>>> 8. Re: File Set configuration and excluding some files?
>>> (Mark Nienberg)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Sun, 20 Mar 2011 15:31:19 +0100
>>> From: "Ondrej PLANKA (Ignum profile)"<ondrej.planka AT ignum DOT cz>
>>> Subject: [Bacula-users] Bacula 5.0.2 - Action On Purge = Truncate not
>>>
>>> working
>>> To: "bacula-users AT lists.sourceforge DOT net"
>>> <bacula-users AT lists.sourceforge DOT net>
>>> Message-ID:<4D860FB7.6020907 AT ignum DOT cz>
>>> Content-Type: text/plain; charset="iso-8859-2"
>>>
>>> Ciao!
>>>
>>> guys do you have experience with new feature "Truncate Volume after
>>> Purge "
>>> (http://bacula.org/5.0.x-manuals/en/main/main/New_Features_in_5_0_1.html#SECTION00410000000000000000)?
>>>
>>> I tried to set up/applied manually from console (purge volume action=all
>>> allpools storage=File) under Bacula 5.0.2 but not working.
>>>
>>> Volume (disk based) has status purged and set up ActionOnPurge=Truncate
>>> but after console command: purge volume action=all allpools storage=File/
>>> /I got this message:
>>> /This command can be DANGEROUS!!!
>>>
>>> It purges (deletes) all Files from a Job,
>>> JobId, Client or Volume; or it purges (deletes)
>>> all Jobs from a Client or Volume without regard
>>> to retention periods. Normally you should use the
>>> PRUNE command, which respects retention periods.
>>> No volume founds to perform all action(s)/
>>>
>>> although volume should be candidate for truncate out of SAN storage (the
>>> SD should set the file to 0 bytes on disk) no action were performed.
>>>
>>> Any hints/ideas what is wrong?
>>>
>>> Thanks, Ondrey.
>>>
>>>
>>>
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Sun, 20 Mar 2011 11:59:38 -0500
>>> From: Mike Eggleston<mikeegg1 AT mac DOT com>
>>> Subject: [Bacula-users] excluding files?
>>> To: bacula-users AT lists.sourceforge DOT net
>>> Message-ID:<20110320165937.GA14958 AT mail.mac DOT com>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> Morning,
>>>
>>> I'm trying to get some spam email folders not included in the backup. I
>>> wonder if there is an ordering issue I'm missing. For this unix server
>>> I have a:
>>>
>>> File Set {
>>> Name = Unix-Files-elo
>>> Ignore Fileset Changes = yes
>>> Include {
>>> Options {
>>> signature = MD5
>>> Exclude = yes
>>> IgnoreCase = yes
>>>
>>> # Exclude files and directories that can be
>>> replaced using the red hat network
>>> WildDir = "/var/spool/imap/*/user/*/spam"
>>> }
>>> File = /var/spool/imap
>>> }
>>> }
>>>
>>> I'm wondering if the included directory at the bottom, '/var/spool/imap',
>>> is overriding the WildDir in the Options/Exclude=yes section above.
>>>
>>> I do want to backup the email, but I don't want to backup any spam
>>> messages.
>>>
>>> If there is a order of specification problem in how I've written the
>>> backup, is there a better way to write this section? Is it possible to
>>> have two Options/Exclude sections?
>>>
>>> Mike
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 3
>>> Date: Sun, 20 Mar 2011 23:52:21 +0100
>>> From: Wouter Verhelst<wouter AT nixsys DOT be>
>>> Subject: [Bacula-users] Troubles with bacula 2.4.4
>>> To: bacula-users AT lists.sourceforge DOT net
>>> Message-ID:<20110320225221.GC13420 AT celtic.nixsys DOT be>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> Hi,
>>>
>>> At a customer, we'd been running bacula since quite some time. This is
>>> now running on a Debian 'lenny' that originally was an etch installation
>>> (with 1.38.11) and has since been upgraded. We will probably upgrade
>>> once more in some time to squeeze (with 5.0.2), but no concrete plans
>>> exist for this. It's running against a PostgreSQL 8.3 database (also the
>>> standard version in debian lenny).
>>>
>>> Originally, bacula ran pretty smoothly. But in recent times, mainly due
>>> to the volumes having gone through the roof, things don't run as
>>> smoothly anymore.
>>>
>>> I understand that 2.4.4 is probably not under development anymore, and
>>> that it's likely that none of this is going to be fixed for this branch.
>>> But if these issues have been fixed long ago, I'd appreciate if people
>>> could tell me, so I know.
>>>
>>> With the original installation, the amount of data that was added and
>>> then removed again on a weekly basis (we have weekly full backups) was
>>> quite detrimental to postgresql's autovacuum feature, to the extent that
>>> it wouldn't work anymore. That is, the amount of data that had been
>>> removed from the table would be so large that the amount of disk space
>>> to be released would be over a particular percentage, which triggered a
>>> sanity check in the autovacuum daemon, causing it to stop doing the
>>> autovacuum. As a result, the database files would balloon in size,
>>> eventually taking up 70G of data (when a dump of the database was just a
>>> few hundred megs). I fixed this by adding an explicit 'vacuumdb -f
>>> bacula' to the 'delete_catalog_backup' script.
>>>
>>> I had however failed to disable autovacuuming, and with the backup
>>> now requiring 3 LTO3 tapes and over 48 hours, eventually the autovacuum
>>> daemon started interfering; when it kicks in, it causes a database-level
>>> lock, which would sometimes cause the backup to fail in the following
>>> manner:
>>>
>>> 06-feb 10:09 belessnas-dir JobId 4241: Fatal error: sql.c:249 sql.c:249
>>> query SELECT count(*) from JobMedia WHERE JobId=4241 failed:
>>> server closed the connection unexpectedly
>>> This probably means the server terminated abnormally
>>> before or while processing the request.
>>>
>>> (with this in the postgres log at around the same time:)
>>>
>>> 2011-02-06 10:09:19 CET LOG: autovacuum launcher started
>>> 2011-02-06 10:09:19 CET LOG: database system is ready to accept connections
>>>
>>> I guess what I'm saying with all this is that it might be nice if bacula
>>> were to play a bit more nicely with postgresql's vacuuming process,
>>> which is fairly essential for it to function nicely.
>>>
>>> That was last february; backups have since been running, sometimes
>>> okayish, sometimes not (there's also the matter of the tape robot
>>> sometimes having issues, but this is hardly bacula's fault).
>>>
>>> Today, then, bacula failed with the following message:
>>>
>>> 20-mrt 22:02 belessnas-dir JobId 4365: Fatal error: Can't fill File table
>>> Query failed: INSERT INTO File (FileIndex, JobId, PathId, FilenameId,
>>> LStat, MD5)SELECT batch.FileIndex, batch.JobId, Path.PathId,
>>> Filename.FilenameId,batch.LStat, batch.MD5 FROM batch JOIN Path ON
>>> (batch.Path = Path.Path) JOIN Filename ON (batch.Name = Filename.Name):
>>> ERR=ERROR: integer out of range
>>>
>>> This was accurate:
>>>
>>> bacula=# SELECT last_value from file_fileid_seq;
>>> last_value
>>> ------------
>>> 2147483652
>>> (1 row)
>>>
>>> Yes, we've been running it for several years now, and apparently we've
>>> written over 2 billion files to tape. I've ran an 'ALTER TABLE File ALTER
>>> fileid TYPE bigint' to change the fileid field into a 64 bit, rather
>>> than a 32 bit, variable, which should fix this for the forseeable
>>> future; however, I have a few questions:
>>> - Is it okay for me to change the data type of the 'fileid' column like
>>> that? Note that I've also changed it in other tables which have a
>>> 'fileid' column. If bacula doesn't internally use the fileid number in
>>> a 32 bit integer, then that shouldn't be a huge problem, but I don't
>>> know whether it does.
>>> - Since things haven't really been running smoothly here, every time the
>>> backup fails, the customer gets less happy with bacula. Are there any
>>> other people here who run bacula to write fairly large volumes of data
>>> to tape, and can they give me some pointers on things to avoid? That
>>> way, I could hopefully avoid common pitfalls before I run into them.
>>> Obviously if there is some documentation on this somewhere that I
>>> missed, a simple pointer would be nice.
>>> - Finally, I realize that many of these issues may be fixed in a more
>>> recent version of bacula, but I have no way to be sure -- this
>>> particular customer is the only place were I have bacula running with
>>> such large data volumes, and obviously just upgrading a particularly
>>> important server without coordination and only a vague idea that it
>>> *might* improve things isn't very well an option. However, if someone
>>> could authoritatively tell me that these issues have been fixed in a
>>> more recent version, then an upgrade would probably be a very good
>>> idea...
>>>
>>> Thanks,
>>>
>>> --
>>> Wouter Verhelst
>>> NixSys BVBA
>>> Louizastraat 14, 2800 Mechelen
>>> T: +32 15 27 69 50 / F: +32 15 27 69 51 / M: +32 486 836 198
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 4
>>> Date: Sun, 20 Mar 2011 19:41:21 -0400
>>> From: Dan Langille<dan AT langille DOT org>
>>> Subject: Re: [Bacula-users] Troubles with bacula 2.4.4
>>> To: Wouter Verhelst<wouter AT nixsys DOT be>
>>> Cc: bacula-users AT lists.sourceforge DOT net
>>> Message-ID:<4D8690A1.4040707 AT langille DOT org>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> On 3/20/2011 6:52 PM, Wouter Verhelst wrote:
>>>> Hi,
>>>>
>>>> At a customer, we'd been running bacula since quite some time. This is
>>>> now running on a Debian 'lenny' that originally was an etch installation
>>>> (with 1.38.11) and has since been upgraded. We will probably upgrade
>>>> once more in some time to squeeze (with 5.0.2), but no concrete plans
>>>> exist for this. It's running against a PostgreSQL 8.3 database (also the
>>>> standard version in debian lenny).
>>>>
>>>> Originally, bacula ran pretty smoothly. But in recent times, mainly due
>>>> to the volumes having gone through the roof, things don't run as
>>>> smoothly anymore.
>>>>
>>>> I understand that 2.4.4 is probably not under development anymore, and
>>>> that it's likely that none of this is going to be fixed for this branch..
>>>> But if these issues have been fixed long ago, I'd appreciate if people
>>>> could tell me, so I know.
>>>>
>>>> With the original installation, the amount of data that was added and
>>>> then removed again on a weekly basis (we have weekly full backups) was
>>>> quite detrimental to postgresql's autovacuum feature, to the extent that
>>>> it wouldn't work anymore. That is, the amount of data that had been
>>>> removed from the table would be so large that the amount of disk space
>>>> to be released would be over a particular percentage, which triggered a
>>>> sanity check in the autovacuum daemon, causing it to stop doing the
>>>> autovacuum. As a result, the database files would balloon in size,
>>>> eventually taking up 70G of data (when a dump of the database was just a
>>>> few hundred megs). I fixed this by adding an explicit 'vacuumdb -f
>>>> bacula' to the 'delete_catalog_backup' script.
>>>>
>>>> I had however failed to disable autovacuuming, and with the backup
>>>> now requiring 3 LTO3 tapes and over 48 hours, eventually the autovacuum
>>>> daemon started interfering; when it kicks in, it causes a database-level
>>>> lock, which would sometimes cause the backup to fail in the following
>>>> manner:
>>>>
>>>> 06-feb 10:09 belessnas-dir JobId 4241: Fatal error: sql.c:249 sql.c:249
>>>> query SELECT count(*) from JobMedia WHERE JobId=4241 failed:
>>>> server closed the connection unexpectedly
>>>> This probably means the server terminated abnormally
>>>> before or while processing the request.
>>>>
>>>> (with this in the postgres log at around the same time:)
>>>>
>>>> 2011-02-06 10:09:19 CET LOG: autovacuum launcher started
>>>> 2011-02-06 10:09:19 CET LOG: database system is ready to accept
>>>> connections
>>>>
>>>> I guess what I'm saying with all this is that it might be nice if bacula
>>>> were to play a bit more nicely with postgresql's vacuuming process,
>>>> which is fairly essential for it to function nicely.
>>>>
>>>> That was last february; backups have since been running, sometimes
>>>> okayish, sometimes not (there's also the matter of the tape robot
>>>> sometimes having issues, but this is hardly bacula's fault).
>>>>
>>>> Today, then, bacula failed with the following message:
>>>>
>>>> 20-mrt 22:02 belessnas-dir JobId 4365: Fatal error: Can't fill File table
>>>> Query failed: INSERT INTO File (FileIndex, JobId, PathId, FilenameId,
>>>> LStat, MD5)SELECT batch.FileIndex, batch.JobId, Path.PathId,
>>>> Filename.FilenameId,batch.LStat, batch.MD5 FROM batch JOIN Path ON
>>>> (batch.Path = Path.Path) JOIN Filename ON (batch.Name = Filename.Name):
>>>> ERR=ERROR: integer out of range
>>>>
>>>> This was accurate:
>>>>
>>>> bacula=# SELECT last_value from file_fileid_seq;
>>>> last_value
>>>> ------------
>>>> 2147483652
>>>> (1 row)
>>>>
>>>> Yes, we've been running it for several years now, and apparently we've
>>>> written over 2 billion files to tape. I've ran an 'ALTER TABLE File ALTER
>>>> fileid TYPE bigint' to change the fileid field into a 64 bit, rather
>>>> than a 32 bit, variable, which should fix this for the forseeable
>>>> future; however, I have a few questions:
>>>> - Is it okay for me to change the data type of the 'fileid' column like
>>>> that? Note that I've also changed it in other tables which have a
>>>> 'fileid' column. If bacula doesn't internally use the fileid number
>>>> in
>>>> a 32 bit integer, then that shouldn't be a huge problem, but I don't
>>>> know whether it does.
>>> Yes, I think you're fine.
>>>
>>>> - Since things haven't really been running smoothly here, every time the
>>>> backup fails, the customer gets less happy with bacula. Are there any
>>>> other people here who run bacula to write fairly large volumes of
>>>> data
>>>> to tape, and can they give me some pointers on things to avoid? That
>>>> way, I could hopefully avoid common pitfalls before I run into them.
>>>> Obviously if there is some documentation on this somewhere that I
>>>> missed, a simple pointer would be nice.
>>>> - Finally, I realize that many of these issues may be fixed in a more
>>>> recent version of bacula, but I have no way to be sure -- this
>>>> particular customer is the only place were I have bacula running with
>>>> such large data volumes, and obviously just upgrading a particularly
>>>> important server without coordination and only a vague idea that it
>>>> *might* improve things isn't very well an option. However, if someone
>>>> could authoritatively tell me that these issues have been fixed in a
>>>> more recent version, then an upgrade would probably be a very good
>>>> idea...
>>> Others will report in on your other questions.
>>>
>>> --
>>> Dan Langille - http://langille.org/
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 5
>>> Date: Sun, 20 Mar 2011 19:52:06 -0500
>>> From: Mike Eggleston<mikeegg1 AT mac DOT com>
>>> Subject: [Bacula-users] File Set configuration and excluding some
>>> files?
>>> To: bacula-users AT lists.sourceforge DOT net
>>> Message-ID:<20110321005206.GA6645 AT mail.mac DOT com>
>>> Content-Type: text/plain; charset=us-ascii
>>>
>>> Afternoon,
>>>
>>> I'm trying to get some spam email folders excluded in the backup. I
>>> wonder if there is an ordering issue I'm missing. For this unix server
>>> I have a:
>>>
>>> File Set {
>>> Name = Unix-Files-elo
>>> Ignore Fileset Changes = yes
>>> Include {
>>> Options {
>>> signature = MD5
>>> Exclude = yes
>>> IgnoreCase = yes
>>>
>>> # Exclude files and directories that can be
>>> replaced using the red hat network
>>> WildDir = "/var/spool/imap/*/user/*/spam"
>>> }
>>> File = /var/spool/imap
>>> }
>>> }
>>>
>>> I'm wondering if the included directory at the bottom, '/var/spool/imap',
>>> is overriding the WildDir in the Options/Exclude=yes section above.
>>>
>>> I do want to backup the email, but I don't want to backup any spam
>>> messages.
>>>
>>> If there is a order of specification problem in how I've written the
>>> backup, is there a better way to write this section? Is it possible to
>>> have two Options/Exclude sections?
>>>
>>> Mike
>>>
>>> baclula 2.0.3
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 6
>>> Date: Mon, 21 Mar 2011 13:48:15 +0000 (UTC)
>>> From: korvus AT comcast DOT net
>>> Subject: Re: [Bacula-users] 3
>>> To: owner-freebsd-hardware AT freebsd DOT org,
>>> owner-freebsd-ports AT freebsd DOT org, bacula-users AT
>>> lists.sourceforge DOT net,
>>> bacula-devel AT lists.sourceforge DOT net, owner-freebsd-stable
>>> AT freebsd DOT org,
>>> owner-feebsd-questions AT freebsd DOT org
>>> Message-ID:
>>> <2107761778.1350649.1300715295583.JavaMail.root AT
>>> sz0140a.westchester..pa.mail.comcast DOT net>
>>>
>>> Content-Type: text/plain; charset=utf-8
>>>
>>> I?m so happy! My new site is wonderful! I hope you?ll like it too...
>>> http://www.zygier.net/page.php?ugoogleId=49tj7
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 7
>>> Date: Mon, 21 Mar 2011 11:55:49 -0700
>>> From: "Carlson, Mike"<carlson39 AT llnl DOT gov>
>>> Subject: Re: [Bacula-users] Bacula Director and SD crashing
>>> To: Dan Langille<dan AT langille DOT org>
>>> Cc: "bacula-users AT lists.sourceforge DOT net"
>>> <bacula-users AT lists.sourceforge DOT net>
>>> Message-ID:
>>> <EF83999E0886C848A62369D2AC8588F1010921ECFB76 AT
>>> NSPEXMBX-A.the-lab.llnl DOT gov>
>>>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> I was not able to get a successful btraceback message after the last
>>> director crash, I simply got a blank email. I'll take a look at the
>>> btraceback script again.
>>>
>>> With that, the bacula-dir process died at 8:40am, and at 8:30am I had about
>>> 105 VirtualFull jobs scheduled. My hypothesis is that the server, which is
>>> the DB server, ZFS filesystem, and Director all in one, cannot handle that
>>> load.
>>>
>>> Thankfully, a new server has been ordered so we can split off the storage
>>> node on its own, and the new server will just be the director and
>>> postgresql database. Until then, I've change the schedule to do a
>>> traditional Full and not a Virtual Full so I can at least maintain a decent
>>> service uptime. If anyone has some advice for running that many virtual
>>> fulls, I'd love to hear their experiences.
>>>
>>> 500 is a lot. If I set the Dir to only have 500 maximum concurrent jobs,
>>> but we have lets say, 2000 clients scheduled, will they just wait in the
>>> queue, or will they never run at all?
>>>
>>> Again, I appreciate the responses.
>>>
>>> Mike C
>>> ________________________________________
>>> From: Dan Langille [dan AT langille DOT org]
>>> Sent: Saturday, March 19, 2011 4:34 PM
>>> To: Carlson, Mike
>>> Cc: bacula-users AT lists.sourceforge DOT net
>>> Subject: Re: [Bacula-users] Bacula Director and SD crashing
>>>
>>> On 3/13/2011 5:07 PM, Carlson, Mike wrote:
>>>> Upfront details of the environemnt:
>>>>
>>>> # uname -a
>>>> FreeBSD write.llnl.gov 8.2-RELEASE FreeBSD 8.2-RELEASE #3: Tue Mar 8
>>>> 11:32:05 PST 2011 root AT write.llnl DOT
>>>> gov:/usr/obj/usr/src/sys/GENERIC amd64
>>>>
>>>> # pkg_info |grep bacula
>>>> bacula-client-5.0.3 The network backup solution (client)
>>>> bacula-server-5.0.3 The network backup solution (server)
>>>> # pkg_info | grep postgresql
>>>> postgresql-client-9.0.3 PostgreSQL database (client)
>>>> postgresql-contrib-9.0.3 The contrib utilities from the PostgreSQL
>>>> distribution
>>>> postgresql-server-9.0.3 The most advanced open-source database available
>>>> anywhere
>>> Any progress?
>>>
>>>
>>> BTW 500 concurrent jobs seems like a lot. :)
>>>
>>>> # kldstat
>>>> Id Refs Address Size Name
>>>> 1 15 0xffffffff80100000 c9fe20 kernel
>>>> 2 1 0xffffffff80da0000 f080 aio.ko
>>>> 3 1 0xffffffff80db0000 16ea40 ispfw.ko
>>>> 4 1 0xffffffff80f20000 4fc8 geom_multipath.ko
>>>> 5 1 0xffffffff81022000 104a7a zfs.ko
>>>> 6 1 0xffffffff81127000 1a22 opensolaris.ko
>>>>
>>>> System memory summary
>>>> Total real memory available: 12264 MB
>>>> Logically used memory: 7605 MB
>>>> Logically available memory: 4659 MB
>>>>
>>>> The Bacula director, database, and storage node are all currently on one
>>>> server.
>>>>
>>>> The problem:
>>>>
>>>> I have a farily large Bacula environment (~1000 clients and servers), and
>>>> there are plans to keep expanding our clients (and storage, and the
>>>> database, etc...). I believe my lack of fully understanding the storage
>>>> and director resources are at the root of it, because it appears that the
>>>> director is running more jobs that the storage node can handle, however,
>>>> it is not failing gracefully. Also, when the VirtualFull job is ran (a LOT
>>>> run at once, between 90 and 100), it almost immediately crashes the SD.
>>>>
>>>> As far as job scheduling, I have 10 timeslots setup, from 7pm to 4am, each
>>>> start time has around 90 - 110 clients.
>>>>
>>>> I have the director configured for 1000 concurrent jobs:
>>>>
>>>> Director {
>>>> Name = write.llnl.gov-dir
>>>> DIRport = 9101
>>>> QueryFile = "/usr/local/share/bacula/query.sql"
>>>> WorkingDirectory = "/var/db/bacula"
>>>> PidDirectory = "/var/run"
>>>> Maximum Concurrent Jobs = 1000
>>>> Password = "XXXXXXXXXXXXXXXXXXXXXXXXXX"
>>>> Messages = Daemon
>>>> }
>>>>
>>>> I have a few different storage node (all File storage, no tapes here), but
>>>> the primary one that is responsible for 900 clients is configured for 1000
>>>> concurrent jobs, and 1000 separate devices.
>>>>
>>>> Storage {
>>>> Name = write.llnl.gov-sd
>>>> SDAddress = write.llnl.gov
>>>> SDPort = 9103
>>>> WorkingDirectory = "/var/db/bacula"
>>>> Pid Directory = "/var/run"
>>>> Maximum Concurrent Jobs = 1001
>>>> }
>>>> Device {
>>>> Name = FileStorageD1
>>>> Media Type = File
>>>> Archive Device = /filevol001/drive01
>>>> LabelMedia = yes;
>>>> Random Access = Yes;
>>>> AutomaticMount = yes;
>>>> RemovableMedia = no;
>>>> AlwaysOpen = no;
>>>> Maximum Concurrent Jobs = 4
>>>> Maximum File Size = 5G
>>>> Maximum Spool Size = 370g
>>>> Maximum Job Spool Size = 20g
>>>> Maximum Block Size = 262144
>>>> Maximum Network Buffer Size = 262144
>>>> }
>>>> Device {
>>>> Name = VF_FileStorageD1
>>>> Media Type = File
>>>> Archive Device = /filevol001/VirtualFull/vf_drive01
>>>> LabelMedia = yes;
>>>> Random Access = Yes;
>>>> AutomaticMount = yes;
>>>> RemovableMedia = no;
>>>> AlwaysOpen = no;
>>>> Maximum Concurrent Jobs = 4
>>>> Maximum File Size = 5G
>>>> Maximum Spool Size = 370g
>>>> Maximum Job Spool Size = 20g
>>>> Maximum Block Size = 262144
>>>> Maximum Network Buffer Size = 262144
>>>> }
>>>>
>>>> There are 500 FileStorageD### and 500 VF_FileStorageD###, all have the
>>>> same above settings.
>>>>
>>>> 500 are for Incremental/Full jobs, the other 500 are for Virtual Fulls.
>>>> Since I hit the 500 or so client mark, I've started to see this error
>>>> message pop up in the logs::
>>>> 13-Mar 09:31 W1.llnl.gov JobId 15841: Fatal error:
>>>> /tmp/bacula/bacula/src/filed/backup.c:1019 Network send error to SD.
>>>> ERR=Input/output error
>>>>
>>>> And then the sd deamon will crash.
>>>> (no debugging symbols found)...ptrace: Operation not permitted.
>>>> /var/db/bacula/1275: No such file or directory.
>>>> $1 = 0
>>>> /usr/local/share/bacula/btraceback.gdb:2: Error in sourced command file:
>>>> No symbol table is loaded. Use the "file" command.
>>>>
>>>> And the director follows shortly afterwards:
>>>> [New Thread 81ad04d40 (LWP 102349)]
>>>> [New Thread 81e889500 (LWP 102334)]
>>>> [New Thread 822935280 (LWP 102228)]
>>>> [New Thread 82649b540 (LWP 102039)]
>>>> [New Thread 821819140 (LWP 101905)]
>>>> [New Thread 827372f40 (LWP 103458)]
>>>> [New Thread 819963f00 (LWP 103457)]
>>>> [New Thread 8267c2840 (LWP 103456)]
>>>> [New Thread 8276135c0 (LWP 103454)]
>>>> [New Thread 8136d1180 (LWP 103453)]
>>>> [New Thread 8170ab540 (LWP 103452)]
>>>> [New Thread 81b446c80 (LWP 103451)]
>>>> [New Thread 81ca45f00 (LWP 103450)]
>>>> [New Thread 825b7dfc0 (LWP 103449)]
>>>> [New Thread 80533b8c0 (LWP 103442)]
>>>> [New Thread 81de9af40 (LWP 103441)]
>>>> [New Thread 811f4ad40 (LWP 103440)]
>>>> [New Thread 814ed6c40 (LWP 103436)]
>>>> [New Thread 824954a40 (LWP 103434)]
>>>> [New Thread 8226ff340 (LWP 103432)]
>>>> [New Thread 8268a16c0 (LWP 103431)]
>>>> [New Thread 81d7c15c0 (LWP 103430)]
>>>> [New Thread 81a0c1900 (LWP 103426)]
>>>> [New Thread 8245ec840 (LWP 103423)]
>>>> [New Thread 81d2ac540 (LWP 103421)]
>>>> [New Thread 81eea00c0 (LWP 102284)]
>>>> [New Thread 822beba80 (LWP 103483)]
>>>> [New Thread 822b17a80 (LWP 101351)]
>>>> [New Thread 81fa81a00 (LWP 103464)]
>>>> [New Thread 8249d1c00 (LWP 103463)]
>>>> [New Thread 81d7165c0 (LWP 103461)]
>>>> [New Thread 8111e1880 (LWP 103460)]
>>>> [New Thread 80a249700 (LWP 103459)]
>>>> [New Thread 81a274340 (LWP 103361)]
>>>> [New Thread 823ea4780 (LWP 103359)]
>>>> [New Thread 825d4a100 (LWP 103357)]
>>>> [New Thread 82565ac40 (LWP 103355)]
>>>> [New Thread 8070e8900 (LWP 103354)]
>>>> [New Thread 8045a9380 (LWP 103351)]
>>>> [New Thread 826d12300 (LWP 103350)]
>>>> [New Thread 821d4cf00 (LWP 103349)]
>>>> [New Thread 805175b40 (LWP 103348)]
>>>> [New Thread 819245a80 (LWP 103343)]
>>>> [New Thread 826298140 (LWP 103341)]
>>>> [New Thread 827073dc0 (LWP 103339)]
>>>> [New Thread 822f028c0 (LWP 103337)]
>>>> [New Thread 8238ae0c0 (LWP 103336)]
>>>> [New Thread 8077cfe00 (LWP 103333)]
>>>> [New Thread 82306d8c0 (LWP 103332)]
>>>> [New Thread 826a1bb80 (LWP 103455)]
>>>> [New Thread 8063e7e40 (LWP 103206)]
>>>> [New Thread 827005f80 (LWP 103204)]
>>>> [New Thread 8190a7a80 (LWP 103205)]
>>>> [New Thread 8093d7840 (LWP 103445)]
>>>> [New Thread 821818f80 (LWP 103202)]
>>>> [New Thread 821818dc0 (LWP 103443)]
>>>> [New Thread 827005dc0 (LWP 103329)]
>>>> [New Thread 8172d1740 (LWP 103195)]
>>>> [New Thread 822d25d80 (LWP 103192)]
>>>> [New Thread 811efcb80 (LWP 103193)]
>>>> [New Thread 8267c2680 (LWP 103328)]
>>>> [New Thread 8249d1a40 (LWP 103435)]
>>>> [New Thread 810c33700 (LWP 103433)]
>>>> [New Thread 806788380 (LWP 103306)]
>>>> [New Thread 824bf08c0 (LWP 103155)]
>>>> [New Thread 8269c7500 (LWP 103158)]
>>>> [New Thread 824e81b40 (LWP 103427)]
>>>> [New Thread 80bdcb480 (LWP 103425)]
>>>> [New Thread 822aa5e00 (LWP 103422)]
>>>> [New Thread 826a1b9c0 (LWP 103134)]
>>>> [New Thread 826d85140 (LWP 100835)]
>>>> [New Thread 824954340 (LWP 103326)]
>>>> [New Thread 822a04fc0 (LWP 103360)]
>>>> [New Thread 81d845e00 (LWP 103358)]
>>>> [New Thread 8268a1500 (LWP 103356)]
>>>> [New Thread 8249d1880 (LWP 103115)]
>>>> [New Thread 8245ec680 (LWP 103352)]
>>>> [New Thread 81a98e8c0 (LWP 103107)]
>>>> [New Thread 805b15400 (LWP 103098)]
>>>> [New Thread 8268a1340 (LWP 103100)]
>>>> [New Thread 82565aa80 (LWP 103344)]
>>>> [New Thread 824e81980 (LWP 103342)]
>>>> [New Thread 81057e740 (LWP 103324)]
>>>> [New Thread 813c52b80 (LWP 103340)]
>>>> [New Thread 8071288c0 (LWP 103322)]
>>>> [New Thread 806f66c80 (LWP 103321)]
>>>> [New Thread 82565a8c0 (LWP 103338)]
>>>> [New Thread 81ec18600 (LWP 103069)]
>>>> [New Thread 81a205500 (LWP 103334)]
>>>> [New Thread 8213faf00 (LWP 103318)]
>>>> [New Thread 8042ab900 (LWP 103317)]
>>>> [New Thread 815bec5c0 (LWP 103067)]
>>>> [New Thread 822efb8c0 (LWP 103330)]
>>>> [New Thread 81ad04b80 (LWP 103314)]
>>>> [New Thread 8172f0580 (LWP 103312)]
>>>> [New Thread 810c338c0 (LWP 103310)]
>>>> [New Thread 826a1b800 (LWP 103308)]
>>>> [New Thread 81ad049c0 (LWP 102273)]
>>>> [New Thread 82167c380 (LWP 103327)]
>>>> [New Thread 8255a4340 (LWP 103304)]
>>>> [New Thread 81a1bd3c0 (LWP 103302)]
>>>> [New Thread 81b65d500 (LWP 103299)]
>>>> [New Thread 81b98f8c0 (LWP 103297)]
>>>> [New Thread 80fb3ffc0 (LWP 103295)]
>>>> [New Thread 825d43100 (LWP 103294)]
>>>> [New Thread 81d7c1240 (LWP 103293)]
>>>> [New Thread 811f4b0c0 (LWP 103289)]
>>>> [New Thread 806f66e40 (LWP 103279)]
>>>> [New Thread 80a595c80 (LWP 102852)]
>>>> [New Thread 815d781c0 (LWP 102849)]
>>>> [New Thread 808171a80 (LWP 101739)]
>>>> [New Thread 81e5be880 (LWP 100915)]
>>>> [New Thread 81a48a0c0 (LWP 102264)]
>>>> [New Thread 821818c00 (LWP 103325)]
>>>> [New Thread 8081718c0 (LWP 102257)]
>>>> [New Thread 81cd26e00 (LWP 102237)]
>>>> [New Thread 817d25540 (LWP 103323)]
>>>> [New Thread 826a1b640 (LWP 103030)]
>>>> [New Thread 8268a1180 (LWP 103319)]
>>>> [New Thread 81dadf7c0 (LWP 103027)]
>>>> [New Thread 82177a1c0 (LWP 103315)]
>>>> [New Thread 806045280 (LWP 103313)]
>>>> [New Thread 825b7de00 (LWP 103311)]
>>>> [New Thread 824e817c0 (LWP 103309)]
>>>> [New Thread 823ea45c0 (LWP 103307)]
>>>> [New Thread 820ffa180 (LWP 103305)]
>>>> [New Thread 814705b00 (LWP 103303)]
>>>> [New Thread 824954880 (LWP 102993)]
>>>> [New Thread 82565a700 (LWP 103300)]
>>>> [New Thread 823655180 (LWP 103298)]
>>>> [New Thread 8228a2280 (LWP 103296)]
>>>> [New Thread 8093d0140 (LWP 102982)]
>>>> [New Thread 826297f80 (LWP 102988)]
>>>> [New Thread 825d49f40 (LWP 103290)]
>>>> [New Thread 824e81600 (LWP 103288)]
>>>> [New Thread 825f7d800 (LWP 103217)]
>>>> [New Thread 80778ba80 (LWP 102851)]
>>>> [New Thread 80d14f380 (LWP 102280)]
>>>> [New Thread 825f7d640 (LWP 102231)]
>>>> [New Thread 811f4af00 (LWP 102850)]
>>>> [New Thread 81016f700 (LWP 102405)]
>>>> [New Thread 819441580 (LWP 101184)]
>>>> [New Thread 8097d7dc0 (LWP 100697)]
>>>> [New Thread 824e81440 (LWP 102239)]
>>>> [New Thread 81eec8280 (LWP 101974)]
>>>> [New Thread 819963d40 (LWP 103287)]
>>>> [New Thread 8213eeb80 (LWP 101543)]
>>>> [New Thread 8252924c0 (LWP 100926)]
>>>> [New Thread 80951f2c0 (LWP 103283)]
>>>> [New Thread 808fdd500 (LWP 103281)]
>>>> [New Thread 8093c9140 (LWP 102396)]
>>>> [New Thread 81f582c80 (LWP 103286)]
>>>> [New Thread 8226c6fc0 (LWP 103284)]
>>>> [New Thread 824e81280 (LWP 103285)]
>>>> [New Thread 8255a4180 (LWP 102222)]
>>>> [New Thread 81afc3980 (LWP 103277)]
>>>> [New Thread 804e85900 (LWP 103282)]
>>>> [New Thread 820457f00 (LWP 102326)]
>>>> [New Thread 8245ec4c0 (LWP 103280)]
>>>> [New Thread 815242f40 (LWP 103278)]
>>>> [New Thread 81f8f6fc0 (LWP 102371)]
>>>> [New Thread 809517f40 (LWP 102360)]
>>>> [New Thread 825292300 (LWP 103276)]
>>>> [New Thread 825139800 (LWP 103270)]
>>>> [New Thread 820d8a180 (LWP 103268)]
>>>> [New Thread 804010c80 (LWP 103267)]
>>>> [New Thread 821fe6140 (LWP 103264)]
>>>> [New Thread 8109e9c40 (LWP 103263)]
>>>> [New Thread 824e5e440 (LWP 103262)]
>>>> [New Thread 819441040 (LWP 103261)]
>>>> [New Thread 81f1aa180 (LWP 103255)]
>>>> [New Thread 82508a800 (LWP 103254)]
>>>> [New Thread 821811340 (LWP 103251)]
>>>> [New Thread 80712bb80 (LWP 102809)]
>>>> [New Thread 825117640 (LWP 103248)]
>>>> [New Thread 820b72080 (LWP 103247)]
>>>> [New Thread 825405340 (LWP 103246)]
>>>> [New Thread 821e12300 (LWP 103241)]
>>>> [New Thread 81fde3880 (LWP 103239)]
>>>> [New Thread 8248b7500 (LWP 103237)]
>>>> [New Thread 80e76b640 (LWP 103236)]
>>>> [New Thread 81c8c7e00 (LWP 103235)]
>>>> [New Thread 8052f61c0 (LWP 103234)]
>>>> [New Thread 8212f0d40 (LWP 103233)]
>>>> [New Thread 81d4300c0 (LWP 103232)]
>>>> [New Thread 8181023c0 (LWP 103231)]
>>>> [New Thread 8212e8d40 (LWP 102167)]
>>>> [New Thread 8220a3fc0 (LWP 101913)]
>>>> [New Thread 8251d19c0 (LWP 102974)]
>>>> [New Thread 824bf0700 (LWP 102218)]
>>>> [New Thread 824d36700 (LWP 103275)]
>>>> [New Thread 80b9c3fc0 (LWP 103274)]
>>>> [New Thread 81c008f40 (LWP 103271)]
>>>> [New Thread 820a10080 (LWP 102971)]
>>>> [New Thread 80951f100 (LWP 103272)]
>>>> [New Thread 80951ef40 (LWP 102376)]
>>>> [New Thread 803f39540 (LWP 102212)]
>>>> [New Thread 8210e6a80 (LWP 103226)]
>>>> [New Thread 824592840 (LWP 103225)]
>>>> [New Thread 804b324c0 (LWP 103219)]
>>>> [New Thread 824d2c1c0 (LWP 103201)]
>>>> [New Thread 810324fc0 (LWP 103198)]
>>>> [New Thread 81e04ac00 (LWP 103190)]
>>>> [New Thread 809498100 (LWP 103189)]
>>>> [New Thread 8093cfdc0 (LWP 103151)]
>>>> [New Thread 8244a4a00 (LWP 103059)]
>>>> [New Thread 81be1cfc0 (LWP 103029)]
>>>> [New Thread 804bfc940 (LWP 102985)]
>>>> [New Thread 80458fe40 (LWP 102966)]
>>>> [New Thread 81ffe86c0 (LWP 102953)]
>>>> [New Thread 824d36540 (LWP 103269)]
>>>> [New Thread 8251d1800 (LWP 102958)]
>>>> [New Thread 8251d1640 (LWP 103265)]
>>>> [New Thread 80fe0a040 (LWP 103257)]
>>>> [New Thread 81e9ab500 (LWP 102905)]
>>>> [New Thread 820f1c340 (LWP 102956)]
>>>> [New Thread 8249546c0 (LWP 102926)]
>>>> [New Thread 824d36380 (LWP 103256)]
>>>> [New Thread 823ea4400 (LWP 102901)]
>>>> [New Thread 825139640 (LWP 103252)]
>>>> [New Thread 824bcb8c0 (LWP 103250)]
>>>> [New Thread 823e9b240 (LWP 103249)]
>>>> [New Thread 82510f640 (LWP 102878)]
>>>> [New Thread 818f07d00 (LWP 102872)]
>>>> [New Thread 81a955380 (LWP 102823)]
>>>> [New Thread 8093df100 (LWP 102814)]
>>>> [New Thread 80cd50100 (LWP 102811)]
>>>> [New Thread 81f07e340 (LWP 102879)]
>>>> [New Thread 825139480 (LWP 102865)]
>>>> [New Thread 823138700 (LWP 102868)]
>>>> [New Thread 822f858c0 (LWP 103242)]
>>>> [New Thread 81addc280 (LWP 102590)]
>>>> [New Thread 81852a340 (LWP 103240)]
>>>> [New Thread 82062c280 (LWP 102577)]
>>>> [New Thread 808aed700 (LWP 102565)]
>>>> [New Thread 80bdcaf40 (LWP 102562)]
>>>> [New Thread 820c98340 (LWP 102560)]
>>>> [New Thread 822aa5c40 (LWP 102556)]
>>>> [New Thread 8245b4680 (LWP 102114)]
>>>> [New Thread 8230a58c0 (LWP 102083)]
>>>> [New Thread 825292140 (LWP 103238)]
>>>> [New Thread 81f5f8e40 (LWP 101888)]
>>>> [New Thread 806908700 (LWP 103228)]
>>>> [New Thread 81fdd9a40 (LWP 102856)]
>>>> [New Thread 82071c0c0 (LWP 103227)]
>>>> [New Thread 805548100 (LWP 102854)]
>>>> [New Thread 81e17ea40 (LWP 102202)]
>>>> [New Thread 824d361c0 (LWP 102192)]
>>>> [New Thread 823ea4240 (LWP 102169)]
>>>> [New Thread 824954500 (LWP 102165)]
>>>> [New Thread 81f582e40 (LWP 100696)]
>>>> [New Thread 815d7ee40 (LWP 103224)]
>>>> [New Thread 825291f80 (LWP 103221)]
>>>> [New Thread 825405180 (LWP 102838)]
>>>> [New Thread 824d32e40 (LWP 103199)]
>>>> [New Thread 807f1d380 (LWP 103092)]
>>>> [New Thread 81ba20180 (LWP 102834)]
>>>> [New Thread 810302180 (LWP 102832)]
>>>> [New Thread 825404fc0 (LWP 102828)]
>>>> [New Thread 823ea4080 (LWP 103065)]
>>>> [New Thread 81ca46280 (LWP 102720)]
>>>> [New Thread 80f34c3c0 (LWP 102986)]
>>>> [New Thread 8200aaa80 (LWP 102607)]
>>>> [New Thread 825291dc0 (LWP 102954)]
>>>> [New Thread 825117480 (LWP 102904)]
>>>> [New Thread 8096dda80 (LWP 102631)]
>>>> [New Thread 8063aa900 (LWP 102625)]
>>>> [New Thread 813b77f00 (LWP 102500)]
>>>> [New Thread 81f8f6e00 (LWP 102498)]
>>>> [New Thread 821818a40 (LWP 102505)]
>>>> [New Thread 825404e00 (LWP 102605)]
>>>> [New Thread 824e5e280 (LWP 102585)]
>>>> [New Thread 805549b40 (LWP 102332)]
>>>> [New Thread 8093def40 (LWP 102341)]
>>>> [New Thread 823654fc0 (LWP 102345)]
>>>> [New Thread 8248c86c0 (LWP 102342)]
>>>> [New Thread 80631c580 (LWP 102526)]
>>>> [New Thread 81c07af40 (LWP 102282)]
>>>> [New Thread 8250749c0 (LWP 101997)]
>>>> [New Thread 8248c8500 (LWP 101992)]
>>>> [New Thread 822dfbbc0 (LWP 100739)]
>>>> [New Thread 817c99700 (LWP 100785)]
>>>> [New Thread 81f8c9380 (LWP 101645)]
>>>> [New Thread 815078a80 (LWP 101167)]
>>>> [New Thread 81995d0c0 (LWP 102533)]
>>>> [New Thread 824e5e0c0 (LWP 102139)]
>>>> [New Thread 8248b76c0 (LWP 102531)]
>>>> [New Thread 80712a680 (LWP 102124)]
>>>> [New Thread 820a09400 (LWP 102072)]
>>>> [New Thread 81e5c5880 (LWP 102063)]
>>>> [New Thread 8093debc0 (LWP 102515)]
>>>> [New Thread 81e980340 (LWP 102306)]
>>>> [New Thread 82364e180 (LWP 102529)]
>>>> [New Thread 815bf4400 (LWP 102302)]
>>>> [New Thread 825074800 (LWP 102516)]
>>>> [New Thread 82506c9c0 (LWP 102452)]
>>>> [New Thread 811c34c80 (LWP 101549)]
>>>> [New Thread 81c99c8c0 (LWP 101629)]
>>>> [New Thread 8238b6280 (LWP 102508)]
>>>> [New Thread 81943a3c0 (LWP 102178)]
>>>> [New Thread 822bf3a80 (LWP 102475)]
>>>> [New Thread 808aed540 (LWP 102494)]
>>>> [New Thread 8202b26c0 (LWP 102485)]
>>>> [New Thread 8092e2f80 (LWP 102482)]
>>>> [New Thread 815bec780 (LWP 102480)]
>>>> [New Thread 804e85ac0 (LWP 102478)]
>>>> [New Thread 824610fc0 (LWP 102471)]
>>>> [New Thread 813c0df00 (LWP 102465)]
>>>> [New Thread 81f8c9540 (LWP 102461)]
>>>> [New Thread 824898880 (LWP 102457)]
>>>> [New Thread 824610c40 (LWP 102454)]
>>>> [New Thread 81bb4fc00 (LWP 102254)]
>>>> [New Thread 8204580c0 (LWP 102248)]
>>>> [New Thread 8093dea00 (LWP 101990)]
>>>> [New Thread 80fb3fa80 (LWP 101155)]
>>>> [New Thread 81f8f6c40 (LWP 102262)]
>>>> [New Thread 8238b60c0 (LWP 103215)]
>>>> [New Thread 821e12140 (LWP 102961)]
>>>> [New Thread 81998eb80 (LWP 102247)]
>>>> [New Thread 819e53fc0 (LWP 102848)]
>>>> [New Thread 81918ca80 (LWP 102250)]
>>>> [New Thread 819389180 (LWP 102235)]
>>>> [New Thread 821b21f80 (LWP 102243)]
>>>> [New Thread 821818880 (LWP 102846)]
>>>> [New Thread 82364dfc0 (LWP 102368)]
>>>> [New Thread 822f85700 (LWP 100935)]
>>>> [New Thread 8224d2a40 (LWP 102035)]
>>>> [New Thread 804f7a540 (LWP 100573)]
>>>> [New Thread 81e891500 (LWP 101898)]
>>>> [New Thread 8224d2500 (LWP 102435)]
>>>> [New Thread 823d16d40 (LWP 102825)]
>>>> [New Thread 81d2ac8c0 (LWP 102431)]
>>>> [New Thread 823432380 (LWP 102817)]
>>>> [New Thread 8192d1180 (LWP 102608)]
>>>> [New Thread 80c082bc0 (LWP 102583)]
>>>> [New Thread 81f84c700 (LWP 102428)]
>>>> [New Thread 810f36e80 (LWP 102426)]
>>>> [New Thread 8214f3880 (LWP 102430)]
>>>> [New Thread 8093cff80 (LWP 101434)]
>>>> [New Thread 81995cf00 (LWP 102180)]
>>>> [New Thread 81998f280 (LWP 102840)]
>>>> [New Thread 80921fc00 (LWP 102836)]
>>>> [New Thread 81c52b500 (LWP 102421)]
>>>> [New Thread 822bf38c0 (LWP 102425)]
>>>> [New Thread 823d8db80 (LWP 102415)]
>>>> [New Thread 823644180 (LWP 102402)]
>>>> [New Thread 813e28900 (LWP 102819)]
>>>> [New Thread 82342a1c0 (LWP 102399)]
>>>> [New Thread 804189c80 (LWP 102387)]
>>>> [New Thread 824610e00 (LWP 102621)]
>>>> [New Thread 821a11f80 (LWP 102155)]
>>>> [New Thread 821dccf00 (LWP 101427)]
>>>> [New Thread 8055458c0 (LWP 101241)]
>>>> [New Thread 80e69c800 (LWP 102446)]
>>>> [New Thread 821dccb80 (LWP 102429)]
>>>> [New Thread 8218186c0 (LWP 102492)]
>>>> [New Thread 8221a0fc0 (LWP 102420)]
>>>> [New Thread 81057e580 (LWP 102407)]
>>>> [New Thread 81f72a440 (LWP 102400)]
>>>> [New Thread 81f8f6a80 (LWP 102389)]
>>>> [New Thread 8093de840 (LWP 102386)]
>>>> [New Thread 822dd5bc0 (LWP 101379)]
>>>> [New Thread 822dfba00 (LWP 102441)]
>>>> [New Thread 81b79a880 (LWP 102364)]
>>>> [New Thread 8230a5700 (LWP 102303)]
>>>> [New Thread 813c0db80 (LWP 102256)]
>>>> [New Thread 80ccc2f40 (LWP 102204)]
>>>> [New Thread 821b21dc0 (LWP 101593)]
>>>> [New Thread 820b7b400 (LWP 101296)]
>>>> [New Thread 805bd0780 (LWP 103223)]
>>>> [New Thread 821e11f80 (LWP 103222)]
>>>> [New Thread 821dccd40 (LWP 103220)]
>>>> [New Thread 821a11dc0 (LWP 103218)]
>>>> [New Thread 803ebd740 (LWP 103216)]
>>>> [New Thread 822b17c40 (LWP 102017)]
>>>> [New Thread 822edaa80 (LWP 103213)]
>>>> [New Thread 8238b5f00 (LWP 103211)]
>>>> [New Thread 8234321c0 (LWP 103209)]
>>>> [New Thread 823430e40 (LWP 103207)]
>>>> [New Thread 81ea9dbc0 (LWP 103197)]
>>>> [New Thread 81df0cd80 (LWP 102795)]
>>>> [New Thread 81d83e700 (LWP 102804)]
>>>> [New Thread 822f02700 (LWP 102784)]
>>>> [New Thread 821e11dc0 (LWP 102783)]
>>>> [New Thread 81f8f68c0 (LWP 103196)]
>>>> [New Thread 822c62d80 (LWP 103194)]
>>>> [New Thread 821818500 (LWP 102627)]
>>>> [New Thread 81fbd2d80 (LWP 102626)]
>>>> [New Thread 809856c40 (LWP 103188)]
>>>> [New Thread 80b28cd80 (LWP 103172)]
>>>> [New Thread 8226ff180 (LWP 103159)]
>>>> [New Thread 8093d7680 (LWP 102597)]
>>>> [New Thread 80ffc9200 (LWP 102586)]
>>>> [New Thread 803854300 (LWP 103147)]
>>>> [New Thread 80c0f7a00 (LWP 103139)]
>>>> [New Thread 821e11c00 (LWP 103136)]
>>>> [New Thread 8093d74c0 (LWP 101192)]
>>>> [New Thread 816658900 (LWP 101479)]
>>>> [New Thread 81dad2600 (LWP 103128)]
>>>> [New Thread 820b7b240 (LWP 103125)]
>>>> [New Thread 821818340 (LWP 103119)]
>>>> [New Thread 8093d7300 (LWP 103111)]
>>>> [New Thread 81f8f6700 (LWP 102552)]
>>>> [New Thread 8226fefc0 (LWP 103101)]
>>>> [New Thread 815d081c0 (LWP 103094)]
>>>> [New Thread 8210efc40 (LWP 102548)]
>>>> [New Thread 80edb2900 (LWP 102544)]
>>>> [New Thread 81d4ac280 (LWP 103091)]
>>>> [New Thread 821780e40 (LWP 103089)]
>>>> [New Thread 8224d2880 (LWP 103086)]
>>>> [New Thread 81c0f6100 (LWP 103075)]
>>>> [New Thread 821d4cd40 (LWP 102525)]
>>>> [New Thread 822d25bc0 (LWP 103063)]
>>>> [New Thread 803855f00 (LWP 103015)]
>>>> [New Thread 8221a0e00 (LWP 103004)]
>>>> [New Thread 822c5abc0 (LWP 103044)]
>>>> [New Thread 8182d72c0 (LWP 103000)]
>>>> [New Thread 809856a80 (LWP 102406)]
>>>> [New Thread 819df86c0 (LWP 103037)]
>>>> [New Thread 821a11c00 (LWP 103034)]
>>>> [New Thread 821a11a40 (LWP 103032)]
>>>> [New Thread 806da7480 (LWP 102542)]
>>>> [New Thread 81c909c40 (LWP 103028)]
>>>> [New Thread 8229350c0 (LWP 102493)]
>>>> [New Thread 822c62bc0 (LWP 103018)]
>>>> [New Thread 822c62a00 (LWP 103010)]
>>>> [New Thread 81d4a3f00 (LWP 103007)]
>>>> [New Thread 811efcf00 (LWP 103005)]
>>>> [New Thread 8093d7140 (LWP 103003)]
>>>> [New Thread 8224d26c0 (LWP 102999)]
>>>> [New Thread 8210efa80 (LWP 102997)]
>>>> [New Thread 80ce66f00 (LWP 102994)]
>>>> [New Thread 8070e8e40 (LWP 102992)]
>>>> [New Thread 820b7b080 (LWP 102990)]
>>>> [New Thread 81f8f6540 (LWP 102451)]
>>>> [New Thread 81c3c77c0 (LWP 102439)]
>>>> [New Thread 818f07ec0 (LWP 102984)]
>>>> [New Thread 819c78180 (LWP 102830)]
>>>> [New Thread 8213fad40 (LWP 102419)]
>>>> [New Thread 819e4ce00 (LWP 101239)]
>>>> [New Thread 80b288e80 (LWP 101374)]
>>>> [New Thread 8226fee00 (LWP 101521)]
>>>> [New Thread 82167c1c0 (LWP 102196)]
>>>> [New Thread 8228a20c0 (LWP 101862)]
>>>> [New Thread 808171c40 (LWP 101878)]
>>>> [New Thread 821e11a40 (LWP 102980)]
>>>> [New Thread 81be1ca80 (LWP 102978)]
>>>> [New Thread 80f703c80 (LWP 102977)]
>>>> [New Thread 821a11880 (LWP 102975)]
>>>> [New Thread 8228a1f00 (LWP 102190)]
>>>> [New Thread 80ce582c0 (LWP 102972)]
>>>> [New Thread 81f8f6380 (LWP 102970)]
>>>> [New Thread 81c901c40 (LWP 102969)]
>>>> [New Thread 81aa28a00 (LWP 102967)]
>>>> [New Thread 81fbd2bc0 (LWP 102112)]
>>>> [New Thread 810acfa80 (LWP 102963)]
>>>> [New Thread 82166de40 (LWP 101948)]
>>>> [New Thread 821e11880 (LWP 102964)]
>>>> [New Thread 821a116c0 (LWP 102416)]
>>>> [New Thread 8213fab80 (LWP 102960)]
>>>> [New Thread 805558180 (LWP 102414)]
>>>> [New Thread 8213fa9c0 (LWP 102410)]
>>>> [New Thread 82166dc80 (LWP 102409)]
>>>> [New Thread 820b7aec0 (LWP 102952)]
>>>> [New Thread 821263f00 (LWP 102403)]
>>>> [New Thread 81dd4bc40 (LWP 102903)]
>>>> [New Thread 80e841440 (LWP 102898)]
>>>> [New Thread 81ff83880 (LWP 102873)]
>>>> [New Thread 81faa2bc0 (LWP 102391)]
>>>> [New Thread 81fc8bc00 (LWP 102401)]
>>>> [New Thread 81842efc0 (LWP 102871)]
>>>> [New Thread 818ffe5c0 (LWP 102867)]
>>>> [New Thread 821818180 (LWP 102863)]
>>>> [New Thread 81fb27840 (LWP 102857)]
>>>> [New Thread 8214f36c0 (LWP 102845)]
>>>> [New Thread 81b484540 (LWP 102844)]
>>>> [New Thread 804b31180 (LWP 102380)]
>>>> [New Thread 81ad23f00 (LWP 102383)]
>>>> [New Thread 8063e73c0 (LWP 102385)]
>>>> [New Thread 8172f03c0 (LWP 102384)]
>>>> [New Thread 820c98180 (LWP 102382)]
>>>> [New Thread 81fb27d80 (LWP 102379)]
>>>> [New Thread 81f07e180 (LWP 102074)]
>>>> [New Thread 8070e4140 (LWP 102060)]
>>>> [New Thread 80b7fee40 (LWP 101984)]
>>>> [New Thread 80e784f00 (LWP 101925)]
>>>> [New Thread 81fb27bc0 (LWP 100720)]
>>>> [New Thread 81f5f8c80 (LWP 100686)]
>>>> [New Thread 81be9efc0 (LWP 102372)]
>>>> [New Thread 816eed100 (LWP 102839)]
>>>> [New Thread 807924340 (LWP 102369)]
>>>> [New Thread 8214f3500 (LWP 102835)]
>>>> [New Thread 820a10240 (LWP 102358)]
>>>> [New Thread 8210ef8c0 (LWP 102337)]
>>>> [New Thread 804b32680 (LWP 102338)]
>>>> [New Thread 803f3f740 (LWP 102339)]
>>>> [New Thread 811037180 (LWP 102782)]
>>>> [New Thread 81f07dfc0 (LWP 102314)]
>>>> [New Thread 8212f0b80 (LWP 102609)]
>>>> [New Thread 82033e500 (LWP 102276)]
>>>> [New Thread 805d58900 (LWP 102581)]
>>>> [New Thread 806788e00 (LWP 102551)]
>>>> [New Thread 81901c8c0 (LWP 102205)]
>>>> [New Thread 81ab1ff40 (LWP 102209)]
>>>> [New Thread 81558bbc0 (LWP 102353)]
>>>> [New Thread 80e7c3280 (LWP 102198)]
>>>> [New Thread 819222e00 (LWP 102143)]
>>>> [New Thread 81fb27a00 (LWP 102141)]
>>>> [New Thread 820c97fc0 (LWP 102351)]
>>>> [New Thread 81dd4ba80 (LWP 102347)]
>>>> [New Thread 814ab0480 (LWP 102094)]
>>>> [New Thread 803634740 (LWP 102092)]
>>>> [New Thread 80e83a440 (LWP 102300)]
>>>> [New Thread 80a595e40 (LWP 102090)]
>>>> [New Thread 81e9ab340 (LWP 102097)]
>>>> [New Thread 81faa2a00 (LWP 102086)]
>>>> [New Thread 81dd4b8c0 (LWP 102298)]
>>>> [New Thread 810f36b00 (LWP 101657)]
>>>> [New Thread 8049b4c80 (LWP 101681)]
>>>> [New Thread 820c97e00 (LWP 101545)]
>>>> [New Thread 81f7ec280 (LWP 101391)]
>>>> [New Thread 82062c0c0 (LWP 101943)]
>>>> [New Thread 813b780c0 (LWP 101776)]
>>>> [New Thread 81d1ca700 (LWP 101771)]
>>>> [New Thread 81e9ab180 (LWP 102266)]
>>>> [New Thread 806cc5bc0 (LWP 102050)]
>>>> [New Thread 81dd4b700 (LWP 101225)]
>>>> [New Thread 81f57c1c0 (LWP 102277)]
>>>> [New Thread 81e04aa40 (LWP 102274)]
>>>> [New Thread 80d150340 (LWP 102271)]
>>>> [New Thread 81b65d6c0 (LWP 102269)]
>>>> [New Thread 81b3fce40 (LWP 102005)]
>>>> [New Thread 81dd4b540 (LWP 102232)]
>>>> [New Thread 8093c8f80 (LWP 102226)]
>>>> [New Thread 81b4401c0 (LWP 101885)]
>>>> [New Thread 81f029fc0 (LWP 102040)]
>>>> [New Thread 81fc8ba40 (LWP 102829)]
>>>> [New Thread 81d1ca540 (LWP 102827)]
>>>> [New Thread 803852e00 (LWP 102030)]
>>>> [New Thread 81998ed40 (LWP 102032)]
>>>> [New Thread 8170ab8c0 (LWP 102159)]
>>>> [New Thread 806d5bcc0 (LWP 102026)]
>>>> [New Thread 81e04a880 (LWP 102021)]
>>>> [New Thread 8092e2dc0 (LWP 102027)]
>>>> [New Thread 81e5c56c0 (LWP 102019)]
>>>> [New Thread 815b9e5c0 (LWP 101998)]
>>>> [New Thread 81c52ba40 (LWP 101639)]
>>>> [New Thread 81bb4fa40 (LWP 100833)]
>>>> [New Thread 8070e8ac0 (LWP 101906)]
>>>> [New Thread 815426f40 (LWP 102397)]
>>>> [New Thread 806925380 (LWP 100917)]
>>>> [New Thread 809df4700 (LWP 102378)]
>>>> [New Thread 8097d7f80 (LWP 102215)]
>>>> [New Thread 8042abac0 (LWP 102289)]
>>>> [New Thread 80f15c540 (LWP 102375)]
>>>> [New Thread 81faa2840 (LWP 102211)]
>>>> [New Thread 8077c96c0 (LWP 102336)]
>>>> [New Thread 8092e2c00 (LWP 102320)]
>>>> [New Thread 80ecef900 (LWP 102137)]
>>>> [New Thread 81bb4f880 (LWP 102207)]
>>>> [New Thread 81eec80c0 (LWP 102193)]
>>>> [New Thread 819246180 (LWP 102184)]
>>>> [New Thread 81f029e00 (LWP 102168)]
>>>> [New Thread 81b3f61c0 (LWP 102166)]
>>>> [New Thread 81bb4f6c0 (LWP 102164)]
>>>> [New Thread 81f029c40 (LWP 100673)]
>>>> [New Thread 808172340 (LWP 102128)]
>>>> [New Thread 81df05d80 (LWP 101810)]
>>>> [New Thread 818ae5880 (LWP 102077)]
>>>> [New Thread 81eec7f00 (LWP 102012)]
>>>> [New Thread 81bb4f500 (LWP 101996)]
>>>> [New Thread 80dde0c80 (LWP 101918)]
>>>> [New Thread 819685100 (LWP 101535)]
>>>> [New Thread 81c52b880 (LWP 101195)]
>>>> [New Thread 81c52b6c0 (LWP 102826)]
>>>> [New Thread 81e04a6c0 (LWP 102824)]
>>>> [New Thread 80e69c640 (LWP 102820)]
>>>> [New Thread 80fe0a3c0 (LWP 102818)]
>>>> [New Thread 81a2a5a40 (LWP 102816)]
>>>> [New Thread 81d7c1400 (LWP 102815)]
>>>> [New Thread 81d1ca380 (LWP 102813)]
>>>> [New Thread 81e0436c0 (LWP 102810)]
>>>> [New Thread 81a1b2740 (LWP 102808)]
>>>> [New Thread 819deffc0 (LWP 102805)]
>>>> [New Thread 81a1bde40 (LWP 102802)]
>>>> [New Thread 818a586c0 (LWP 102715)]
>>>> [New Thread 812898040 (LWP 102343)]
>>>> [New Thread 803ebd3c0 (LWP 102340)]
>>>> [New Thread 81d845c40 (LWP 102666)]
>>>> [New Thread 81d845a80 (LWP 102634)]
>>>> [New Thread 81e891340 (LWP 102629)]
>>>> [New Thread 80b287ec0 (LWP 102327)]
>>>> [New Thread 81a4820c0 (LWP 102328)]
>>>> [New Thread 81d2ac700 (LWP 102329)]
>>>> [New Thread 81e04a500 (LWP 102606)]
>>>> [New Thread 8170a48c0 (LWP 102322)]
>>>> [New Thread 80ccbbd80 (LWP 102592)]
>>>> [New Thread 81df0cbc0 (LWP 102318)]
>>>> [New Thread 80921fa40 (LWP 102317)]
>>>> [New Thread 80778c500 (LWP 102579)]
>>>> [New Thread 8137f7340 (LWP 102568)]
>>>> [New Thread 80921f880 (LWP 100772)]
>>>> [New Thread 81dadf600 (LWP 102566)]
>>>> [New Thread 815b96940 (LWP 102564)]
>>>> [New Thread 805e0ae40 (LWP 102561)]
>>>> [New Thread 81bb4f340 (LWP 102559)]
>>>> [New Thread 81d4ac0c0 (LWP 102555)]
>>>> [New Thread 81d716400 (LWP 102297)]
>>>> [New Thread 81bb4f180 (LWP 102549)]
>>>> [New Thread 8040f0700 (LWP 102294)]
>>>> [New Thread 80ccad680 (LWP 102292)]
>>>> [New Thread 808ae61c0 (LWP 102293)]
>>>> [New Thread 80ccacdc0 (LWP 102290)]
>>>> [New Thread 81dadf440 (LWP 102534)]
>>>> [New Thread 81c99ca80 (LWP 102532)]
>>>> [New Thread 81130f340 (LWP 102530)]
>>>> [New Thread 8111e16c0 (LWP 102527)]
>>>> [New Thread 81b729a40 (LWP 102278)]
>>>> [New Thread 81cd26fc0 (LWP 102521)]
>>>> [New Thread 805559bc0 (LWP 102517)]
>>>> [New Thread 8063e1380 (LWP 101299)]
>>>> [New Thread 81cc37180 (LWP 101204)]
>>>> [New Thread 81a2a5880 (LWP 102469)]
>>>> [New Thread 81a98e700 (LWP 102514)]
>>>> [New Thread 815aa7b00 (LWP 102511)]
>>>> [New Thread 81d8458c0 (LWP 102507)]
>>>> [New Thread 81d845700 (LWP 102261)]
>>>> [New Thread 81a1bdc80 (LWP 102496)]
>>>> [New Thread 804bfb280 (LWP 102258)]
>>>> [New Thread 805f831c0 (LWP 102486)]
>>>> [New Thread 81d845540 (LWP 102484)]
>>>> [New Thread 81b729880 (LWP 102481)]
>>>> [New Thread 81d4abf00 (LWP 102479)]
>>>> [New Thread 81897ef00 (LWP 102477)]
>>>> [New Thread 8068fa700 (LWP 102472)]
>>>> [New Thread 81d4abd40 (LWP 102470)]
>>>> [New Thread 81c909a80 (LWP 102468)]
>>>> [New Thread 80921f6c0 (LWP 102233)]
>>>> [New Thread 81cc36fc0 (LWP 102458)]
>>>> [New Thread 805dad900 (LWP 102456)]
>>>> [New Thread 81d1ca1c0 (LWP 102224)]
>>>> [New Thread 81a2a56c0 (LWP 102152)]
>>>> [New Thread 817d1e540 (LWP 102440)]
>>>> [New Thread 8188e5f00 (LWP 102438)]
>>>> [New Thread 8188e5b80 (LWP 102437)]
>>>> [New Thread 81a98e540 (LWP 101764)]
>>>> [New Thread 8111e1500 (LWP 101151)]
>>>> [New Thread 80edb2740 (LWP 102197)]
>>>> [New Thread 810469900 (LWP 102191)]
>>>> [New Thread 81cc36e00 (LWP 101901)]
>>>> [New Thread 813c0d9c0 (LWP 101642)]
>>>> [New Thread 80e883440 (LWP 101817)]
>>>> [New Thread 810b2cc40 (LWP 101853)]
>>>> [New Thread 81be1cc40 (LWP 102433)]
>>>> [New Thread 805bcf980 (LWP 102219)]
>>>> [New Thread 81a2a5500 (LWP 101900)]
>>>> [New Thread 81bb4efc0 (LWP 102427)]
>>>> [New Thread 80b168180 (LWP 101897)]
>>>> [New Thread 81943a580 (LWP 100599)]
>>>> [New Thread 81d046380 (LWP 102424)]
>>>> [New Thread 81a2a5340 (LWP 100919)]
>>>> [New Thread 817f32240 (LWP 102206)]
>>>> [New Thread 80b981180 (LWP 102422)]
>>>> [New Thread 81a1bdac0 (LWP 102200)]
>>>> [New Thread 81a98e380 (LWP 102418)]
>>>> [New Thread 81c0f5f40 (LWP 102199)]
>>>> [New Thread 81b7296c0 (LWP 102194)]
>>>> [New Thread 803856b40 (LWP 102411)]
>>>> [New Thread 81afc37c0 (LWP 102188)]
>>>> [New Thread 81afc3600 (LWP 102187)]
>>>> [New Thread 81c0f5d80 (LWP 102404)]
>>>> [New Thread 814621e80 (LWP 102172)]
>>>> [New Thread 80fb3fe00 (LWP 102186)]
>>>> [New Thread 81a98e1c0 (LWP 102175)]
>>>> [New Thread 81558bd80 (LWP 102183)]
>>>> [New Thread 80cd50f00 (LWP 102392)]
>>>> [New Thread 81ca460c0 (LWP 102390)]
>>>> [New Thread 8111e1340 (LWP 102388)]
>>>> [New Thread 808500500 (LWP 102163)]
>>>> [New Thread 808b5d8c0 (LWP 102162)]
>>>> [New Thread 80b28c140 (LWP 102156)]
>>>> [New Thread 81afc3440 (LWP 101755)]
>>>> [New Thread 819441200 (LWP 101601)]
>>>> [New Thread 817f8b240 (LWP 101005)]
>>>> [New Thread 80fe0a200 (LWP 101540)]
>>>> [New Thread 815aa75c0 (LWP 101421)]
>>>> [New Thread 8147ee940 (LWP 101396)]
>>>> [New Thread 818ffe400 (LWP 100571)]
>>>> [New Thread 818ffe240 (LWP 101307)]
>>>> [New Thread 8055571c0 (LWP 102140)]
>>>> [New Thread 80974f380 (LWP 102370)]
>>>> [New Thread 81c07ad80 (LWP 102135)]
>>>> [New Thread 81addc0c0 (LWP 102359)]
>>>> [New Thread 80f703ac0 (LWP 102324)]
>>>> [New Thread 8101768c0 (LWP 102123)]
>>>> [New Thread 80b393880 (LWP 102113)]
>>>> [New Thread 81b484380 (LWP 100945)]
>>>> [New Thread 81a1bd900 (LWP 102315)]
>>>> [New Thread 80544fe80 (LWP 102071)]
>>>> [New Thread 80f703900 (LWP 102288)]
>>>> [New Thread 806c381c0 (LWP 102047)]
>>>> [New Thread 81c008d80 (LWP 102229)]
>>>> [New Thread 81be9ee00 (LWP 102217)]
>>>> [New Thread 810d32e00 (LWP 102120)]
>>>> [New Thread 804a44c40 (LWP 102002)]
>>>> [New Thread 818102740 (LWP 102118)]
>>>> [New Thread 806788c40 (LWP 101988)]
>>>> [New Thread 81addbf00 (LWP 101983)]
>>>> [New Thread 81be1ce00 (LWP 102115)]
>>>> [New Thread 81afc3280 (LWP 102100)]
>>>> [New Thread 8109e9fc0 (LWP 101839)]
>>>> [New Thread 81afc30c0 (LWP 102056)]
>>>> [New Thread 818f87400 (LWP 101801)]
>>>> [New Thread 81addbd40 (LWP 101814)]
>>>> [New Thread 811316fc0 (LWP 101822)]
>>>> [New Thread 8047a3380 (LWP 101957)]
>>>> [New Thread 81b664880 (LWP 101576)]
>>>> [New Thread 81ba1ffc0 (LWP 101358)]
>>>> [New Thread 81a2a5180 (LWP 101274)]
>>>> [New Thread 8081361c0 (LWP 100807)]
>>>> [New Thread 819df8500 (LWP 102065)]
>>>> [New Thread 806214900 (LWP 102062)]
>>>> [New Thread 80544d800 (LWP 102061)]
>>>> [New Thread 81ad23d40 (LWP 100711)]
>>>> [New Thread 81a1bd740 (LWP 102052)]
>>>> [New Thread 815bf45c0 (LWP 102048)]
>>>> [New Thread 81b4841c0 (LWP 101792)]
>>>> [New Thread 81b6646c0 (LWP 102006)]
>>>> [New Thread 807923a80 (LWP 102004)]
>>>> [New Thread 81ab1fd80 (LWP 101975)]
>>>> [New Thread 811f4b440 (LWP 100746)]
>>>> [New Thread 81b446e40 (LWP 101907)]
>>>> [New Thread 81b664500 (LWP 102366)]
>>>> [New Thread 8079238c0 (LWP 102362)]
>>>> [New Thread 81318cf40 (LWP 102104)]
>>>> [New Thread 8134c5580 (LWP 101850)]
>>>> [New Thread 815f2e8c0 (LWP 101860)]
>>>> [New Thread 819388fc0 (LWP 101861)]
>>>> [New Thread 819df8340 (LWP 101873)]
>>>> [New Thread 80b7b41c0 (LWP 101857)]
>>>> [New Thread 80554b580 (LWP 102080)]
>>>> [New Thread 80ea0da80 (LWP 100595)]
>>>> [New Thread 8176b4e40 (LWP 101537)]
>>>> [New Thread 805c59cc0 (LWP 102181)]
>>>> [New Thread 805557fc0 (LWP 102179)]
>>>> [New Thread 80ccb44c0 (LWP 102161)]
>>>> [New Thread 81ad23b80 (LWP 102158)]
>>>> [New Thread 806214740 (LWP 101989)]
>>>> [New Thread 80d010700 (LWP 102081)]
>>>> [New Thread 819df8180 (LWP 102127)]
>>>> [New Thread 819df7fc0 (LWP 102111)]
>>>> [New Thread 81ab1fbc0 (LWP 102107)]
>>>> [New Thread 815b9e940 (LWP 101724)]
>>>> [New Thread 8192d0fc0 (LWP 102025)]
>>>> [New Thread 8066d7540 (LWP 101982)]
>>>> [New Thread 805bd0940 (LWP 101963)]
>>>> [New Thread 8197a3d80 (LWP 101932)]
>>>> [New Thread 8040f6e40 (LWP 101927)]
>>>> [New Thread 819e53e00 (LWP 101922)]
>>>> [New Thread 80660a380 (LWP 101911)]
>>>> [New Thread 806c3c600 (LWP 100616)]
>>>> [New Thread 81918c8c0 (LWP 101788)]
>>>> [New Thread 806788540 (LWP 101637)]
>>>> [New Thread 8111e1180 (LWP 101237)]
>>>> [New Thread 818ae56c0 (LWP 101397)]
>>>> [New Thread 81ac2df00 (LWP 101329)]
>>>> [New Thread 81aa28bc0 (LWP 100782)]
>>>> [New Thread 818f87240 (LWP 102365)]
>>>> [New Thread 81ab1fa00 (LWP 102363)]
>>>> [New Thread 819795f40 (LWP 102361)]
>>>> [New Thread 80533ba80 (LWP 101348)]
>>>> [New Thread 81a1bd580 (LWP 102357)]
>>>> [New Thread 80bcf82c0 (LWP 102356)]
>>>> [New Thread 819a08280 (LWP 102354)]
>>>> [New Thread 80660a1c0 (LWP 102352)]
>>>> [New Thread 819c05180 (LWP 102350)]
>>>> [New Thread 8051c7940 (LWP 102348)]
>>>> [New Thread 819df7e00 (LWP 102346)]
>>>> [New Thread 819245fc0 (LWP 102344)]
>>>> [New Thread 80b528e40 (LWP 102133)]
>>>> [New Thread 8155e3f40 (LWP 102132)]
>>>> [New Thread 807b6b380 (LWP 102129)]
>>>> [New Thread 818a58340 (LWP 102130)]
>>>> [New Thread 81a0c1740 (LWP 102335)]
>>>> [New Thread 81a0c1580 (LWP 102333)]
>>>> [New Thread 807f1d540 (LWP 102331)]
>>>> [New Thread 81a0c13c0 (LWP 102117)]
>>>> [New Thread 806047e40 (LWP 102116)]
>>>> [New Thread 80a11f840 (LWP 102323)]
>>>> [New Thread 80554ae80 (LWP 102109)]
>>>> [New Thread 81a489f00 (LWP 102319)]
>>>> [New Thread 803f39700 (LWP 102313)]
>>>> [New Thread 81a205340 (LWP 102103)]
>>>> [New Thread 80e570800 (LWP 102105)]
>>>> [New Thread 818f08080 (LWP 102310)]
>>>> [New Thread 80bd6cf40 (LWP 102307)]
>>>> [New Thread 81a205180 (LWP 100691)]
>>>> [New Thread 80ccb4680 (LWP 102304)]
>>>> [New Thread 815eb28c0 (LWP 101201)]
>>>> [New Thread 817d25380 (LWP 102301)]
>>>> [New Thread 806047c80 (LWP 102299)]
>>>> [New Thread 819a080c0 (LWP 102082)]
>>>> [New Thread 81995d280 (LWP 102295)]
>>>> [New Thread 819245e00 (LWP 102287)]
>>>> [New Thread 811170340 (LWP 102041)]
>>>> [New Thread 819245c40 (LWP 102286)]
>>>> [New Thread 818f87080 (LWP 102075)]
>>>> [New Thread 818636fc0 (LWP 102285)]
>>>> [New Thread 80b28ac40 (LWP 102283)]
>>>> [New Thread 81998f0c0 (LWP 102281)]
>>>> [New Thread 8199640c0 (LWP 102279)]
>>>> [New Thread 819e53c40 (LWP 102136)]
>>>> [New Thread 8190a78c0 (LWP 102275)]
>>>> [New Thread 809856e00 (LWP 101287)]
>>>> [New Thread 811170180 (LWP 102272)]
>>>> [New Thread 8185f3180 (LWP 102270)]
>>>> [New Thread 804382a80 (LWP 102267)]
>>>> [New Thread 806047ac0 (LWP 101089)]
>>>> [New Thread 805b155c0 (LWP 101371)]
>>>> [New Thread 80ce66d40 (LWP 102265)]
>>>> [New Thread 812897e80 (LWP 102263)]
>>>> [New Thread 8050607c0 (LWP 102260)]
>>>> [New Thread 81998ef00 (LWP 102259)]
>>>> [New Thread 819c77fc0 (LWP 102037)]
>>>> [New Thread 814ed6700 (LWP 102255)]
>>>> [New Thread 8155dbf40 (LWP 102253)]
>>>> [New Thread 819a07f00 (LWP 102251)]
>>>> [New Thread 818f86ec0 (LWP 102249)]
>>>> [New Thread 814b4e100 (LWP 102246)]
>>>> [New Thread 80b7b4540 (LWP 102245)]
>>>> [New Thread 806da7640 (LWP 102240)]
>>>> [New Thread 819c04fc0 (LWP 102238)]
>>>> [New Thread 8176b4c80 (LWP 102236)]
>>>> [New Thread 818ae5500 (LWP 102234)]
>>>> [New Thread 815c2b540 (LWP 102014)]
>>>> [New Thread 810c33c40 (LWP 102015)]
>>>> [New Thread 80400a700 (LWP 102013)]
>>>> [New Thread 81116ffc0 (LWP 102227)]
>>>> [New Thread 819a07d40 (LWP 102225)]
>>>> [New Thread 80554a5c0 (LWP 101968)]
>>>> [New Thread 808171fc0 (LWP 101924)]
>>>> [New Thread 80b3d6dc0 (LWP 101919)]
>>>> [New Thread 8130f2100 (LWP 101893)]
>>>> [New Thread 81897ed40 (LWP 101880)]
>>>> [New Thread 8077d0a40 (LWP 100903)]
>>>> [New Thread 80b28ca00 (LWP 101728)]
>>>> [New Thread 81982e440 (LWP 102221)]
>>>> [New Thread 80ea0d8c0 (LWP 101993)]
>>>> [New Thread 818a58500 (LWP 102216)]
>>>> [New Thread 8109f8340 (LWP 102213)]
>>>> [New Thread 818636e00 (LWP 102214)]
>>>> [New Thread 8069088c0 (LWP 100745)]
>>>> [New Thread 81107e180 (LWP 101159)]
>>>> [New Thread 8194413c0 (LWP 102210)]
>>>> [New Thread 8190a7700 (LWP 100609)]
>>>> [New Thread 80b3106c0 (LWP 101985)]
>>>> [New Thread 8148a0e00 (LWP 102208)]
>>>> [New Thread 80544db80 (LWP 101977)]
>>>> [New Thread 8182d7100 (LWP 101976)]
>>>> [New Thread 8188e5d40 (LWP 102203)]
>>>> [New Thread 808b5da80 (LWP 101970)]
>>>> [New Thread 804ad6680 (LWP 101972)]
>>>> [New Thread 8136d1500 (LWP 101971)]
>>>> [New Thread 813e28e40 (LWP 102195)]
>>>> [New Thread 814621cc0 (LWP 101964)]
>>>> [New Thread 80e76bd40 (LWP 101962)]
>>>> [New Thread 805ca3200 (LWP 102189)]
>>>> [New Thread 804b32a00 (LWP 101935)]
>>>> [New Thread 8113176c0 (LWP 101959)]
>>>> [New Thread 817f32400 (LWP 101933)]
>>>> [New Thread 80a2f6c40 (LWP 101947)]
>>>> [New Thread 81016f8c0 (LWP 102177)]
>>>> [New Thread 81901c700 (LWP 102176)]
>>>> [New Thread 805b14d00 (LWP 102174)]
>>>> [New Thread 818636c40 (LWP 102171)]
>>>> [New Thread 807180a00 (LWP 101915)]
>>>> [New Thread 8130f22c0 (LWP 101917)]
>>>> [New Thread 806ea3e40 (LWP 101912)]
>>>> [New Thread 80b2d18c0 (LWP 101909)]
>>>> [New Thread 80b289c80 (LWP 101832)]
>>>> [New Thread 80a590500 (LWP 101215)]
>>>> [New Thread 81842f500 (LWP 101189)]
>>>> [New Thread 807af1540 (LWP 101182)]
>>>> [New Thread 80b28a1c0 (LWP 101161)]
>>>> [New Thread 80b289e40 (LWP 101156)]
>>>> [New Thread 803853a40 (LWP 101075)]
>>>> [New Thread 81842f340 (LWP 100946)]
>>>> [New Thread 80e7c30c0 (LWP 102149)]
>>>> [New Thread 80e7c2f00 (LWP 101879)]
>>>> [New Thread 805d58740 (LWP 102126)]
>>>> [New Thread 804bf9680 (LWP 101868)]
>>>> [New Thread 8038559c0 (LWP 102079)]
>>>> [New Thread 8147eeb00 (LWP 101856)]
>>>> [New Thread 8182d6f40 (LWP 101866)]
>>>> [New Thread 80cd50d40 (LWP 101869)]
>>>> [New Thread 805546f80 (LWP 101867)]
>>>> [New Thread 80a07e500 (LWP 102073)]
>>>> [New Thread 807340380 (LWP 101800)]
>>>> [New Thread 80cd50b80 (LWP 102059)]
>>>> [New Thread 805d58580 (LWP 101758)]
>>>> [New Thread 81842f180 (LWP 101471)]
>>>> [New Thread 80e784d40 (LWP 100702)]
>>>> [New Thread 8038560c0 (LWP 101940)]
>>>> [New Thread 805547a00 (LWP 101931)]
>>>> [New Thread 813e25b80 (LWP 101630)]
>>>> [New Thread 805558a40 (LWP 101620)]
>>>> [New Thread 80f15c8c0 (LWP 101619)]
>>>> [New Thread 805d583c0 (LWP 101930)]
>>>> [New Thread 80cd509c0 (LWP 101914)]
>>>> [New Thread 80bcf7f40 (LWP 101714)]
>>>> [New Thread 80bc7f480 (LWP 101426)]
>>>> [New Thread 80fb9ee00 (LWP 101388)]
>>>> [New Thread 818102580 (LWP 101369)]
>>>> [New Thread 80e76bb80 (LWP 101676)]
>>>> [New Thread 8172f0740 (LWP 101234)]
>>>> [New Thread 8158ddfc0 (LWP 101211)]
>>>> [New Thread 805545700 (LWP 100932)]
>>>> [New Thread 80b9c3540 (LWP 100713)]
>>>> [New Thread 805ca3040 (LWP 101186)]
>>>> [New Thread 815d08380 (LWP 101430)]
>>>> [New Thread 816eecf40 (LWP 101667)]
>>>> [New Thread 8070e4300 (LWP 101844)]
>>>> [New Thread 80d150180 (LWP 101811)]
>>>> [New Thread 80fb9efc0 (LWP 101777)]
>>>> [New Thread 80d14f540 (LWP 101708)]
>>>> [New Thread 810325180 (LWP 101531)]
>>>> [New Thread 8136d1340 (LWP 101910)]
>>>> [New Thread 810302340 (LWP 101789)]
>>>> [New Thread 80cd50800 (LWP 101575)]
>>>> [New Thread 811317500 (LWP 101554)]
>>>> [New Thread 816eecd80 (LWP 102150)]
>>>> [New Thread 80cd50640 (LWP 102145)]
>>>> [New Thread 8077c9500 (LWP 101967)]
>>>> [New Thread 80b289200 (LWP 101896)]
>>>> [New Thread 805ca2e80 (LWP 101954)]
>>>> [New Thread 80d14ffc0 (LWP 101953)]
>>>> [New Thread 811317340 (LWP 101876)]
>>>> [New Thread 810399500 (LWP 101875)]
>>>> [New Thread 80b28a540 (LWP 101923)]
>>>> [New Thread 815a2c940 (LWP 101908)]
>>>> [New Thread 810d32c40 (LWP 101660)]
>>>> [New Thread 80bc0ee40 (LWP 101903)]
>>>> [New Thread 80e76b9c0 (LWP 101902)]
>>>> [New Thread 80ee30540 (LWP 100341)]
>>>> [New Thread 81398e280 (LWP 101872)]
>>>> [New Thread 813c0dd40 (LWP 101405)]
>>>> [New Thread 80e76b800 (LWP 101836)]
>>>> [New Thread 80b288b00 (LWP 101855)]
>>>> [New Thread 80cd50480 (LWP 101849)]
>>>> [New Thread 810f37040 (LWP 101794)]
>>>> [New Thread 80b570c80 (LWP 101741)]
>>>> [New Thread 80d010540 (LWP 101410)]
>>>> [New Thread 8170ab700 (LWP 101383)]
>>>> [New Thread 805175d00 (LWP 101366)]
>>>> [New Thread 805c59b00 (LWP 101345)]
>>>> [New Thread 8048abc80 (LWP 100576)]
>>>> [New Thread 810d32a80 (LWP 101596)]
>>>> [New Thread 805b15780 (LWP 101362)]
>>>> [New Thread 80ef1fe40 (LWP 101250)]
>>>> [New Thread 80555ab80 (LWP 101176)]
>>>> [New Thread 80d010380 (LWP 100934)]
>>>> [New Thread 8163ccf40 (LWP 101222)]
>>>> [New Thread 803852700 (LWP 101094)]
>>>> [New Thread 80f15c700 (LWP 100954)]
>>>> [New Thread 80b7b4380 (LWP 100768)]
>>>> [New Thread 803f82380 (LWP 101404)]
>>>> [New Thread 80554a400 (LWP 102153)]
>>>> [New Thread 80d14f8c0 (LWP 102151)]
>>>> [New Thread 8158dde00 (LWP 102148)]
>>>> [New Thread 80bda0100 (LWP 102146)]
>>>> [New Thread 810597900 (LWP 101265)]
>>>> [New Thread 815f2e700 (LWP 102142)]
>>>> [New Thread 8158ddc40 (LWP 102068)]
>>>> [New Thread 8077d0c00 (LWP 102138)]
>>>> [New Thread 80e16f480 (LWP 101785)]
>>>> [New Thread 811317180 (LWP 102134)]
>>>> [New Thread 809715c40 (LWP 102125)]
>>>> [New Thread 8055e0800 (LWP 101736)]
>>>> [New Thread 80b7b4700 (LWP 102122)]
>>>> [New Thread 807e42700 (LWP 101769)]
>>>> [New Thread 8163ccd80 (LWP 101752)]
>>>> [New Thread 8051c7780 (LWP 101738)]
>>>> [New Thread 80739c700 (LWP 102121)]
>>>> [New Thread 80b44f700 (LWP 102119)]
>>>> [New Thread 80dde0e40 (LWP 101713)]
>>>> [New Thread 816658740 (LWP 101695)]
>>>> [New Thread 80544f780 (LWP 101691)]
>>>> [New Thread 80b288780 (LWP 101378)]
>>>> [New Thread 80f2bd740 (LWP 102110)]
>>>> [New Thread 80bdcb2c0 (LWP 101689)]
>>>> [New Thread 8051c75c0 (LWP 102106)]
>>>> [New Thread 80b28c840 (LWP 102102)]
>>>> [New Thread 80eec9380 (LWP 102101)]
>>>> [New Thread 809c75700 (LWP 102099)]
>>>> [New Thread 80bdcb100 (LWP 102095)]
>>>> [New Thread 808500340 (LWP 101321)]
>>>> [New Thread 8051c7400 (LWP 102093)]
>>>> [New Thread 8051c7080 (LWP 102091)]
>>>> [New Thread 8063aa580 (LWP 102089)]
>>>> [New Thread 8134c5740 (LWP 102088)]
>>>> [New Thread 811efcd40 (LWP 102085)]
>>>> [New Thread 80bd9ff40 (LWP 102084)]
>>>> [New Thread 80b28ae00 (LWP 101942)]
>>>> [New Thread 80ce5ff00 (LWP 101941)]
>>>> [New Thread 80b28a8c0 (LWP 101607)]
>>>> [New Thread 815b9e780 (LWP 102078)]
>>>> [New Thread 80ccb4a00 (LWP 102070)]
>>>> [New Thread 8055497c0 (LWP 101939)]
>>>> [New Thread 809990fc0 (LWP 101937)]
>>>> [New Thread 805060980 (LWP 101938)]
>>>> [New Thread 805547d80 (LWP 102069)]
>>>> [New Thread 80ccb4840 (LWP 102067)]
>>>> [New Thread 80a11d700 (LWP 102064)]
>>>> [New Thread 8134cc900 (LWP 102058)]
>>>> [New Thread 8098b7a80 (LWP 101848)]
>>>> [New Thread 80a450a80 (LWP 101435)]
>>>> [New Thread 80a445540 (LWP 102057)]
>>>> [New Thread 8052f6700 (LWP 102055)]
>>>> [New Thread 80544cd80 (LWP 102053)]
>>>> [New Thread 8134cc740 (LWP 102051)]
>>>> [New Thread 803f398c0 (LWP 102049)]
>>>> [New Thread 808d27340 (LWP 101671)]
>>>> [New Thread 815aa7940 (LWP 102023)]
>>>> [New Thread 80705cf00 (LWP 102009)]
>>>> [New Thread 8134cc580 (LWP 102044)]
>>>> [New Thread 80ccad4c0 (LWP 102042)]
>>>> [New Thread 804a07c00 (LWP 100830)]
>>>> [New Thread 806daa740 (LWP 102038)]
>>>> [New Thread 803ebdac0 (LWP 102036)]
>>>> [New Thread 815aa7780 (LWP 102033)]
>>>> [New Thread 8155e3d80 (LWP 102031)]
>>>> [New Thread 80ccad300 (LWP 102029)]
>>>> [New Thread 80a445700 (LWP 101889)]
>>>> [New Thread 806961540 (LWP 102024)]
>>>> [New Thread 805e048c0 (LWP 102022)]
>>>> [New Thread 80ccad140 (LWP 102020)]
>>>> [New Thread 80ccacf80 (LWP 102018)]
>>>> [New Thread 80678bd40 (LWP 102016)]
>>>> [New Thread 80a2f6e00 (LWP 101854)]
>>>> [New Thread 814ed6a80 (LWP 102011)]
>>>> [New Thread 8155dbd80 (LWP 102010)]
>>>> [New Thread 80dddb340 (LWP 102007)]
>>>> [New Thread 80b28c300 (LWP 102003)]
>>>> [New Thread 8109e98c0 (LWP 101858)]
>>>> [New Thread 80b167fc0 (LWP 102000)]
>>>> [New Thread 804a45dc0 (LWP 101999)]
>>>> [New Thread 80bcf8100 (LWP 101995)]
>>>> [New Thread 804b355c0 (LWP 101881)]
>>>> [New Thread 8109f8500 (LWP 101882)]
>>>> [New Thread 8109f1340 (LWP 101991)]
>>>> [New Thread 8155dbbc0 (LWP 100737)]
>>>> [New Thread 804bfb0c0 (LWP 100519)]
>>>> [New Thread 80b7f88c0 (LWP 101698)]
>>>> [New Thread 80692b900 (LWP 101587)]
>>>> [New Thread 80705c9c0 (LWP 101500)]
>>>> [New Thread 814705cc0 (LWP 101034)]
>>>> [New Thread 806c3bd40 (LWP 100759)]
>>>> [New Thread 80b3d6c00 (LWP 101411)]
>>>> [New Thread 804bfaf00 (LWP 101987)]
>>>> [New Thread 80842be40 (LWP 101981)]
>>>> [New Thread 804f7a380 (LWP 101408)]
>>>> [New Thread 808500180 (LWP 101401)]
>>>> [New Thread 814ed68c0 (LWP 101980)]
>>>> [New Thread 808d56a40 (LWP 101400)]
>>>> [New Thread 80bc0ad80 (LWP 101385)]
>>>> [New Thread 81191fac0 (LWP 101973)]
>>>> [New Thread 815243100 (LWP 101966)]
>>>> [New Thread 806f66900 (LWP 101381)]
>>>> [New Thread 804bfad40 (LWP 101380)]
>>>> [New Thread 804bfab80 (LWP 101965)]
>>>> [New Thread 80b310500 (LWP 101952)]
>>>> [New Thread 814b4e2c0 (LWP 101364)]
>>>> [New Thread 804ad9e80 (LWP 101951)]
>>>> [New Thread 805aceec0 (LWP 101357)]
>>>> [New Thread 8100ef380 (LWP 101353)]
>>>> [New Thread 80a083900 (LWP 101950)]
>>>> [New Thread 80b310340 (LWP 101936)]
>>>> [New Thread 80b310180 (LWP 101934)]
>>>> [New Thread 804bfa9c0 (LWP 101346)]
>>>> [New Thread 804b32f40 (LWP 101343)]
>>>> [New Thread 803852c40 (LWP 101341)]
>>>> [New Thread 80544fb00 (LWP 101335)]
>>>> [New Thread 805558f80 (LWP 101337)]
>>>> [New Thread 803854d80 (LWP 101333)]
>>>> [New Thread 804bf8180 (LWP 101318)]
>>>> [New Thread 804b32d80 (LWP 101314)]
>>>> [New Thread 810b2ce00 (LWP 101272)]
>>>> [New Thread 80ce5e300 (LWP 100592)]
>>>> [New Thread 807128a80 (LWP 101084)]
>>>> [New Thread 8047a31c0 (LWP 100999)]
>>>> [New Thread 80b289580 (LWP 100912)]
>>>> [New Thread 808172180 (LWP 101887)]
>>>> [New Thread 80bc7f2c0 (LWP 101300)]
>>>> [New Thread 80bc7f100 (LWP 101870)]
>>>> [New Thread 80bc7ef40 (LWP 101298)]
>>>> [New Thread 804b32bc0 (LWP 101288)]
>>>> [New Thread 814688e80 (LWP 100956)]
>>>> [New Thread 807340540 (LWP 101835)]
>>>> [New Thread 80b288080 (LWP 101290)]
>>>> [New Thread 805545540 (LWP 101826)]
>>>> [New Thread 814688cc0 (LWP 101275)]
>>>> [New Thread 80bc7ed80 (LWP 101781)]
>>>> [New Thread 80660e0c0 (LWP 101270)]
>>>> [New Thread 80554bc80 (LWP 101744)]
>>>> [New Thread 805bcc500 (LWP 101701)]
>>>> [New Thread 80b2d1700 (LWP 101690)]
>>>> [New Thread 810b2c8c0 (LWP 101651)]
>>>> [New Thread 80554b200 (LWP 101223)]
>>>> [New Thread 8055482c0 (LWP 101230)]
>>>> [New Thread 8038552c0 (LWP 101229)]
>>>> [New Thread 803e2fac0 (LWP 101228)]
>>>> [New Thread 813e28c80 (LWP 101573)]
>>>> [New Thread 808d56880 (LWP 101546)]
>>>> [New Thread 808d566c0 (LWP 101190)]
>>>> [New Thread 804ad6300 (LWP 101451)]
>>>> [New Thread 805d56600 (LWP 101339)]
>>>> [New Thread 803f3f580 (LWP 101179)]
>>>> [New Thread 804ad6140 (LWP 101175)]
>>>> [New Thread 813e28ac0 (LWP 101332)]
>>>> [New Thread 8100ef8c0 (LWP 101164)]
>>>> [New Thread 805172340 (LWP 101178)]
>>>> [New Thread 80813ce40 (LWP 101330)]
>>>> [New Thread 8143c7640 (LWP 100951)]
>>>> [New Thread 8100ef540 (LWP 100939)]
>>>> [New Thread 814447480 (LWP 100695)]
>>>> [New Thread 8055586c0 (LWP 101269)]
>>>> [New Thread 813c52d40 (LWP 101152)]
>>>> [New Thread 80a11ea40 (LWP 101210)]
>>>> [New Thread 80739cfc0 (LWP 101532)]
>>>> [New Thread 80717efc0 (LWP 101360)]
>>>> [New Thread 80a868700 (LWP 101442)]
>>>> [New Thread 8097d84c0 (LWP 101354)]
>>>> [New Thread 806da5500 (LWP 101865)]
>>>> [New Thread 806da56c0 (LWP 101859)]
>>>> [New Thread 806da5880 (LWP 101843)]
>>>> [New Thread 806da5a40 (LWP 101837)]
>>>> [New Thread 806da5dc0 (LWP 101808)]
>>>> [New Thread 806da5f80 (LWP 101783)]
>>>> [New Thread 806da6140 (LWP 101761)]
>>>> [New Thread 806da6300 (LWP 101723)]
>>>> [New Thread 806da64c0 (LWP 101709)]
>>>> [New Thread 806da6680 (LWP 101685)]
>>>> [New Thread 806da6840 (LWP 101673)]
>>>> [New Thread 806da6a00 (LWP 101634)]
>>>> [New Thread 806da6bc0 (LWP 101622)]
>>>> [New Thread 806da6d80 (LWP 101608)]
>>>> [New Thread 806da6f40 (LWP 101600)]
>>>> [New Thread 806da7100 (LWP 101599)]
>>>> [New Thread 806da72c0 (LWP 101569)]
>>>> [New Thread 806da7800 (LWP 101547)]
>>>> [New Thread 806da79c0 (LWP 101524)]
>>>> [New Thread 806da7b80 (LWP 101516)]
>>>> [New Thread 806da7d40 (LWP 101514)]
>>>> [New Thread 806da7f00 (LWP 101468)]
>>>> [New Thread 806da8440 (LWP 101416)]
>>>> [New Thread 806da8600 (LWP 101414)]
>>>> [New Thread 806da87c0 (LWP 101395)]
>>>> [New Thread 806da8980 (LWP 101393)]
>>>> [New Thread 806da8b40 (LWP 101377)]
>>>> [New Thread 8084ffa80 (LWP 101327)]
>>>> [New Thread 8052fcc80 (LWP 101323)]
>>>> [New Thread 804ad4fc0 (LWP 101311)]
>>>> [New Thread 8055e1b40 (LWP 101303)]
>>>> [New Thread 80bda3040 (LWP 100685)]
>>>> [New Thread 80b4db380 (LWP 101280)]
>>>> [New Thread 8070e8c80 (LWP 101208)]
>>>> [New Thread 8068fa8c0 (LWP 101273)]
>>>> [New Thread 805545e00 (LWP 101244)]
>>>> [New Thread 80973f700 (LWP 101218)]
>>>> [New Thread 808d57d80 (LWP 101148)]
>>>> [New Thread 80b28c680 (LWP 101162)]
>>>> [New Thread 80b5221c0 (LWP 101158)]
>>>> [New Thread 8087bde00 (LWP 101079)]
>>>> [New Thread 80c8c5100 (LWP 101003)]
>>>> [New Thread 80bdcde80 (LWP 101213)]
>>>> [New Thread 80b28b880 (LWP 101011)]
>>>> [New Thread 804e7f540 (LWP 101000)]
>>>> [New Thread 80bd26580 (LWP 100751)]
>>>> [New Thread 804f63e40 (LWP 100943)]
>>>> [New Thread 803f83180 (LWP 101823)]
>>>> [New Thread 80978b340 (LWP 101525)]
>>>> [New Thread 805450200 (LWP 101961)]
>>>> [New Thread 8055474c0 (LWP 101960)]
>>>> [New Thread 80b28bc00 (LWP 101958)]
>>>> [New Thread 80bc0a140 (LWP 101955)]
>>>> [New Thread 80582ee40 (LWP 101390)]
>>>> [New Thread 80b289ac0 (LWP 101928)]
>>>> [New Thread 80544d640 (LWP 101320)]
>>>> [New Thread 805dab280 (LWP 101384)]
>>>> [New Thread 80bc09f80 (LWP 101921)]
>>>> [New Thread 80544e0c0 (LWP 101895)]
>>>> [New Thread 804a06c40 (LWP 100625)]
>>>> [New Thread 80b287d00 (LWP 101894)]
>>>> [New Thread 80ef191c0 (LWP 101890)]
>>>> [New Thread 80876b540 (LWP 101883)]
>>>> [New Thread 807340700 (LWP 101871)]
>>>> [New Thread 80c081c00 (LWP 101863)]
>>>> [New Thread 80544eb40 (LWP 101805)]
>>>> [New Thread 80be8e700 (LWP 101361)]
>>>> [New Thread 806ea3c80 (LWP 101798)]
>>>> [New Thread 80a07f140 (LWP 101356)]
>>>> [New Thread 804f7aa80 (LWP 101787)]
>>>> [New Thread 80876ba80 (LWP 101745)]
>>>> [New Thread 804e85c80 (LWP 101313)]
>>>> [New Thread 8096ddc40 (LWP 101729)]
>>>> [New Thread 80f34c580 (LWP 101325)]
>>>> [New Thread 80ecafac0 (LWP 101665)]
>>>> [New Thread 807f58540 (LWP 101652)]
>>>> [New Thread 80705d440 (LWP 101949)]
>>>> [New Thread 80b30f8c0 (LWP 101944)]
>>>> [New Thread 81191f580 (LWP 101643)]
>>>> [New Thread 809c758c0 (LWP 101751)]
>>>> [New Thread 80a11fa00 (LWP 101790)]
>>>> [New Thread 805174800 (LWP 101946)]
>>>> [New Thread 80b30f700 (LWP 101530)]
>>>> [New Thread 80b28b340 (LWP 101631)]
>>>> [New Thread 8097d7a40 (LWP 101284)]
>>>> [New Thread 80974fe00 (LWP 101515)]
>>>> [New Thread 80973fe00 (LWP 101548)]
>>>> [New Thread 804a07dc0 (LWP 101557)]
>>>> [New Thread 805450e40 (LWP 101534)]
>>>> [New Thread 80a868540 (LWP 101904)]
>>>> [New Thread 80ce5fd40 (LWP 101899)]
>>>> [New Thread 80554a940 (LWP 101766)]
>>>> [New Thread 80b30f540 (LWP 101285)]
>>>> [New Thread 80b49b540 (LWP 101719)]
>>>> [New Thread 80544d100 (LWP 101874)]
>>>> [New Thread 80472d8c0 (LWP 101386)]
>>>> [New Thread 80554a080 (LWP 101310)]
>>>> [New Thread 8098f98c0 (LWP 101615)]
>>>> [New Thread 8047a3540 (LWP 101564)]
>>>> [New Thread 804b35cc0 (LWP 101589)]
>>>> [New Thread 8049ae1c0 (LWP 101319)]
>>>> [New Thread 805559d80 (LWP 101550)]
>>>> [New Thread 80631c740 (LWP 101884)]
>>>> [New Thread 805549440 (LWP 101142)]
>>>> [New Thread 804a45f80 (LWP 100636)]
>>>> [New Thread 80ce5fb80 (LWP 100582)]
>>>> [New Thread 808d27180 (LWP 101926)]
>>>> [New Thread 8097d76c0 (LWP 101929)]
>>>> [New Thread 8097d7500 (LWP 101920)]
>>>> [New Thread 807b72540 (LWP 101627)]
>>>> [New Thread 80c0f7bc0 (LWP 101852)]
>>>> [New Thread 806da8d00 (LWP 101851)]
>>>> [New Thread 806da8ec0 (LWP 101847)]
>>>> [New Thread 80c082d80 (LWP 101846)]
>>>> [New Thread 806da9080 (LWP 101845)]
>>>> [New Thread 806da9240 (LWP 101842)]
>>>> [New Thread 80a11fbc0 (LWP 101841)]
>>>> [New Thread 806da9400 (LWP 101840)]
>>>> [New Thread 806da95c0 (LWP 101838)]
>>>> [New Thread 80a10a840 (LWP 101834)]
>>>> [New Thread 806da9780 (LWP 101833)]
>>>> [New Thread 806da9940 (LWP 101831)]
>>>> [New Thread 80a07f300 (LWP 101830)]
>>>> [New Thread 806da9b00 (LWP 101829)]
>>>> [New Thread 806da9cc0 (LWP 101828)]
>>>> [New Thread 8077d0dc0 (LWP 101825)]
>>>> [New Thread 806da9e80 (LWP 101824)]
>>>> [New Thread 806daa040 (LWP 101821)]
>>>> [New Thread 8077c9880 (LWP 101820)]
>>>> [New Thread 806daa200 (LWP 101818)]
>>>> [New Thread 806daa3c0 (LWP 101815)]
>>>> [New Thread 806c3c980 (LWP 101813)]
>>>> [New Thread 806c3cb40 (LWP 101812)]
>>>> [New Thread 806c3cd00 (LWP 101809)]
>>>> [New Thread 80bf4a700 (LWP 101807)]
>>>> [New Thread 806c3cec0 (LWP 101806)]
>>>> [New Thread 806c3d080 (LWP 101804)]
>>>> [New Thread 806c3d240 (LWP 101803)]
>>>> [New Thread 806c3d400 (LWP 101799)]
>>>> [New Thread 80be8e8c0 (LWP 101797)]
>>>> [New Thread 806c3d5c0 (LWP 101795)]
>>>> [New Thread 806c3d780 (LWP 101793)]
>>>> [New Thread 806c3d940 (LWP 101791)]
>>>> [New Thread 809c75a80 (LWP 101603)]
>>>> [New Thread 806c3db00 (LWP 101602)]
>>>> [New Thread 806c3dcc0 (LWP 101597)]
>>>> [New Thread 806c3de80 (LWP 101595)]
>>>> [New Thread 806c3e040 (LWP 101594)]
>>>> [New Thread 806c3e200 (LWP 101786)]
>>>> [New Thread 806c3e3c0 (LWP 101784)]
>>>> [New Thread 806c3e580 (LWP 101782)]
>>>> [New Thread 809df48c0 (LWP 101780)]
>>>> [New Thread 806c3e740 (LWP 101778)]
>>>> [New Thread 806c3e900 (LWP 101775)]
>>>> [New Thread 806fb8700 (LWP 101774)]
>>>> [New Thread 806c3eac0 (LWP 101773)]
>>>> [New Thread 806c3ec80 (LWP 101770)]
>>>> [New Thread 806f86540 (LWP 101768)]
>>>> [New Thread 806c3ee40 (LWP 101767)]
>>>> [New Thread 806cc31c0 (LWP 101763)]
>>>> [New Thread 80ecefac0 (LWP 101760)]
>>>> [New Thread 806cc3380 (LWP 101759)]
>>>> [New Thread 806cc3540 (LWP 101756)]
>>>> [New Thread 80ecafc80 (LWP 101754)]
>>>> [New Thread 806cc3700 (LWP 101753)]
>>>> [New Thread 806cc38c0 (LWP 101749)]
>>>> [New Thread 80b570e40 (LWP 101748)]
>>>> [New Thread 806cc3e00 (LWP 101747)]
>>>> [New Thread 806cc3fc0 (LWP 101746)]
>>>> [New Thread 806cc4180 (LWP 101742)]
>>>> [New Thread 806cc4340 (LWP 101740)]
>>>> [New Thread 8042abc80 (LWP 101735)]
>>>> [New Thread 806cc4500 (LWP 101732)]
>>>> [New Thread 80b56a1c0 (LWP 101731)]
>>>> [New Thread 806cc46c0 (LWP 101730)]
>>>> [New Thread 806cc4880 (LWP 101727)]
>>>> [New Thread 80b522380 (LWP 101726)]
>>>> [New Thread 806cc4a40 (LWP 101725)]
>>>> [New Thread 806cc4c00 (LWP 101722)]
>>>> [New Thread 80b4db540 (LWP 101721)]
>>>> [New Thread 806cc4dc0 (LWP 101720)]
>>>> [New Thread 806cc4f80 (LWP 101718)]
>>>> [New Thread 80b49b700 (LWP 101716)]
>>>> [New Thread 806cc5140 (LWP 101715)]
>>>> [New Thread 806cc5300 (LWP 101712)]
>>>> [New Thread 80b44f8c0 (LWP 101711)]
>>>> [New Thread 806cc54c0 (LWP 101710)]
>>>> [New Thread 806cc5680 (LWP 101707)]
>>>> [New Thread 8098f9a80 (LWP 101706)]
>>>> [New Thread 806cc5840 (LWP 101705)]
>>>> [New Thread 806cc5a00 (LWP 101702)]
>>>> [New Thread 8098b7c40 (LWP 101700)]
>>>> [New Thread 806cc5d80 (LWP 101699)]
>>>> [New Thread 806cc5f40 (LWP 101697)]
>>>> [New Thread 807264e00 (LWP 101694)]
>>>> [New Thread 806cc6100 (LWP 101693)]
>>>> [New Thread 806cc62c0 (LWP 101688)]
>>>> [New Thread 809856fc0 (LWP 101687)]
>>>> [New Thread 806cc6480 (LWP 101686)]
>>>> [New Thread 806cc6640 (LWP 101683)]
>>>> [New Thread 806cc6800 (LWP 101679)]
>>>> [New Thread 809991180 (LWP 101678)]
>>>> [New Thread 806cc69c0 (LWP 101677)]
>>>> [New Thread 806cc6b80 (LWP 101675)]
>>>> [New Thread 806cc6d40 (LWP 101672)]
>>>> [New Thread 8087bdfc0 (LWP 101669)]
>>>> [New Thread 806cc6f00 (LWP 101668)]
>>>> [New Thread 806cc7280 (LWP 101666)]
>>>> [New Thread 80876bfc0 (LWP 101664)]
>>>> [New Thread 806cc7980 (LWP 101662)]
>>>> [New Thread 806cc7b40 (LWP 101659)]
>>>> [New Thread 80739c8c0 (LWP 101654)]
>>>> [New Thread 806cc7d00 (LWP 101653)]
>>>> [New Thread 806cc7ec0 (LWP 101650)]
>>>> [New Thread 8073408c0 (LWP 101649)]
>>>> [New Thread 806cc8080 (LWP 101648)]
>>>> [New Thread 806cc8240 (LWP 101644)]
>>>> [New Thread 8068faa80 (LWP 101641)]
>>>> [New Thread 808d57f40 (LWP 101640)]
>>>> [New Thread 808d58100 (LWP 101636)]
>>>> [New Thread 80ea0dc40 (LWP 101633)]
>>>> [New Thread 808d58480 (LWP 101632)]
>>>> [New Thread 808d58640 (LWP 101628)]
>>>> [New Thread 806961c40 (LWP 101625)]
>>>> [New Thread 808d58800 (LWP 101623)]
>>>> [New Thread 808d589c0 (LWP 101621)]
>>>> [New Thread 806925540 (LWP 101618)]
>>>> [New Thread 808d58b80 (LWP 101617)]
>>>> [New Thread 808d58d40 (LWP 101614)]
>>>> [New Thread 804f7ac40 (LWP 101613)]
>>>> [New Thread 80b286b80 (LWP 101612)]
>>>> [New Thread 80b286f00 (LWP 101606)]
>>>> [New Thread 80b2870c0 (LWP 101605)]
>>>> [New Thread 80b287280 (LWP 101604)]
>>>> [New Thread 805f83380 (LWP 101556)]
>>>> [New Thread 80b287440 (LWP 101555)]
>>>> [New Thread 804e85e40 (LWP 101553)]
>>>> [New Thread 80b287600 (LWP 101552)]
>>>> [New Thread 80b287980 (LWP 101551)]
>>>> [New Thread 804183a80 (LWP 101541)]
>>>> [New Thread 804ad5a40 (LWP 101538)]
>>>> [New Thread 805557c40 (LWP 101536)]
>>>> [New Thread 805558c00 (LWP 101533)]
>>>> [New Thread 803f828c0 (LWP 101529)]
>>>> [New Thread 808d58f00 (LWP 101528)]
>>>> [New Thread 804b35b00 (LWP 101526)]
>>>> [New Thread 80554b900 (LWP 101523)]
>>>> [New Thread 803853dc0 (LWP 101520)]
>>>> [New Thread 80b2869c0 (LWP 101517)]
>>>> [New Thread 80554acc0 (LWP 101506)]
>>>> [New Thread 806da80c0 (LWP 101503)]
>>>> [New Thread 805548f00 (LWP 101473)]
>>>> [New Thread 805176240 (LWP 101470)]
>>>> [New Thread 8049ae380 (LWP 101469)]
>>>> [New Thread 805549ec0 (LWP 101458)]
>>>> [New Thread 8084ff1c0 (LWP 101456)]
>>>> [New Thread 803633b00 (LWP 101455)]
>>>> [New Thread 80739c540 (LWP 101449)]
>>>> [New Thread 807791ac0 (LWP 101448)]
>>>> [New Thread 805f2fe40 (LWP 101445)]
>>>> [New Thread 805ca17c0 (LWP 101443)]
>>>> [New Thread 805545fc0 (LWP 101441)]
>>>> [New Thread 805547bc0 (LWP 101436)]
>>>> [New Thread 80705b140 (LWP 100596)]
>>>> [New Thread 803853340 (LWP 101429)]
>>>> [New Thread 803f88580 (LWP 101424)]
>>>> [New Thread 806cc3a80 (LWP 101422)]
>>>> [New Thread 803e7e900 (LWP 100710)]
>>>> [New Thread 805c556c0 (LWP 101425)]
>>>> [New Thread 8063e7200 (LWP 101413)]
>>>> [New Thread 803852540 (LWP 101180)]
>>>> [New Thread 803856d00 (LWP 100620)]
>>>> [New Thread 803f883c0 (LWP 101412)]
>>>> [New Thread 80554b740 (LWP 100927)]
>>>> [New Thread 80631c900 (LWP 101403)]
>>>> [New Thread 80544d2c0 (LWP 101081)]
>>>> [New Thread 803f39380 (LWP 101402)]
>>>> [New Thread 8054503c0 (LWP 101392)]
>>>> [New Thread 80712c600 (LWP 101163)]
>>>> [New Thread 80a1088c0 (LWP 101149)]
>>>> [New Thread 803634040 (LWP 101387)]
>>>> [New Thread 80544cf40 (LWP 101372)]
>>>> [New Thread 8055489c0 (LWP 101370)]
>>>> [New Thread 803634c80 (LWP 101368)]
>>>> [New Thread 805546a40 (LWP 101072)]
>>>> [New Thread 806d5a280 (LWP 100608)]
>>>> [New Thread 805548d40 (LWP 100587)]
>>>> [New Thread 80b28d9c0 (LWP 101367)]
>>>> [New Thread 803853500 (LWP 101365)]
>>>> [New Thread 803856280 (LWP 101202)]
>>>> [New Thread 8070e44c0 (LWP 101355)]
>>>> [New Thread 80555bb40 (LWP 101188)]
>>>> [New Thread 805547140 (LWP 101185)]
>>>> [New Thread 80b289740 (LWP 101324)]
>>>> [New Thread 805557380 (LWP 101316)]
>>>> [New Thread 80bc081c0 (LWP 101154)]
>>>> [New Thread 8051c7240 (LWP 101315)]
>>>> [New Thread 805450580 (LWP 101114)]
>>>> [New Thread 8038536c0 (LWP 101093)]
>>>> [New Thread 806f861c0 (LWP 100929)]
>>>> [New Thread 8097dc740 (LWP 100931)]
>>>> [New Thread 80544d9c0 (LWP 100918)]
>>>> [New Thread 8041831c0 (LWP 100876)]
>>>> [New Thread 80a07d700 (LWP 100493)]
>>>> [New Thread 80544eec0 (LWP 100742)]
>>>> [New Thread 806cc7600 (LWP 101309)]
>>>> [New Thread 80b28cbc0 (LWP 101304)]
>>>> [New Thread 80544e440 (LWP 101302)]
>>>> [New Thread 80b28b500 (LWP 101277)]
>>>> [New Thread 80b289900 (LWP 101086)]
>>>> [New Thread 80712bd40 (LWP 100767)]
>>>> [New Thread 80b2893c0 (LWP 101276)]
>>>> [New Thread 804010e40 (LWP 101092)]
>>>> [New Thread 805c58980 (LWP 100941)]
>>>> [New Thread 804a4a040 (LWP 100677)]
>>>> [New Thread 806cc70c0 (LWP 100681)]
>>>> [New Thread 8077c88c0 (LWP 101001)]
>>>> [New Thread 804a456c0 (LWP 101271)]
>>>> [New Thread 8047a9ac0 (LWP 101214)]
>>>> [New Thread 804a07500 (LWP 100817)]
>>>> [New Thread 80a11d8c0 (LWP 101240)]
>>>> [New Thread 80554be40 (LWP 101233)]
>>>> [New Thread 806214c80 (LWP 101166)]
>>>> [New Thread 805548800 (LWP 101171)]
>>>> [New Thread 805546c00 (LWP 101169)]
>>>> [New Thread 8097d6540 (LWP 101174)]
>>>> [New Thread 805557e00 (LWP 101193)]
>>>> [New Thread 80bd70200 (LWP 101115)]
>>>> [New Thread 805549280 (LWP 101132)]
>>>> [New Thread 80b28aa80 (LWP 101181)]
>>>> [New Thread 80544fcc0 (LWP 101105)]
>>>> [New Thread 803633780 (LWP 101017)]
>>>> [New Thread 80505d180 (LWP 101088)]
>>>> [New Thread 806d57180 (LWP 101019)]
>>>> [New Thread 803634ac0 (LWP 100994)]
>>>> [New Thread 803855800 (LWP 100972)]
>>>> [New Thread 80544e7c0 (LWP 100787)]
>>>> [New Thread 803856980 (LWP 100747)]
>>>> [New Thread 8084ff540 (LWP 100740)]
>>>> [New Thread 80778b1c0 (LWP 100679)]
>>>> [New Thread 80544f5c0 (LWP 100815)]
>>>> [New Thread 80b28bf80 (LWP 101611)]
>>>> [New Thread 80554a240 (LWP 101542)]
>>>> [New Thread 80b288940 (LWP 101680)]
>>>> [New Thread 80b289040 (LWP 101674)]
>>>> [New Thread 80b2d3f40 (LWP 101670)]
>>>> [New Thread 80b28a700 (LWP 101656)]
>>>> [New Thread 80978a540 (LWP 101626)]
>>>> [New Thread 80a109880 (LWP 101610)]
>>>> [New Thread 806da5c00 (LWP 101544)]
>>>> [New Thread 805176400 (LWP 101444)]
>>>> [New Thread 806da8280 (LWP 101432)]
>>>> [New Thread 807553d40 (LWP 100921)]
>>>> [New Thread 8036335c0 (LWP 101359)]
>>>> [New Thread 805557a80 (LWP 101336)]
>>>> [New Thread 80554a780 (LWP 101658)]
>>>> [New Thread 808d29f00 (LWP 101635)]
>>>> [New Thread 805547680 (LWP 100561)]
>>>> [New Thread 803855100 (LWP 101382)]
>>>> [New Thread 805546500 (LWP 101373)]
>>>> [New Thread 803633940 (LWP 100760)]
>>>> [New Thread 8038567c0 (LWP 101278)]
>>>> [New Thread 807553800 (LWP 101242)]
>>>> [New Thread 8070080c0 (LWP 101340)]
>>>> [New Thread 80555bd00 (LWP 101289)]
>>>> [New Thread 803857080 (LWP 100473)]
>>>> [New Thread 803857240 (LWP 100357)]
>>>> [New Thread 802a041c0 (LWP 100394)]
>>>> [Switching to Thread 81ad04d40 (LWP 102349)]
>>>> 0x0000000801ddbfea in read () from /lib/libc.so.7
>>>> $1 = 1953067639
>>>> $2 = 44097640
>>>> $3 = 44085944
>>>> $4 = 47441064
>>>> $5 = 14848638
>>>> $6 = 14848681
>>>> $7 = 14848706
>>>> $8 = 14848669
>>>> $9 = 1953067639
>>>> $10 = 14848661
>>>> /usr/local/share/bacula/btraceback.gdb:11: Error in sourced command file:
>>>> No symbol table is loaded. Use the "file" command.
>>>>
>>>> I'm still having some problems getting enough information out of the
>>>> btraceback script, both bacula-dir and sd are running with "-d 3", but I
>>>> don;t know if that is an appropriate level. Also, I have to use gdb 7.2
>>>> from Ports, the build in 6.x version of GDB would just hang, waiting for
>>>> input ( some sort of y/n question ).
>>>>
>>>> I'm sorry for the long email, please let me know if more or less
>>>> information would be useful, I'd like to understand how I can properly
>>>> throttle Bacula, or find out why it is crashing.
>>>>
>>>> Thanks!
>>>> Mike C
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Colocation vs. Managed Hosting
>>>> A question and answer guide to determining the best fit
>>>> for your organization - today and in the future.
>>>> http://p.sf.net/sfu/internap-sfd2d
>>>> _______________________________________________
>>>> Bacula-users mailing list
>>>> Bacula-users AT lists.sourceforge DOT net
>>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>>
>>> --
>>> Dan Langille - http://langille.org/
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> Message: 8
>>> Date: Mon, 21 Mar 2011 14:06:14 -0700
>>> From: Mark Nienberg<gmane AT tippingmar DOT com>
>>> Subject: Re: [Bacula-users] File Set configuration and excluding some
>>> files?
>>> To: bacula-users AT lists.sourceforge DOT net
>>> Message-ID:<im8ek6$27l$1 AT dough.gmane DOT org>
>>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>>
>>> On 3/20/2011 5:52 PM, Mike Eggleston wrote:
>>>> Afternoon,
>>>>
>>>> I'm trying to get some spam email folders excluded in the backup. I
>>>> wonder if there is an ordering issue I'm missing. For this unix server
>>>> I have a:
>>>>
>>>> File Set {
>>>> Name = Unix-Files-elo
>>>> Ignore Fileset Changes = yes
>>>> Include {
>>>> Options {
>>>> signature = MD5
>>>> Exclude = yes
>>>> IgnoreCase = yes
>>>>
>>>> # Exclude files and directories that can be
>>>> replaced using the red hat network
>>>> WildDir = "/var/spool/imap/*/user/*/spam"
>>>> }
>>>> File = /var/spool/imap
>>>> }
>>>> }
>>> Here is a similar working example from my backups:
>>>
>>> FileSet {
>>> Name = "gecko Files"
>>> Ignore FileSet Changes = yes
>>> Include {
>>> Options {
>>> signature = MD5
>>> wildDir = "/share/office/TmpShare/*/personal"
>>> wildDir = "/share/homes/*/thunderbird/*/ImapMail"
>>> wildDir = "/share/drawings/*dwgs/*/Received"
>>> ignore case = yes #so the above will match Personal, etc
>>> Exclude = yes
>>> }
>>> File = /etc
>>> File = /root
>>> File = /share
>>> } # end Include
>>> } # end FileSet
>>>
>>> I wonder if it is a problem with Ignore Case? The documentation is
>>> inconsistent about
>>> the space between Ignore and Case.
>>>
>>> --
>>> Mark Nienberg
>>> Sent from an invalid address. Please reply to the group.
>>>
>>>
>>>
>>>
>>> ------------------------------
>>>
>>> ------------------------------------------------------------------------------
>>> Enable your software for Intel(R) Active Management Technology to meet the
>>> growing manageability and security demands of your customers. Businesses
>>> are taking advantage of Intel(R) vPro (TM) technology - will your software
>>> be a part of the solution? Download the Intel(R) Manageability Checker
>>> today! http://p.sf.net/sfu/intel-dev2devmar
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Bacula-users mailing list
>>> Bacula-users AT lists.sourceforge DOT net
>>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>>>
>>>
>>> End of Bacula-users Digest, Vol 59, Issue 20
>>> ********************************************
>>>
>>>
>>> -----
>>> Zprava neobsahuje viry.
>>> Zkontrolovano AVG - www.avg.cz
>>> Verze: 10.0.1204 / Virova baze: 1498/3518 - Datum vydani: 20.3.2011
>>>
>>>
>>> .
>>>
>> ------------------------------------------------------------------------------
>> Enable your software for Intel(R) Active Management Technology to meet the
>> growing manageability and security demands of your customers. Businesses
>> are taking advantage of Intel(R) vPro (TM) technology - will your software
>> be a part of the solution? Download the Intel(R) Manageability Checker
>> today! http://p.sf.net/sfu/intel-dev2devmar
>> _______________________________________________
>> Bacula-users mailing list
>> Bacula-users AT lists.sourceforge DOT net
>> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 23 Mar 2011 23:18:47 -0400
> From: Dan Langille<dan AT langille DOT org>
> Subject: Re: [Bacula-users] waiting on max Storage jobs
> To: bacula-users AT lists.sourceforge DOT net
> Message-ID:<4D8AB817.9020802 AT langille DOT org>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> On 3/19/2011 11:29 AM, Dan Langille wrote:
>> On 3/4/2011 7:32 AM, Dan Langille wrote:
>>> This situation occurs often. Usually every morning. If I cancel the top
>>> job (in this ca 54502), the other jobs will proceed.
>>>
>>>
>>>
>>> Running Jobs:
>>> Console connected at 04-Mar-11 12:23
>>> JobId Level Name Status
>>> ======================================================================
>>> 54502 Differe CopyToTape-Diff.2011-03-04_08.32.04_23 is waiting on
>>> max Storage jobs
>>> 54503 Full CopyToTape-Full.2011-03-04_08.32.04_24 is waiting
>>> execution
>>> 54504 Full CopyToTape-Full.2011-03-04_08.32.04_25 is waiting for
>>> higher priority jobs to finish
>>> 54505 Increme nyi_maildir_tarball.2011-03-04_08.32.05_26 is waiting
>>> execution
>>> 54506 Full BackupCatalog.2011-03-04_08.32.05_27 is waiting execution
>>> ====
>>>
>>> My configuration is:
>>>
>>> http://www.langille.org/tmp/bacula-dir.conf
>>> http://www.langille.org/tmp/bacula-sd.conf
>>>
>>> Any clues as to why this hangs so often?
>> A clue.
>>
>> If the CopyToTape-Diff job finds no jobs to copy, the above situation
>> arises. If there is a job to copy, no problem occurs. Interesting, yet
>> very fricking annoying. :)
> My 'solution': change the schedule for the CopyToTape-Diff job so it is
> similar to the differentials schedule. Thus, it will always find a job
> to copy.
>
> I consider this a bug within Copy Job, but I do not have a simple test
> to reproduce it. Apart from my existing system. ;)
>
> --
> Dan Langille - http://langille.org/
>
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 24 Mar 2011 01:49:28 -0700
> From: farmershort<bacula-forum AT backupcentral DOT com>
> Subject: [Bacula-users] Bacula jobs causing VM server reboot&
> consequent job fail
> To: bacula-users AT lists.sourceforge DOT net
> Message-ID:<1300956568.m2f.351850 AT www.backupcentral DOT com>
>
> Hi All,
>
> I have, what I think, Is a fairly obscure bacula issue.
>
> my bacula-dir server is a CentOS 5.5 server, running as a VM on MS Hyper-V,
> on a windows 2008R2 host. I'm managed to successfully backup a number of
> windows servers, and I'm happy that all the client-fd stuff is setup
> correctly.
>
> however, I had scheduled the first big test of this new bacula system a
> couple of days ago - and asked it to back up about 350GB from each of 2
> servers - at the same time. Both of these backups should run concurrently,
> and back up to the same esata disk.
>
> I managed to get this to happen yesterday, by manually running both jobs from
> bconsole, but for some reason, whenever they run from schedule, then cause
> the VM to reboot. If you watch it happening, you see the VM go into "saving"
> mode, then "saved".... then some minutes later it reboots.
>
> What's also wierd, is that one of the jobs still finishes successfully, but
> the other one fails by timing out after just over 2 hours. the timeout errors
> just says: fatal error: error reading data from fd.
>
> Does anyone have any ideas where/how to start diagnosing this one?
>
> Thanks
>
> Adam
>
> +----------------------------------------------------------------------
> |This was sent by adam_scuba AT btinternet DOT com via Backup Central.
> |Forward SPAM to abuse AT backupcentral DOT com.
> +----------------------------------------------------------------------
>
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 24 Mar 2011 10:08:44 +0100
> From: Geert Stappers<Geert.Stappers AT vanadgroup DOT com>
> Subject: Re: [Bacula-users] bacula-dir.conf client config
> To: "bacula-users AT lists.sourceforge DOT net"
> <bacula-users AT lists.sourceforge DOT net>
> Message-ID:<20110324090843.GJ22009 AT tosher.capdata.vanadcc DOT nl>
> Content-Type: text/plain; charset="us-ascii"
>
> Op 20110318 om 09:43 schreef newtobacula:
>> Any idea bout the other questions I need help with?
>>
> Yes, it is farely well documented
> at http://www.catb.org/~esr/faqs/smart-questions.html
>
>
> I know that I should have sent this out of band,
> but the original poster is using a "webforum"
> so I have to use the mailinglist.
>
>
> Stappers
> sysadmin
>
>
>
> ------------------------------
>
> ------------------------------------------------------------------------------
> Enable your software for Intel(R) Active Management Technology to meet the
> growing manageability and security demands of your customers. Businesses
> are taking advantage of Intel(R) vPro (TM) technology - will your software
> be a part of the solution? Download the Intel(R) Manageability Checker
> today! http://p.sf.net/sfu/intel-dev2devmar
>
> ------------------------------
>
> _______________________________________________
> Bacula-users mailing list
> Bacula-users AT lists.sourceforge DOT net
> https://lists.sourceforge.net/lists/listinfo/bacula-users
>
>
> End of Bacula-users Digest, Vol 59, Issue 23
> ********************************************
>
>
> -----
> Zprava neobsahuje viry.
> Zkontrolovano AVG - www.avg.cz
> Verze: 10.0.1204 / Virova baze: 1498/3525 - Datum vydani: 23.3.2011
>
>
> .
>
------------------------------------------------------------------------------
Enable your software for Intel(R) Active Management Technology to meet the
growing manageability and security demands of your customers. Businesses
are taking advantage of Intel(R) vPro (TM) technology - will your software
be a part of the solution? Download the Intel(R) Manageability Checker
today! http://p.sf.net/sfu/intel-dev2devmar
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
|
