|
continuing problems - hung client, etc
2009-07-16 10:10:57
Continuation of previous initiated thread.
Server, Solaris 10/x86 amanda 2.6.1p1, I believe -20090227
Client, also 2.6.1p1, Solaris 10/Sparc.
Have reduced maxdumps to 3, client still hanging...
[lyra] ~ 117> ps -ef | grep amanda
amanda 9087 7436 0 - ? 0:00 <defunct>
amanda 7644 7436 0 - ? 0:00 <defunct>
amanda 8487 7436 0 - ? 0:13 <defunct>
amanda 8304 7436 0 - ? 0:00 <defunct>
amanda 7437 7436 0 - ? 0:04 <defunct>
amanda 7438 7436 0 - ? 0:00 <defunct>
amanda 8099 7436 0 - ? 0:00 <defunct>
amanda 9086 7436 0 - ? 0:02 <defunct>
amanda 7465 7436 0 - ? 0:00 <defunct>
amanda 7436 15684 0 20:00:01 ? 16:56
/usr/local/libexec/amanda/amandad
amanda 9521 7436 0 - ? 0:00 <defunct>
brian 12593 12573 0 09:39:24 pts/3 0:00 grep amanda
amanda 8488 7436 0 - ? 0:00 <defunct>
amanda 11972 7436 0 - ? 0:00 <defunct>
amanda 8303 7436 0 - ? 0:13 <defunct>
amanda 10597 7436 0 - ? 0:00 <defunct>
amanda 9520 7436 0 - ? 0:04 <defunct>
Not sure what to try next...
We are also seeing some partial dumps, is there a work around
for that ? Is there a switch or something I can set in the dumptype
to cause dump to continue, rather than failing when it doesn't
get input ?
Interestingly, frustraitingly, we are seeing a lot of open files,
dispite using snapshots. I'd contended with my peers here that
using snapshots doesn't quiess the partition, it mearly seems to
capture open files in an open state, it does not in and of itself
produce a more reliable backup in terms of backuping up static data.
I'd really like to run down the application in order to get a
good snapshot - or am I misinterpreting what I see ?
The dump of curie:/ always seems to fail, and there is nothing
special about it, ufsdump of the root of the server itself. We
always seem to have the same broken pipe...
As always, happy to attach logs, but the initial post was already
long and the logs are unlikely to differ from ones I send a week
or so ago.
----- Forwarded message from Amanda/Curie <amanda AT wadsworth DOT org> -----
Hostname: curie
Org : Curie
Config : curie
Date : July 15, 2009
These dumps were to tape Curie10.
The next 2 tapes Amanda expects to use are: Curie11, Curie12.
FAILURE DUMP SUMMARY:
lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number]
lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number]
lyra /db3 lev 0 FAILED [spawn /bin/gzip: dup2 err: Bad file number]
lyra /db3 lev 0 FAILED [too many dumper retry: "[request failed:
timeout waiting for REP]"]
curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number]
curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number]
mailserv /usr1 lev 1 FAILED [missing size line from sendbackup]
mailserv /usr1 lev 1 FAILED [missing size line from sendbackup]
mailserv /usr1 lev 1: partial taper:
STRANGE DUMP SUMMARY:
squidzone1 /sqcache1/var lev 1 STRANGE (see below)
squidzone2 /sqcache2/var lev 1 STRANGE (see below)
squidzone2 /sqcache2/squidguard lev 0 STRANGE (see below)
STATISTICS:
Total Full Incr.
-------- -------- --------
Estimate Time (hrs:min) 1:38
Run Time (hrs:min) 10:03
Dump Time (hrs:min) 22:20 13:41 8:39
Output Size (meg) 345339.0 151458.9 193880.1
Original Size (meg) 447849.3 209115.9 238733.5
Avg Compressed Size (%) 70.1 58.7 78.0 (level:#disks ...)
Filesystems Dumped 45 15 30 (1:30)
Avg Dump Rate (k/s) 4399.0 3149.5 6374.6
Tape Time (hrs:min) 8:09 3:59 4:09
Tape Size (meg) 330212.9 151458.9 178754.0
Tape Used (%) 41.1 18.9 22.3 (level:#disks ...)
Filesystems Taped 46 15 31 (1:31)
(level:#chunks ...)
Chunks Taped 46 15 31 (1:31)
Avg Tp Write Rate (k/s) 11526.5 10793.1 12230.6
USAGE BY TAPE:
Label Time Size % Nb Nc
Curie10 8:09 330213M 41.1 46 46
FAILED DUMP DETAILS:
/-- lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number]
sendbackup: start [lyra:/ level 1]
sendbackup: info BACKUP=/usr/sbin/ufsdump
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ...
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
sendbackup: error [spawn /bin/gzip: dup2 out: Bad file number]
| DUMP: Date of this level 1 dump: Wed Jul 15 20:20:39 2009
| DUMP: Date of last level 0 dump: Fri Jul 10 20:02:27 2009
| DUMP: Dumping /dev/rdsk/c0t0d0s4 (lyra:/) to standard output.
| DUMP: Mapping (Pass I) [regular files]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Writing 32 Kilobyte records
| DUMP: Estimated 3508630 blocks (1713.20MB) on 0.03 tapes.
? sendbackup: index tee cannot write [Broken pipe]
? DUMP: Write error 0 blocks into volume 1
? DUMP: Write error on standard output
| DUMP: Cannot recover
| DUMP: The ENTIRE dump is aborted.
? dump (8307) /usr/sbin/ufsdump returned 3
? index index returned 1
? compress (8305) compress returned 1
sendbackup: error [dump (8307) /usr/sbin/ufsdump returned 3, compress (8305)
compress returned 1]
\--------
/-- lyra / lev 1 FAILED [spawn /bin/gzip: dup2 out: Bad file number]
sendbackup: start [lyra:/ level 1]
sendbackup: error [spawn /bin/gzip: dup2 out: Bad file number]
sendbackup: info BACKUP=/usr/sbin/ufsdump
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ...
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
| DUMP: Date of this level 1 dump: Wed Jul 15 20:23:01 2009
| DUMP: Date of last level 0 dump: Fri Jul 10 20:02:27 2009
| DUMP: Dumping /dev/rdsk/c0t0d0s4 (lyra:/) to standard output.
| DUMP: Mapping (Pass I) [regular files]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Writing 32 Kilobyte records
| DUMP: Estimated 3508658 blocks (1713.21MB) on 0.03 tapes.
? sendbackup: index tee cannot write [Broken pipe]
? DUMP: Write error 0 blocks into volume 1
? DUMP: Write error on standard output
| DUMP: Cannot recover
| DUMP: The ENTIRE dump is aborted.
? dump (8493) /usr/sbin/ufsdump returned 3
? index index returned 1
? compress (8491) compress returned 1
sendbackup: error [dump (8493) /usr/sbin/ufsdump returned 3, compress (8491)
compress returned 1]
\--------
/-- lyra /db3 lev 0 FAILED [spawn /bin/gzip: dup2 err: Bad file number]
sendbackup: error [spawn /bin/gzip: dup2 err: Bad file number]
\--------
/-- curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number]
sendbackup: start [curie:/ level 0]
sendbackup: error [spawn /usr/bin/gzip: dup2 out: Bad file number]
sendbackup: info BACKUP=/usr/sbin/ufsdump
sendbackup: info RECOVER_CMD=/usr/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ...
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
| DUMP: Date of this level 0 dump: Wed Jul 15 21:13:09 2009
| DUMP: Date of last level 0 dump: the epoch
| DUMP: Dumping /dev/md/rdsk/d10 (curie:/) to standard output.
| DUMP: Mapping (Pass I) [regular files]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Writing 32 Kilobyte records
| DUMP: Estimated 15915568 blocks (7771.27MB) on 0.12 tapes.
? sendbackup: index tee cannot write [Broken pipe]
? DUMP: Write error 0 blocks into volume 1
? DUMP: Write error on standard output
| DUMP: Cannot recover
| DUMP: The ENTIRE dump is aborted.
? dump (5453) /usr/sbin/ufsdump returned 3
? index index returned 1
? compress (5451) compress returned 1
sendbackup: error [dump (5453) /usr/sbin/ufsdump returned 3, compress (5451)
compress returned 1]
\--------
/-- curie / lev 0 FAILED [spawn /usr/bin/gzip: dup2 out: Bad file number]
sendbackup: start [curie:/ level 0]
sendbackup: error [spawn /usr/bin/gzip: dup2 out: Bad file number]
sendbackup: info BACKUP=/usr/sbin/ufsdump
sendbackup: info RECOVER_CMD=/usr/bin/gzip -dc |/usr/sbin/ufsrestore -xpGf - ...
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
| DUMP: Date of this level 0 dump: Wed Jul 15 21:13:38 2009
| DUMP: Date of last level 0 dump: the epoch
| DUMP: Dumping /dev/md/rdsk/d10 (curie:/) to standard output.
| DUMP: Mapping (Pass I) [regular files]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Writing 32 Kilobyte records
| DUMP: Estimated 15915574 blocks (7771.28MB) on 0.12 tapes.
? sendbackup: index tee cannot write [Broken pipe]
? DUMP: Write error 0 blocks into volume 1
? DUMP: Write error on standard output
| DUMP: Cannot recover
| DUMP: The ENTIRE dump is aborted.
? index index returned 1
? compress (5468) compress returned 1
? dump (5471) /usr/sbin/ufsdump returned 3
sendbackup: error [compress (5468) compress returned 1, dump (5471)
/usr/sbin/ufsdump returned 3]
\--------
/-- mailserv /usr1 lev 1 FAILED [missing size line from sendbackup]
sendbackup: start [mailserv:/usr1 level 1]
sendbackup: info BACKUP=/usr/sbin/ufsdump
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -f... -
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
| DUMP: Date of this level 1 dump: Wed Jul 15 20:07:47 2009
| DUMP: Date of last level 0 dump: Tue Apr 28 18:48:36 2009
| DUMP: Dumping /dev/rdsk/c3t0d0s6 (mailserv:/usr1) to standard output.
| DUMP: Mapping (Pass I) [regular files]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Writing 32 Kilobyte records
| DUMP: Estimated 58390164 blocks (28510.82MB) on 0.42 tapes.
| DUMP: Dumping (Pass III) [directories]
| DUMP: Dumping (Pass IV) [regular files]
| DUMP: 7.70% done, finished in 1:59
| DUMP: 16.06% done, finished in 1:44
| DUMP: 25.73% done, finished in 1:26
| DUMP: 35.33% done, finished in 1:13
| DUMP: 44.51% done, finished in 1:02
| DUMP: 53.46% done, finished in 0:52
| DUMP: 64.81% done, finished in 0:38
| DUMP: 75.73% done, finished in 0:25
| DUMP: 87.42% done, finished in 0:12
| DUMP: Warning - block 1957714610 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 2363809498 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3704413812 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3436370122 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 1520493790 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3268468446 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 2593299082 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3368739956 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 1957087964 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 1956299474 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3872579292 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 1521273572 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3899951306 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 1519964902 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3637697742 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 2833151128 is beyond the end of `/dev/rdsk/c3t0d0s6'
<--- many lines removed --->
| DUMP: Warning - block 2694866122 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 2934736084 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 3672140910 is beyond the end of `/dev/rdsk/c3t0d0s6'
? DUMP: More than 32 block read errors from dump device `/dev/rdsk/c3t0d0s6'
| DUMP: NEEDS ATTENTION: Do you want to attempt to continue? ("yes" or "no")
DUMP: The ENTIRE dump is aborted.
??error [/usr/sbin/ufsdump returned 3]
? dumper: strange [missing size line from sendbackup]
\--------
/-- mailserv /usr1 lev 1 FAILED [missing size line from sendbackup]
sendbackup: start [mailserv:/usr1 level 1]
sendbackup: info BACKUP=/usr/sbin/ufsdump
sendbackup: info RECOVER_CMD=/bin/gzip -dc |/usr/sbin/ufsrestore -f... -
sendbackup: info COMPRESS_SUFFIX=.gz
sendbackup: info end
| DUMP: Date of this level 1 dump: Wed Jul 15 21:48:01 2009
| DUMP: Date of last level 0 dump: Tue Apr 28 18:48:36 2009
| DUMP: Dumping /dev/rdsk/c3t0d0s6 (mailserv:/usr1) to standard output.
| DUMP: Mapping (Pass I) [regular files]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Mapping (Pass II) [directories]
| DUMP: Writing 32 Kilobyte records
| DUMP: Estimated 58437252 blocks (28533.81MB) on 0.42 tapes.
| DUMP: Dumping (Pass III) [directories]
| DUMP: Dumping (Pass IV) [regular files]
| DUMP: 13.16% done, finished in 1:06
| DUMP: 24.46% done, finished in 1:01
| DUMP: 35.23% done, finished in 0:55
| DUMP: 44.94% done, finished in 0:49
| DUMP: 55.60% done, finished in 0:39
| DUMP: 66.37% done, finished in 0:30
| DUMP: 78.36% done, finished in 0:19
| DUMP: 88.62% done, finished in 0:10
| DUMP: Warning - block 2453935036 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 2453882400 is beyond the end of `/dev/rdsk/c3t0d0s6'
| DUMP: Warning - block 2454043076 is beyond the end of `/dev/rdsk/c3t0d0s6'
<--- many lines removed --->
| DUMP: Warning - block 3806913748 is beyond the end of `/dev/rdsk/c3t0d0s6'
? DUMP: More than 32 block read errors from dump device `/dev/rdsk/c3t0d0s6'
| DUMP: NEEDS ATTENTION: Do you want to attempt to continue? ("yes" or "no")
DUMP: The ENTIRE dump is aborted.
??error [/usr/sbin/ufsdump returned 3]
? dumper: strange [missing size line from sendbackup]
\--------
STRANGE DUMP DETAILS:
/-- squidzone1 /sqcache1/var lev 1 STRANGE
sendbackup: start [squidzone1:/sqcache1/var level 1]
sendbackup: info BACKUP=/usr/local/bin/gtar
sendbackup: info RECOVER_CMD=/usr/local/bin/gtar -xpGf - ...
sendbackup: info end
? gtar: ./logs/access.log: file changed as we read it
| Total bytes written: 523622400 (500MiB, 9.7MiB/s)
sendbackup: size 511350
sendbackup: end
\--------
/-- squidzone2 /sqcache2/var lev 1 STRANGE
sendbackup: start [squidzone2:/sqcache2/var level 1]
sendbackup: info BACKUP=/usr/local/bin/gtar
sendbackup: info RECOVER_CMD=/usr/local/bin/gtar -xpGf - ...
sendbackup: info end
? gtar: ./logs/access.log: file changed as we read it
| Total bytes written: 742635520 (709MiB, 7.4MiB/s)
sendbackup: size 725230
sendbackup: end
\--------
/-- squidzone2 /sqcache2/squidguard lev 0 STRANGE
sendbackup: start [squidzone2:/sqcache2/squidguard level 0]
sendbackup: info BACKUP=/usr/local/bin/gtar
sendbackup: info RECOVER_CMD=/usr/local/bin/gtar -xpGf - ...
sendbackup: info end
? gtar: ./log/squidGuard.log: file changed as we read it
| Total bytes written: 3191879680 (3.0GiB, 9.0MiB/s)
sendbackup: size 3117070
sendbackup: end
\--------
NOTES:
planner: disk mailserv:/usr1, estimate of level 0 failed.
planner: Full dump of squidzone2:/sqcache2/squidguard promoted from 6 days
ahead.
planner: Full dump of squidzone1:/sqcache1/squidguard promoted from 6 days
ahead.
planner: Full dump of trel:/Users promoted from 1 day ahead.
planner: Full dump of muninn:/var promoted from 2 days ahead.
planner: Full dump of h220:/ promoted from 2 days ahead.
planner: Full dump of squidtwo:/ promoted from 2 days ahead.
planner: Full dump of pavlov:/ promoted from 2 days ahead.
planner: Full dump of gatem:/usr1 promoted from 2 days ahead.
planner: Full dump of c110:/ promoted from 2 days ahead.
planner: Full dump of nlascar:/ promoted from 2 days ahead.
taper: tape Curie10 kb 338138026 fm 46 [OK]
big estimate: nlascar / 0
est: 6408M out 4101M
big estimate: nlascar /var 0
est: 327M out 164M
small estimate: finsen / 1
est: 57M out 199M
DUMP SUMMARY:
DUMPER STATS TAPER STATS
HOSTNAME DISK L ORIG-MB OUT-MB COMP% MMM:SS KB/s MMM:SS KB/s
-------------------------- ------------------------------------- -------------
c110 / 0 1165 619 53.2 10:30 1006.2 0:50 12646.7
c110 /opt 1 0 0 1.6 0:41 0.1 0:00 797.6
curie / 0 FAILED --------------------------------------------
curie /export 1 96060 74970 78.0 123:31 10358.8 102:53 12436.7
curie /thump/flar 1 0 0 -- 0:01 0.8 0:00 102.5
curie -ump/source 1 36474 29879 81.9 57:15 8905.9 39:57 12761.9
curie -p/vmfs-bak 1 0 0 -- 0:01 0.7 0:00 72.2
dnix /dev/sda1 1 1841 1841 -- 18:54 1662.2 3:26 9150.0
everest /images3 1 0 0 -- 0:01 1.2 0:00 183.6
finsen / 1 547 199 36.3 2:53 1178.2 0:55 3663.5
finsen /export 1 33204 18265 55.0 105:25 2957.2 24:10 12901.2
gatem / 1 629 629 -- 7:37 1409.6 1:49 5885.2
gatem /usr1 0 63830 63830 -- 298:25 3650.6 84:18 12923.4
h220 / 0 4678 3130 66.9 49:56 1069.9 4:14 12593.6
h220 /opt 1 591 43 7.3 10:50 68.1 0:03 12696.7
huginn / 1 30479 30479 -- 73:26 7083.5 40:58 12695.2
ldap1 / 0 7784 2524 32.4 18:36 2315.9 3:25 12581.8
ldap1 -xport/home 1 0 0 1.0 0:05 1.0 0:00 1740.6
ldap1 /usr1 1 627 51 8.2 1:05 805.2 0:04 12750.4
lyra / 1 FAILED --------------------------------------------
lyra /3rdparty 0 40359 35645 88.3 215:49 2818.9 47:08 12904.5
lyra /db1 1 14077 2303 16.4 31:52 1233.7 3:06 12652.9
lyra /db2 1 18560 2078 11.2 37:03 957.3 2:46 12794.0
lyra /db3 0 FAILED --------------------------------------------
lyra /db4 0 25112 3516 14.0 59:40 1005.6 4:42 12764.7
lyra -port/home0 1 2 0 8.3 0:23 5.9 0:00 10717.5
lyra /ndevelop 0 9474 7037 74.3 58:25 2055.7 9:31 12614.7
lyra /space 1 104 28 26.4 6:37 71.1 0:03 8438.3
mailserv / 1 2618 499 19.1 7:01 1215.7 0:42 12145.9
mailserv /usr1 1 15100 -- PARTIAL 22:10 11629.3
PARTIAL
muninn / 1 554 165 29.8 17:55 157.5 0:14 12531.6
muninn /var 0 3822 617 16.2 15:08 696.1 1:04 9897.2
ngato / 1 0 0 5.8 0:49 0.5 0:00 8948.1
nlascar / 0 3102 4101 132.2 13:30 5186.2 5:45 12172.8
nlascar /boot 1 0 0 -- 0:03 0.8 0:00 454.4
nlascar /data 1 0 0 0.9 0:36 0.1 0:00 1526.5
nlascar /var 0 292 164 56.1 1:39 1688.8 0:32 5233.7
panther / 1 1 0 7.8 2:33 0.7 0:00 10244.1
panther /data 1 0 0 -- 0:01 0.7 0:00 188.1
pavlov / 0 15079 9726 64.5 24:31 6770.9 40:30 4098.5
squidone / 1 159 19 11.9 5:24 59.9 0:02 12613.5
squidtwo / 0 25568 12758 49.9 36:15 6006.7 25:07 8667.4
squidzone1 -squidguard 0 2549 2549 -- 4:44 9185.5 3:27 12598.9
squidzone1 -cache1/var 1 499 499 -- 0:52 9875.7 0:41 12603.1
squidzone2 -squidguard 0 3044 3044 -- 5:41 9149.8 5:01 10369.9
squidzone2 -cache2/var 1 708 708 -- 1:38 7420.4 1:18 9337.0
trel /Users 0 3258 2201 67.5 7:57 4724.8 3:54 9612.1
trel /trelAQ 1 995 995 -- 4:26 3830.6 4:08 4113.7
trel /trelRZ 1 2 2 -- 0:08 219.3 0:00 12825.2
(brought to you by Amanda version 2.6.1p1)
----- End forwarded message -----
---
Brian R Cuttler brian.cuttler AT wadsworth DOT org
Computer Systems Support (v) 518 486-1697
Wadsworth Center (f) 518 473-6384
NYS Department of Health Help Desk 518 473-0773
IMPORTANT NOTICE: This e-mail and any attachments may contain
confidential or sensitive information which is, or may be, legally
privileged or otherwise protected by law from further disclosure. It
is intended only for the addressee. If you received this in error or
from someone who was not authorized to send it to you, please do not
distribute, copy or use it or any attachments. Please notify the
sender immediately by reply e-mail and delete this from your
system. Thank you for your cooperation.
|
|
|
