Re: [Networker] Limit on number of savesets in an nsrclone?
2008-03-10 13:34:06
On 10 Mar 08, at 1350, Ian G Batten wrote:
A look at /var/nsr/logs/daemon.log would have been a good idea. I
can now see what's happening, the question is why it's happening...
I tracked it down. Somehow (and there was some disturbance in the
storage a week ago, so it's not impossible) one saveset out of 400 in
the volume was missing. This in turn caused the whole clone or stage
operation to be aborted with extreme prejudice, without doing any
clones. I found it by staging the savesets one at a time, until one
of them failed, then (after checking it was a saveset I could live
without, but as it was missing anyway that was rather moot) using
nsrmm -d to delete the one that failed. Then the whole thing would
stage successfully.
Interestingly, scanner -i ran successfully: I presume it doesn't purge
from the media database savesets which are in the database as being on
the volume but which scanner doesn't find. Another route to achieve
the same result would be to delete all savesets that are on the
volume, then scan the volume in.
ian
ian
03/10/08 13:38:20 nsrd: media info: nsrmmd #3 on backup-
srv.ftel.co.uk started a
s requested
03/10/08 13:38:41 nsrmmd #20: filesys_retrieve: failed to read:
cannot open /var
/remotestage/incrementals/offsite-4/85/38/8f2df281-00000006-
ddc77fdd-47c77fdd-01
2e0000-ac100301 file: No such file or directory
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrd: cloning session:1 of 25 save set(s) starting
read from I
ncrementalsStaging.003.RO of 915 MB
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
03/10/08 13:38:41 nsrmmd #20: Read operation failed and aborted.
On 10 Mar 08, at 1342, Ian G Batten wrote:
It's not the size of the list: I'm seeing failures at 25 (I've
modified my script to pre-split the input into batches of 25)
What's happening is that the nsrclone starts, the adv_file and the
tape device are mounted, and then the nsrclone fails. The fact that
(in this example) the volume mounts have been notified indicates
that things have progressed, and the session is actually appearing
in the GUI as in progress (briefly). You can see in this nsrwatch
output that the save sessions actually come up, and then
immediately `complete'. I'm worried by the flurry of nsrmmd
restarts, though.
ian
Mon 13:37:24 media info: restarting nsrmmd #4 on backup-
srv.ftel.co.uk in 2 minute(s)
Mon 13:37:25 media waiting event: Waiting for 1 writable volumes to
backup pool 'Incrementals' disk(s) or tape(s) on sonic.ftel.co.uk
Mon 13:37:25 /var/remotestage/incrementals/offsite-4/_AF_readonly
mounted adv_file disk IncrementalsStaging.003.RO (write protected)
Mon 13:37:50 media info: restarting nsrmmd #3 on backup-
srv.ftel.co.uk now
Mon 13:37:50 rd=sonic.ftel.co.uk:/dev/rmt/3cbn Verify label
operation in progress
Mon 13:37:54 rd=sonic.ftel.co.uk:/dev/rmt/3cbn verified label of
002455L3
Mon 13:37:54 rd=sonic.ftel.co.uk:/dev/rmt/3cbn Mount operation in
progress
Mon 13:38:12 rd=sonic.ftel.co.uk:/dev/rmt/3cbn mounted LTO
Ultrium-2 tape 002455L3
Mon 13:38:12 media event cleared: Waiting for 1 writable volumes to
backup pool 'Incrementals' disk(s) or tape(s) on sonic.ftel.co.uk
Mon 13:38:12 cloning session:1 of 25 save set(s) starting read from
IncrementalsStaging.003.RO of 915 MB
Mon 13:38:12 write completion notice: Writing to volume 002456L3
complete
Mon 13:38:13 sonic.ftel.co.uk:cloning session done saving
Mon 13:38:13 cloning session:25 of 25 save set(s) done reading from
IncrementalsStaging.003.RO
Mon 13:38:13 /var/remotestage/incrementals/offsite-4/_AF_readonly
enabled; nsrmmd not available
Mon 13:38:13 media info: restarting nsrmmd #11 on backup-
srv.ftel.co.uk in 2 minute(s)
Mon 13:38:14 /var/remotestage/incrementals/offsite-4/_AF_readonly
mounted adv_file disk IncrementalsStaging.003.RO (write protected)
Mon 13:38:20 media info: nsrmmd #3 on backup-srv.ftel.co.uk started
as requested
Mon 13:38:41 cloning session:1 of 25 save set(s) starting read from
IncrementalsStaging.003.RO of 915 MB
Mon 13:38:42 sonic.ftel.co.uk:cloning session done saving
Mon 13:38:42 cloning session:25 of 25 save set(s) done reading from
IncrementalsStaging.003.RO
Mon 13:38:42 /var/remotestage/incrementals/offsite-4/_AF_readonly
enabled; nsrmmd not available
Mon 13:38:42 media info: restarting nsrmmd #20 on backup-
srv.ftel.co.uk in 2 minute(s)
Mon 13:39:26 media info: restarting nsrmmd #4 on backup-
srv.ftel.co.uk now
Mon 13:39:26 media info: nsrmmd #4 on backup-srv.ftel.co.uk started
as requested
Mon 13:40:18 media info: restarting nsrmmd #11 on backup-
srv.ftel.co.uk now
Mon 13:40:18 media info: nsrmmd #11 on backup-srv.ftel.co.uk
started as requested
Mon 13:40:47 write completion notice: Writing to volume 002357L3
complete
Mon 13:40:51 media info: restarting nsrmmd #20 on backup-
srv.ftel.co.uk now
Mon 13:40:51 media info: nsrmmd #20 on backup-srv.ftel.co.uk
started as requested
On 09 Mar 08, at 0733, Vincent Lin wrote:
On one of my 7.2.2 build 494 Networker servers,
I've got 1,700+ savesets devided into 2 tape pools across 4 x 1 TB
AFTDs.
I break them down to some small batches, 100 savesets per run per
LTO2 tape drive over 4 drives.
It is much more manageable this way.
----- Original Message ----
From: Peter Viertel <Peter.Viertel AT MACQUARIE DOT COM>
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Sent: Sunday, March 9, 2008 12:26:09 PM
Subject: Re: [Networker] Limit on number of savesets in an nsrclone?
I clone absolutely everything by ssid under some of my 7.2.2.494
systems. I've bigger lists than that in by file and by stdin
without any
issues. Just looking at recent tempfiles I have lying around - one
has
528 ssid's in it and adds up to 11435 bytes, and that went
through just
fine. Not to say there's not a bug in 7.3.3 though. I will be
doing
a big batch on a 7.3.3 system this afternoon and will keep Ian's
reported problem in mind for that.
-----Original Message-----
From: EMC NetWorker discussion
[mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
Behalf Of Werth, Dave
Sent: Saturday, 8 March 2008 4:37 AM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: Re: [Networker] Limit on number of savesets in an nsrclone?
I don't know if this applies but often there is a limit on the
length of
a command line. In Solaris I believe it is 2048 bytes. So even
though
you're putting the savesets to be cloned in a file it may be
parsing it
into a command line as if you had tried to write it all without the
savesets file. So it's not so much the number of savesets you can
clone
but the total length of the command line that results. For
instance if
your savesets were name a, b, c, etc. you might be able to clone
well
over 1000 savesets.
Dave
Dave Werth
Garmin AT, Inc.
Salem, Oregon
-----Original Message-----
From: EMC NetWorker discussion
[mailto:NETWORKER AT LISTSERV.TEMPLE DOT EDU] On
Behalf Of Ian G Batten
Sent: Friday, March 07, 2008 8:35 AM
To: NETWORKER AT LISTSERV.TEMPLE DOT EDU
Subject: [Networker] Limit on number of savesets in an nsrclone?
I'm in the process of binary chopping to find the limit, but it
appears that there is a limit to the number of savesets I can
reference in an nsrclone -S -f /file/name (Networker 7.3.3 on
Solaris
10). 175 is OK, while 250 is too many.
I have a script which implements a policy of writing a daily tape of
activity in a given pool, but not deleting the files until a week
has
passed. Recently we wrote a lot of savesets during one day, taking
the amount of activity to go to tape at night over the magic
threshold, from which there is no return. nsrwatch shows the clone
job start (cloning session: XXX save sets reading from...) but then
immediately reports completion, while the client sees the messages
below.
/etc/tools/clonetotape.test: saving 313 savesets from
IncrementalsStaging.003.RO to Incrementals...
+ nsrclone -b Incrementals -s backup-srv.ftel.co.uk -S -f /tmp/
clonetotape.test.25004
nsrclone: RPC error: RPC receive operation failed. A network
connection could not be established with the host.
nsrclone: Cannot open nsrclone session with backup-srv.ftel.co.uk
nsrclone: Cannot open nsrclone session with backup-srv.ftel.co.uk.
Error is 'RPC receive operation failed. A network connection could
not be established with the host.'
nsrclone: Failed to clone any save sets
To sign off this list, send email to listserv AT listserv.temple DOT edu
and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems
with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
-------------------------
This e-mail and any attachments may contain confidential material
for
the sole use of the intended recipient. If you are not the intended
recipient, please be aware that any disclosure, copying,
distribution or
use of this e-mail or any attachment is prohibited. If you have
received
this e-mail in error, please contact the sender and delete all
copies.
Thank you for your cooperation
To sign off this list, send email to listserv AT listserv.temple DOT edu
and
type "signoff networker" in the body of the email. Please write to
networker-request AT listserv.temple DOT edu if you have any problems
with this
list. You can access the archives at
http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
NOTICE
This e-mail and any attachments are confidential and may contain
copyright material of Macquarie Group Limited or third parties. If
you are not the intended recipient of this email you should not
read, print, re-transmit, store or act in reliance on this e-mail
or any attachments, and should destroy all copies of them.
Macquarie Group Limited does not guarantee the integrity of any
emails or any attached files. The views or opinions expressed are
the author's own and may not reflect the views or opinions of
Macquarie Group Limited.
To sign off this list, send email to listserv AT listserv.temple DOT edu
and type "signoff networker" in the body of the email. Please
write to networker-request AT listserv.temple DOT edu if you have any
problems with this list. You can access the archives at http://listserv.temple.edu/archives/networker.html
or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now.
http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ
To sign off this list, send email to listserv AT listserv.temple DOT edu
and type "signoff networker" in the body of the email. Please
write to networker-request AT listserv.temple DOT edu if you have any
problems with this list. You can access the archives at http://listserv.temple.edu/archives/networker.html
or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
To sign off this list, send email to listserv AT listserv.temple DOT edu
and type "signoff networker" in the body of the email. Please write
to networker-request AT listserv.temple DOT edu if you have any problems
with this list. You can access the archives at http://listserv.temple.edu/archives/networker.html
or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
To sign off this list, send email to listserv AT listserv.temple DOT edu
and type "signoff networker" in the body of the email. Please write
to networker-request AT listserv.temple DOT edu if you have any problems
with this list. You can access the archives at http://listserv.temple.edu/archives/networker.html
or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
To sign off this list, send email to listserv AT listserv.temple DOT edu and type
"signoff networker" in the body of the email. Please write to networker-request
AT listserv.temple DOT edu if you have any problems with this list. You can access the
archives at http://listserv.temple.edu/archives/networker.html or
via RSS at http://listserv.temple.edu/cgi-bin/wa?RSS&L=NETWORKER
|
|
|