ADSM-L

Re: Slowness of the Migration Process

1997-04-09 09:24:54
Subject: Re: Slowness of the Migration Process
From: "Dwight E. Cook" <decook AT AMOCO DOT COM>
Date: Wed, 9 Apr 1997 08:24:54 -0500
Item Subject: 1.txt "internet headers"
Could not convert BINARY FILE item to text.
Will attempt to 'shar' item as file '028g5s9' at end of msg.

.......................................................................

Item Subject: Slowness of the Migration Process
     You can increase the number of migration processes you are running.
     1) the device class for your tape drives has a mount limit greater
     than or equal to the number of migration processes you want to run...
     2) update the disk storage pool for "migpr=#"
     3) odd note: make sure that your tape storage pool has its "maxscr"
     set to the max possible for your library... In a shared library I had
     a server's maxscr=100, then I was running short on scratches and
     inserted 50 more... initially I was clueless when I had a scratch
     request denied when I had 50 scratch tapes in the library (but 100
     private that came from scratch, ARGH!)

     And I would bet your files are around 5-25KB each, right?
     I don't have my final numbers yet but 2 of my servers were doing 4.5TB
     monthly, one was on easy street, the other took us 3 months to finish
     migrating diskpool to tapepool.  The difference was the QUANTITY of
     files being delt with (and the associated extra overhead of DB
     updates) Talk about chain reaction: Migrations didn't complete in
     their window (have to let them continue) which slowed down the DB
     backup and the expiration, which then went outside its window into
     reclamation's window, which resulted in everything trying to run when
     the next nights backups kicked in to only rub salt in the wound...

     Things are much better now.

     I would either look into an additional server (not just a bigger one)
     ***OR*** look into what is being backed up off those desktop machines.
      You wouldn't believe the number of "netscape cache files" I found
     being backed up along with all sorts of other trash... Ya'know if it
     weren't for these dang users, I'd have a pretty good computing
     environment ;-)  Educate the users either by billing them heavily or
     having a class in "the art of include/exclude lists".


     *****one last trick ! *************
     When a migration task initiates it SEEMS to generate a list of items
     it is going to deal with and sticks to that list until done.  Now this
     is kind of a catch-22 because... small files are the problem 'cause of
     the db overhead time but if you kill a migration process that has been
     running for a long time (ie started before a wave of backups put more
     into the storage pool) and start a new one in its place, the new
     process will gather a new list of files to migrate STARTING WITH THE
     LARGEST FILES ('cause that gains you the most) but the catch is the
     net result is you keep building more and more smaller files that take
     more time (per GB space) to migrate.  Just like the 5 gallon bucket
     with a hole that drains 1 quart/minute and you're filling it at 1.5
     quart/minute, only a matter of time until its full....

     Don't feel bad, my problem was 250 GB big, that's why it took 3 months
     and relocating some clients to other servers to fix...

     To double check things, pull your accounting records for an average
     week on all servers, you will probably find that the troubled server
     is dealing with at least 10 times the number of items as the other
     servers...


     Good luck,
               later
                    Dwight


_____________________________ Forward Header __________________________________
Subject: Slowness of the Migration Process
Author:  INTERNET.OWNERAD at SNADGATE
Date:    4/9/97 7:37 AM


I suspect I'm not the only one having a rather trying week due to the
32-bit Windows clients doing full backups in honor of Daylight Savings
(I have about 1200 of them).

Of my three servers (all on VM, all V1), two are holding together, but
the largest (1800 clients) is not.  I've expanded the disk storage pool
on all of them and cranked the maximum number of sessions down, but the
disk storage pool (6G) on the largest keeps filling up.

The problem is that the migration process can't keep up, running around
the clock.  Am I right in believing that only one migration process is
allowed at a time?  Is there any way to up the priority of the migration
task?  (It appears to be relatively low.)  I've given the server virtual
machine massive favoring, and the tapes are in an STK silo, so it never
waits long for tape mounts.  Any suggestions for speeding up the process?

Melinda Varian,
Princeton University


# This is a shell archive.  Remove anything before this line,
# then unpack it by saving it in a file and typing "sh file".
#
# Wrapped by Openmail for HP9000 <openmail@tuleosm1> on Wed Apr  9 08:25:49 1997
#
# This archive contains:
#       028g5s9
#
# Error checking via wc(1) will be performed.

LANG=""; export LANG
PATH=/bin:/usr/bin:$PATH; export PATH


rm -f /tmp/uud$$
(echo "begin 666 /tmp/uud$$\n#;VL*n#6%@x\n \nend" | uudecode) >/dev/null 2>&1
if [ X"`cat /tmp/uud$$ 2>&1`" = Xok ]
then
        unpacker=uudecode
else
        echo Compiling unpacker for non-ascii files
        pwd=`pwd`; cd /tmp
        cat >unpack$$.c <<'EOF'
#include <stdio.h>
#define C (*p++ - ' ' & 077)
main()
{
        int n;
        char buf[128], *p, a,b;

        scanf("begin %o ", &n);
        gets(buf);

        if (freopen(buf, "w", stdout) == NULL) {
                perror(buf);
                exit(1);
        }

        while (gets(p=buf) && (n=C)) {
                while (n>0) {
                        a = C;
                        if (n-- > 0) putchar(a << 2 | (b=C) >> 4);
                        if (n-- > 0) putchar(b << 4 | (a=C) >> 2);
                        if (n-- > 0) putchar(a << 6 | C);
                }
        }
        exit(0);
}
EOF
        cc -o unpack$$ unpack$$.c
        rm unpack$$.c
        cd $pwd
        unpacker=/tmp/unpack$$
fi
rm -f /tmp/uud$$

echo x - 028g5s9 '[non-ascii]'
$unpacker <<'@eof'
begin 660 028g5s9
M4F5C96EV960Z(&9R;VT@8V]R<&UX,#(@*&-O<G!M># R+FAO=2YA;6]C;RYCX
M;VTI(&)Y('1U;&5O<VTQ('=I=&@@15--5% -"@DH,2XS-RXQ,#DN,C O,38NX
M,BD@:60@04$Q,S<R.34X-34[(%=E9"[email protected]!!<'(@,3DY-R P-CHS-SHS-2 MX
M,#4P, T*4F5C96EV960Z(&9R;VT@:&]S;3$R,F$@8GD@8V]R<&UX,#(@*%--X
M22TX+C8O4TU)+30N,"D-"@EI9"!'04$Q,3<S,SL@5V5D+" Y($%P<B Q.3DWX
M(# V.C(X.C(X("TP-3 P#0I296-E:79E9#H@9G)O;2!I;G1E<FQO8VLN86UOX
M8V\N8V]M(&)Y(&AO<VTQ,C)A("A334DM."XV+U--22U35E(T*0T*"6ED($=!X
M03 R-3$U.R!7960L(#D@07!R(#$Y.3<@,#8Z-#$Z,S0@+3 U,# -"E)E8V5IX
M=F5D.B!F<F]M('9M+FUA<FES="YE9'4@8GD@<&]R=&%L+F%M;V-O+F-O;2!WX
M:71H(%--5% @:60@04$Q,[email protected]*(" H26YT97),;V-K(%--5% @1V%T97=AX
M>2 S+C @9F]R(#QD96-O;VM 04U/0T\N0T]-/BD[#0H@(%=E9"[email protected]!!<'(@X
M,3DY-R P-CHS-SHQ,B M,#4P, T*4F5C96EV960Z(&9R;VT@5DTN34%225-4X
M+D5$52!B>2!632Y-05))4U0N1415("A)0DT@5DT@4TU44"!6,E(S*0T*(" @X
M=VET:"!"4TU44"!I9" T-3$Y.R!7960L(# Y($%P<B Y-R P-SHS-CHS-2!%X
M1%0-"E)E8V5I=F5D.B!F<F]M(%9-+DU!4DE35"Y%1%4@*$Y*12!O<FEG:6X@X
M3$E35%-%4E9 34%225-4*2!B>2!632Y-05))4U0N1415("A,36%I;"!6,2XRX
M8B\Q+CAB*2!W:71H($)33510(&ED(#<X,3<[(%=E9"[email protected]!!<'(@,3DY-R PX
M-SHS-CHS,2 M,#0P, T*4F5C96EV960Z(&9R;VT@5DTN34%225-4+D5$52!BX
M>2!632Y-05))4U0N1415("A,25-44T525B!R96QE87-E(#$N.&,I('=I=&@@X
M3DI%#0H@(" @(" @(" @:60@,34S-"!F;W(@041332U,0%9-+DU!4DE35"Y%X
M1%4[(%=E9"[email protected]!!<'(@,3DY-R P-SHS-CHR.2 M,#0P, T*4F5C96EV960ZX
M(&9R;VT@34%225-4("A.2D4@;W)I9VEN(%--5%! 34%225-4*2!B>2!632Y-X
M05))4U0N1415("A,36%I; T*(" @(" @(" @(%8Q+C)B+S$N.&(I('=I=&@@X
M0E--5% @:60@-S@P,SL@5V5D+" Y($%P<B Q.3DW(# W.C,V.C(Y("TP-# PX
M#0I296-E:79E9#H@9G)O;2!I;G1E<FQO8VLN:71T:&%R=&9O<F0N8V]M(&)YX
M(%9-+DU!4DE35"Y%1%4@*$E"32!632!33510(%8R4C,I#0H@(" @(" @(" @X
M=VET:"!40U [(%=E9"P@,#D@07!R(#DW(# W.C,V.C$W($5$5 T*4F5C96EVX
M960Z(&)Y(&EN=&5R;&]C:RYI='1H87)T9F]R9"YC;VT@:60@04$P,3DQ-2 HX
M26YT97),;V-K(%--5% @1V%T97=A>2 S+C -"B @(" @(" @("!F;W(@861SX
M;2UL0%9-+DU!4DE35"Y%1%4I.R!7960L(#D@07!R(#$Y.3<@,#<Z,S8Z,3$@X
M+3 T,# -"E@T,# M3W)I9VEN871O<CH@:FQA=W-O;D!H:6=M>"YI='1H87)TX
M9F]R9"YC;VT-"E@T,# M4F5C:7!I96YT<SH@861S;2UL0%9-+DU!4DE35"Y%X
M1%4-"E@T,# M371S+4ED96YT:69I97(Z(%LO041-1#U)3E1%4DY%5"]#/553X
M+SLP,#$R.3 P,# Q-#,V,S,Q,# P,# R70T*6#0P,"U#;VYT96YT+51Y<&4ZX
M(% R+3$Y.#@@*#(R*0T*4')I;W)I='DZ(%5R9V5N= T*365S<V%G92U)9#H@X
M(#PP,#$R.3 P,# Q-#,V,S,Q,# P,# R*D!-2%,^#0I$871E.B @(" @(" @X
M(%=E9"[email protected]!!<'(@,3DY-R P-SHU,3HT," M,#0P, T*4F5P;'DM5&\Z(")!X
M1%--.B!$:7-T(%-T;W(@36%N86=E<B(@/$%$4TTM3$!632Y-05))4U0N1415X
M/@T*4V5N9&5R.B B041333H@1&ES="!3=&]R($UA;F%G97(B(#Q!1%--+4Q X
M5DTN34%225-4+D5$53X-"D9R;VTZ($IE<G)Y($QA=W-O;B \:FQA=W-O;D!4X
M2$5(05)41D]21"Y#3TT^#0I3=6)J96-T.B @(" @(%-L;W=N97-S(&]F('1HX
M92!-:6=R871I;VX@4')O8V5S<PT*5&\Z($%$4TTM3$!632Y-05))4U0N1415X
"#0I-                                                        X
                                                             X
end
@eof
set `wc -lwc <028g5s9`
if test $1$2$3 != 322231892
then
        echo ERROR: wc results of 028g5s9 are $* should be 32 223 1892
fi

chmod 660 028g5s9

rm -f /tmp/unpack$$
exit 0
<Prev in Thread] Current Thread [Next in Thread>