Compression Utility Summary

This summary references the bulletin posted March 7
discussing the compression buffering problem.

We tested the compression utility internally and we also
got some customer input.  We'd like to pass on these
preliminary findings and hope that this summary will be
helpful to those who will be running the compression
utility.

Bill Chang - ADSM Technical Support

==============================================================

Audit Compression Utility Summary

An AUDIT COMPRESSION utility has been introduced in the service
stream for the ADSM servers.  PTFs UN73480 (VM SES) and UN73483,
UN73484 (MVS) bring these servers to level 13.  Similarly, PTFs
IP20254 (AIX) and IP20255 (OS/2) bring these servers to level 7.

This utility is intended to be run only once to address the
files that may have been affected by the compression buffering
problem.  There are 2 levels of analysis, Level 1 and Level 2.
Each level has the "checkpoint/restart" capability; each runs
as a background process and if the process is canceled using a
"CANCEL PROCESS nn" command, then it will restart from where it
left off.  If the process is interrupted in any other fashion,
then the analysis process will start from the beginning again.

General steps :

1)  AUDIT COMPRESSION LEVEL=1
2)  QUERY COMPRESSION           (to preview candidate list)
3)  AUDIT COMPRESSION LEVEL=2
3a) QUERY COMPRESSION           (to preview candidate list)
4)  FORCE BACKUP                (backup selected candidates)
5)  REMOVE COMPRESSION          (remove candidates from list)
6)  QUERY COMPRESSION           (to preview candidate list)
6a) DELETE COMPRESSION          (delete candidate files)
7)  CLEANUP COMPRESSION         (remove candidate entries in DB)

The README file gives detailed information on the syntax of each
of the commands with its options.  We STRONGLY recommend that
you read this file before proceeding with the AUDIT COMPRESSION
process.

Level 1 analysis :

Level 1 will read the database and generate a list of candidates
based on storage size.  Level 1 DOES NOT KNOW whether the files
were compressed or not.  This method covers a broader range of
selection.  As a result, the list of candidates can be long.
If your database contains entries for millions of files & their
copies, your generated list could contain several thousand
candidates.  Each version or copy is treated as a separate file,
thus contributing to this high candidate count.

In this level of analysis, files with sizes that fall within 16
bytes of a 32K boundary are identified.

Level 2 analysis :

Level 2 will process the list of candidates and will actually
read the control information stored in each file.  Size and
compression information are further analyzed to see if a file
is still affected by the compression buffering problem.  If
it isn't, it is eliminated from the list.  After this process
completes, the candidate list is reduced considerably.

Files that reside on disk will have undergone a Level 2 analysis
during the Level 1 run.  Tapes will have to be mounted for the
files residing on sequential media.

A feature will be added to identify the volumes by node(s) so
that the tape volumes can be pre-pooled to facilitate tape
mounts for the Level 2 analysis.  This command will be made
available in Level 14 of the MVS & VM servers.  The output can
be redirected to a file when issued from an admin client.

   SHOW VOLUMEUSEAGE <nodename>

Discussion:

1) Level 1 can run a long time if your database is large.
   However this analysis runs as a background process.  You can
   query the process for its status.  The VERBOSE option will
   produce detailed information for EACH file processed.

   Some results are captured below from brief discussions with
   internal sites and customer.   This will illustrate how
   varied the analysis can be depending on the environment.

   a) Database - 500 Meg      Clients - 30
      Level 1 analysis : 276 candidates / 1.1 million files
      Lvl 1 elapsed time : 2 hours
      Lvl 2 elapsed time : 3.5 hours      40+ tapes mounted
      Final candidate ct.: 83 / 1.1 million   ---> 0.0007 % hit

   b) Database - 7.0 Gig      Clients - 700
      Server was recycled by 2 system IPLs, thus each time the
      level 1 analysis had to start from the beginning.

      Level 1 analysis : 4433 candidates / 4.97 million files
      Lvl 1 elapsed time : 3 days and still going

   c) One customer reported an elapsed time of 42 hours for the
      Level 1 analysis during which some 9 million files were
      examined.

The length of time and hit ratio depend on the types of files,
the number of files (and its copies) and the size of your data-
base.

2) We ENCOURAGE you to upgrade your clients to PTF level 7. The
   server will FORCE compression off if it detects a back-level
   client, thus using more disk space.  The indicative message
   that you will see is :

   "ANR0446W ...  client .. compression method unsupported -
     compression forced OFF"

   To continue using your back-level client code, add this to
   your server option file - RISK.COMPRESSION.CORRUPTION.  This
   will allow you to use compression.  BUT, you will be risking
   a potential exposure to the compression buffering problem.

3) While both Level 1 and Level 2 analysis can run a long time,
   they can be stopped and restarted.  In addition, the AUDIT,
   QUERY, REMOVE & DELETE COMPRESSION and FORCE BACKUP commands
   provide finer granularity.  Thus you can plan for which node,
   filespace and/or filetype you want to AUDIT or REMOVE.  For
   example :

   AUD COMP bill_os2 Level=2 FILESpace=c_drive FILEType=Backup

The README file provides detailed description and examples for
running this utility.  Again, we RECOMMEND that you familiarize
yourselves with the write up of the utility before starting the
process.