Hi Allen,

I've been down this road too - quite a way, in fact.  I had a pretty
good filtering system that did all sorts of filtering and logging.  I
was trying to find the needles in the haystacks, or separate the
wheat from the chaff.  Anyway, the point is that I'm now using
Servergraph and can tell you that it has some very fine capabilities
in this area.  Might be worth a look.

You can't really use the "S, E, W, I" flag to indicate what to do
with the message.  It's not even close to being that simple.

My code did stuff such as multiple levels of filtering.  The really
good stuff would be e-mailed to me every day.  The not quite so
useful stuff was logged on disk, so I could refer to it for problem
research.  The totally useless stuff was just dropped and not logged
anywhere.  I also had a separate logfile for tape-related messages,
so that I could easily research the history of a particular tape
drive or tape volume by simply grepping through this file.  Very
useful.  The filter would also summarize important messages that
tended to repeat, by reporting the message, the number of
occurrences, and the time of the first and last message.  Some
messages would vary just a bit, so I added some code to ignore the
variable part if it wasn't important.  Thinks like "moving colocation
cluster N of M" - interesting, but I don't need to see all of them,
just a summary.

Servergraph's capabilities are different in many ways, but there are
many capabilities which I wouldn't even have attempted in my
code.  You can capture local knowledge and retain it for future
reference.  I.e., this message usually means this - contact so and so
or do such and such.  You can send pages, emails, to admins or users.

Take a look - I think you'll like it.

Regards,
..Paul

At 12:51 PM 4/26/2006, Allen S. Rout wrote:

Greetings, all.

I'm working on code to help me automatically cull the activity logs
for my TSM servers, summarizing the normal messages and leaving the
unusual or important messages for consideration.

Woo, BUNCH of value-laden words there.  "Normal", "unusual",
"important".  Yeesh.


So I wanted to toss my selections of messages-to-ignore before this
august body, and encourage kibitzing, before I dub the code
production.

I've tried to do several things here: Not just identify messages, but
also give a short-hand summary of what the message is related to, so I
can say something a little more informative than 'ignored 200K
messages'.

I've also tried to identify messages I can tie to a specific host, and
forward them to the relevant contact.

So I've got two gross classes of message: Messages tied to a host,
matched with the regexp which will pull the host out of the message
string, and messages to ignore, matched with an explanation.


I'm especially interested in several different classes of criticism:


"If you organized your messages in [foo] manner, it'd be easier for me
 to understand",

"Why do you want to ignore [foo], it's a symptom of problem Q",

"Oh, we also ignore [foo]",


I am specifically thinking that this might be an irritating grain of
sand around which a pearl of community wisdom might accrete; so please
have at me if you can think of ways I could improve the display of
these lists.  Regexp kibitzing also welcome.

- Allen S. Rout




my $collect =
  {
   '        ' => "Node: (\\S+)\\\)\\s",
   'ANE4007E' => "Node: (\\S+)\\\)\\s",
   'ANE4008E' => "Node: (\\S+)\\\)\\s",
   'ANE4987E' => "Node: (\\S+)\\\)\\s",
   'ANE4993E' => "Node: (\\S+)\\\)\\s",
   'ANE4994S' => "Node: (\\S+)\\\)\\s",
   'ANR0424W' => "node (\\S+)\\s",
   'ANR0425W' => "node (\\S+)\\s",
   'ANR0479W' => "server (\\S+)\\s",
   'ANR0480W' => "node (\\S+)\\s",
   'ANR0481W' => "node (\\S+)\\s",
   'ANR0500W' => "node (\\S+)\\s",
   'ANR2579E' => "node (\\S+)\\s",
   'ANR2716E' => "client (\\S+)\\s",
  };



my $ignore =
{

 'ANE4005E' => "Client file not found",
 'ANE4037E' => "Changing file was skipped",
 'ANE4952I' => "Client session postmortem",
 'ANE4953I' => "Client session postmortem",
 'ANE4954I' => "Client session postmortem",
 'ANE4955I' => "Client session postmortem",
 'ANE4957I' => "Client session postmortem",
 'ANE4958I' => "Client session postmortem",
 'ANE4959I' => "Client session postmortem",
 'ANE4960I' => "Client session postmortem",
 'ANE4961I' => "Client session postmortem",
 'ANE4963I' => "Client session postmortem",
 'ANE4964I' => "Client session postmortem",
 'ANE4965I' => "Client session postmortem",
 'ANE4966I' => "Client session postmortem",
 'ANE4967I' => "Client session postmortem",
 'ANE4968I' => "Client session postmortem",
 'ANE4969I' => "Client session postmortem",
 'ANE4970I' => "Client session postmortem",
 'ANE4991I' => "TDP MSExchange informational msg",
 'ANR0402I' => "Server session start",
 'ANR0403I' => "Session end",
 'ANR0405I' => "Admin session end",
 'ANR0406I' => "Session start",
 'ANR0407I' => "Admin session start",
 'ANR0408I' => "Library client session start",
 'ANR0409I' => "Library client session end",
 'ANR0482W' => "Idle Timeout",
 'ANR0510I' => "Volume open",
 'ANR0511I' => "Volume open",
 'ANR0512I' => "Volume open",
 'ANR0513I' => "Remote volume open",
 'ANR0514I' => "Volume close",
 'ANR0515I' => "Remote volume close",
 'ANR0538I' => "Resource waiter aborted",
 'ANR0609I' => "Move drmedia processing",
 'ANR0610I' => "Import status",
 'ANR0611I' => "Import status",
 'ANR0614I' => "Import status",
 'ANR0617I' => "Import status",
 'ANR0620I' => "Import status",
 'ANR0621I' => "Import status",
 'ANR0622I' => "Import status",
 'ANR0623I' => "Import status",
 'ANR0624I' => "Import status",
 'ANR0625I' => "Import status",
 'ANR0626I' => "Import status",
 'ANR0627I' => "Import status",
 'ANR0628I' => "Import status",
 'ANR0629I' => "Import status",
 'ANR0638I' => "Import status",
 'ANR0811I' => "Expiration Processing",
 'ANR0812I' => "Expiration Processing",
 'ANR0891I' => "Import status",
 'ANR0911I' => "Archive conversion",
 'ANR0940I' => "Cancel request",
 'ANR0944E' => "No processes found",
 'ANR0984I' => "Stgpool backup",
 'ANR0985I' => "Stgpool backup",
 'ANR0986I' => "Process completion",
 'ANR0987I' => "Expiration Processing",
 'ANR1000I' => "Automatic migration Initiated",
 'ANR1001I' => "Migration ended",
 'ANR1040I' => "Reclamation processesing",
 'ANR1041I' => "Reclamation processesing",
 'ANR1044I' => "Volume required for reclamation",
 'ANR1080W' => "Reclamation processesing",
 'ANR1080W' => "Reclamation processesing",
 'ANR1176I' => "Reclamation processesing",
 'ANR1210I' => "Stgpool backup",
 'ANR1212I' => "Stgpool backup",
 'ANR1213I' => "Cancelled Process",
 'ANR1214I' => "Stgpool backup",
 'ANR1228I' => "Stgpool backup",
 'ANR1340I' => "Scratch definition",
 'ANR1341I' => "Scratch volume deletion",
 'ANR1342I' => "Pending notification",
 'ANR1360I' => "Remote volume open",
 'ANR1361I' => "Remote volume close",
 'ANR1454I' => "Script management",
 'ANR1457I' => "Script management",
 'ANR1462I' => "run command completed",
 'ANR1496I' => "Database backup status",
 'ANR1639I' => "Client attributes changed",
 'ANR1687I' => "Command processing",
 'ANR1688I' => "Command processing",
 'ANR1694I' => "Command processing",
 'ANR1695W' => "Command processing",
 'ANR1697I' => "Command processing",
 'ANR1699I' => "Command processing",
 'ANR2000E' => "Unknown command",
 'ANR2017I' => "Command processing",
 'ANR2017I' => "Command processing",
 'ANR2034E' => "No mounts found",
 'ANR2063I' => "Node updated",
 'ANR2071I' => "Import status",
 'ANR2102I' => "Administrivia",
 'ANR2103I' => "Administrivia",
 'ANR2110I' => "Stgpool backup",
 'ANR2111W' => "Stgpool backup with nothing to do",
 'ANR2149E' => "Import status",
 'ANR2156E' => "Import status",
 'ANR2158E' => "Admin auth",
 'ANR2160E' => "Import status",
 'ANR2202I' => "Storage pool updated",
 'ANR2207I' => "Volume updated",
 'ANR2212I' => "Volume not updated",
 'ANR2281I' => "Database backup status",
 'ANR2369I' => "Expiration Processing",
 'ANR2405E' => "Volume in use",
 'ANR2457E' => "Stgpool backup",
 'ANR2467I' => "Del volhist",
 'ANR2507I' => "Schedule completion",
 'ANR2561I' => "Schedule prompter",
 'ANR2562I' => "Administrivia",
 'ANR2563I' => "Administrivia",
 'ANR2564I' => "Administrivia",
 'ANR2565I' => "Administrivia",
 'ANR2578W' => "Missed schedule",
 'ANR2750I' => "scheduled command invocation",
 'ANR2751I' => "schedule completed",
 'ANR2753I' => "Repeated message in script context",
 'ANR2756I' => "Schedule initiation",
 'ANR4298I' => "Expiration Processing",
 'ANR4388W' => "No expiration to cancel",
 'ANR4389I' => "Expiration Processing",
 'ANR4391I' => "Expiration Processing",
 'ANR4399I' => "Expiration Processing",
 'ANR4551I' => "Database backup status",
 'ANR4554I' => "Database backup status",
 'ANR4711I' => "Import status",
 'ANR4930I' => "Reclamation processesing",
 'ANR4931I' => "Reclamation processesing",
 'ANR4932I' => "Reclamation processesing",
 'ANR4936I' => "Reclamation processesing",
 'ANR6682I' => "Move drmedia processing",
 'ANR6683I' => "Move drmedia processing",
 'ANR6696I' => "Move drmedia processing",
 'ANR6697I' => "Move drmedia processing",
 'ANR8213W' => "Session timeout",
 'ANR8214E' => "Connect refusal",
 'ANR8325I' => "Mount retention expired",
 'ANR8329I' => "Mount status",
 'ANR8330I' => "Volume mount",
 'ANR8333I' => "Mount status",
 'ANR8334I' => "Matches found...",
 'ANR8336I' => "Veifying Label",
 'ANR8337I' => "Volume mount",
 'ANR8340I' => "Remote volume mount",
 'ANR8341I' => "End of volume",
 'ANR8379I' => "Mount status",
 'ANR8468I' => "Dismount notification",
 'ANR8499I' => "Command processing",

};

--
Paul Zarnowski                            Ph: 607-255-4757
Manager, Storage Systems                  Fx: 607-255-8521
719 Rhodes Hall, Ithaca, NY 14853-3801    Em: psz1 AT cornell DOT edu