ADSM-L

[ADSM-L] Linux inotify to identify changed stuff?

2010-04-21 15:51:11
Subject: [ADSM-L] Linux inotify to identify changed stuff?
From: "Allen S. Rout" <asr AT UFL DOT EDU>
To: ADSM-L AT VM.MARIST DOT EDU
Date: Wed, 21 Apr 2010 15:48:23 -0400
I've got a few clients with rather large filesystems: ~6K dirs, ~2.2M
files, 10 filesystems to the client.

Walking these FS trees, especially given the size of the directories,
can be very slow.  I'm noodling with methods that would let me take
some shortcuts, for daily purposes, and then do a normal incremental
less frequently.

Linux has this neat feature 'inotify'. It lets you erect a low-level
FS monitor on a directory or file, and get warnings when something
happens to it.

I started to write my own direct inotify interface, but then found
inotify-tools, which is capable of dumping stuff out with substantial
flexibility.


My current state of experimentation is using an inotifywait
invocation, and the following PERL code thus:


#  inotifywait  -c -m   -r  [path] | ./timesplit

which yields a series of timestamped "changes" files.  I'll be doing a
periodical

dsmc incr -filelist=/some/path/yadda/timestamped-change-file

in the next phase.



I'm wondering if anyone else is doing something similar, or is
interested in comparing notes.


- Allen S. Rout



-----
#!/usr/bin/perl -- -*-Perl-*-

use warnings;
use strict;
use Getopt::Long;
use Time::CTime;
use FileHandle;



my $outdir = "./out";

my $prefix="changes";

my $formatstring= "%Y-%m-%d--%H-%M";

my $outfile = &newfile() ;

my $last = time;

my $delay = 60;
# Seconds.  If this delay doesn't dependably
# result in a change in the formatted time,
# you'll overwrite stuff.


while (<>)
{
    my $now = time;

    if ( ($now - $last) > $delay  )
    {
        $outfile->close();
        $outfile = &newfile();
        $last = $now;
    }

    my $l = &procline($_);
    defined ($l) && print $outfile  $l;
    print STDERR ".";
}


sub newfile
{
    $last = time;
    my $fname = "$outdir/$prefix-".strftime($formatstring,localtime($last));
    print "\n".$fname."\n";
    my $fh = new FileHandle(">$fname");
    return $fh;
}





sub procline
{
    my @fields;
    undef @fields;
    push(@fields, defined($1) ? $1:$3)
        while m/"([^"\\]*(\\.[^"\\]*)*)"|([^,]+)/g; # \\]

                              my ($path,$op,$fname) = @fields;

#      return undef if ($fname =~ /cyrus.(header|index|cache)/);

                              return "$path$fname";
                          }

-----

<Prev in Thread] Current Thread [Next in Thread>