Bacula-users

Re: [Bacula-users] Files are added to catalog at the end of backup

2017-02-02 09:15:55
Subject: Re: [Bacula-users] Files are added to catalog at the end of backup
From: Daniel Heitepriem <daniel.heitepriem AT pribas DOT com>
To: Kern Sibbald <kern AT sibbald DOT com>, "bacula-users AT lists.sourceforge DOT net" <bacula-users AT lists.sourceforge DOT net>
Date: Thu, 2 Feb 2017 14:59:06 +0100

Hello Kern,

thank you for your answer. Here is my job definition:

Job {
  Name = "monitor-host.logs"
  JobDefs = "DefaultJob"
  Storage = EAST
  Schedule = "Early"
  Write Bootstrap = "/opt/bacula/var/bacula/working/monitor-host.bsr"
  Client = monitor-host
  FileSet = "monitor-host.logs"
  RunScript {
    RunsWhen = After
    FailJobOnError = No
    Command = "/opt/bacula/libexec/cleanup.sh %i"
    RunsOnSuccess = Yes
    RunsOnClient = Yes
  }
}

FileSet {
  Name = "monitor-host.logs"
  Include {
    Options {
      signature = MD5
      compression=GZIP
    }
    File = /logs/nagios/
  }
  Exclude {
    File = /logs/nagios/nagios.log
  }
}

As you can see the jox uses the "RunScript"-directive. Here is the cleanup.$jobid$.set file

llist files jobid="79"
Automatically selected Catalog: MyCatalog
Using Catalog "MyCatalog"
           JobId: 79
             Job: monitor-host.logs.2017-02-01_23.59.00_14
            Name: monitor-host.logs
     PurgedFiles: 0
            Type: B
           Level: I
        ClientId: 31
      ClientName: monitor-host
       JobStatus: R
       SchedTime: 2017-02-01 23:59:00
       StartTime: 2017-02-01 23:59:11
         EndTime: NULL
     RealEndTime: NULL
        JobTDate: 1,485,989,951
    VolSessionId: 0
  VolSessionTime: 0
        JobFiles: 0
        JobBytes: 0
       ReadBytes: 0
       JobErrors: 0
 JobMissingFiles: 0
          PoolId: 1
        PooLname: Incremental
      PriorJobId: 0
       FileSetId: 65
         FileSet: monitor-host.logs
         HasBase: 0
        HasCache: 0
         Comment:

You have messages.

and the corresponding log file

-bash-4.1$ cat cleanup.79.log
Cleaning job 79
-bash-4.1$

When I run the file daemon with debug options this is the output when the script is running:

HOST-fd: job.c:314-79 Quit command loop. Canceled=0
HOST-fd: runscript.c:99-79 runscript: running all RUNSCRIPT object (ClientAfterJob) JobStatus=T
HOST-fd: runscript.c:120-79 runscript: try to run *None*:/opt/bacula/libexec/cleanup.sh %i
HOST-fd: runscript.c:158-79 runscript: Run it because SCRIPT_After (/opt/bacula/libexec/cleanup.sh %i,1,0,T)
HOST-fd: runscript.c:219-79 runscript: running a RUNSCRIPT object type=79
HOST-fd: util.c:831-79 edit_job_codes: /opt/bacula/libexec/cleanup.sh %i
HOST-fd: runscript.c:226-79 runscript: running '/opt/bacula/libexec/cleanup.sh 79'...
HOST-fd: runscript.c:254-79 runscript OK

As you can see the output of "llist files jobid=79" is empty and therefore the script does not run as intended. I hope that I made this a little bit clearer.

Best regards,

Daniel


Am 02.02.17 um 14:39 schrieb Kern Sibbald:
Hello,

When a Bacula backup job terminates, all the File table entries are already in the catalog.  So about the only thing that makes any sense is that you are running the script before the backup completes.  Perhaps by simply adding a
"wait" just before your llist, would solve the problem.  However, be aware "wait" waits until no jobs are running which may delay the output if you are running multiple jobs.  There is a "wait JobId=nnn" but I am not 100% sure it really works correctly (I have a vague recollection of some problems).

Alternatively, it seems to me that use the appropriate RunScript + options to do the work, you can guarantee that it runs at the end of the job.

Best regards,
Kern

On 02/02/2017 09:57 AM, Daniel Heitepriem wrote:
Hi guys,

I'm using Bacula 7.4.4 and got a script which runs after every backup
and expects a list of file names which were backed up. Now I noticed
that bacula is writing the files to the Catalog at the end of the backup
process, thus the script receives an empty list and doesn't run
properly. Below I got an excerpt of this script:

export HOME; HOME="/opt/bacula"
export LOG; LOG="$HOME/var/cleanup.$1.log"

echo "Cleaning job $1">>$LOG

$HOME/bin/bconsole <<EOF
@output /dev/null
messages
@output $HOME/var/cleanup.$1.set
llist files jobid="$1"
EOF


for file in $( egrep "^\|" $HOME/var/cleanup.$1.set | sed -e "s/\|//g" )
do
    if [ $file != "Filename" ]
    then
      AGE=$(perl -e  "print (-M \"$file\");"|cut -d'.' -f1) ## Get age of
file

      if test -d $file ; then ## If line is  a directory
        echo "Directory $file ... skipping...">>$LOG
      elif echo $file|egrep '.gz$' 2>&1 >/dev/null ; then ## Test if file
has ".gz" suffix
        if [ "$AGE" -gt "$KEEP" ] ; then
          echo "$file is older than $KEEP days -> removing">>$LOG
          pfexec rm $file  ## Removing files older than 7 days
        else
          echo "$file is younger than $KEEP days -> keeping">>$LOG

In Bacula 5 this issue isn't present so the file list is available
almost instantly. Did something change between the versions regarding
this behavior?

Thanks and best regards,
Daniel Heitepriem

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users



--
Mit freundlichen Gruessen / Best regards
Daniel Heitepriem

pribas GmbH

Valterweg 24-25
65817 Eppstein-Bremthal
Germany

Phone  +49 (0) 6198 57146400
Fax       +49 (0) 6198 57146433
eMail   daniel.heitepriem AT pribas DOT com

Corporate Headquarters: Huenfelden-Dauborn Managing Director: Arnulf Pribas
Registration: Amtsgericht Limburg a. d. Lahn 7HRB874 Tax ID: DE113840457
________________________________________________________________________
This e-mail is confidential. Information in this e-mail is intended for the exclusive use of the individual or entity named above and may constitute information that is privileged or confidential or otherwise protected from disclosure. The information in this e-mail may be read, published, copied and/or forwarded only by the individual or entity named above. Dissemination, distribution, forwarding or copying of this e-mail by anyone other than the intended recipient is prohibited. If you have received this e-mail in error, please notify us immediately by telephone or e-mail and completely delete or destroy any and all disseminated, distributed, forwarded electronic or other copies of the original message and any attachments.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Bacula-users mailing list
Bacula-users AT lists.sourceforge DOT net
https://lists.sourceforge.net/lists/listinfo/bacula-users
<Prev in Thread] Current Thread [Next in Thread>