Expiring Inventory

[Jcage]

I've discovered we haven't run expire inventory on directories forever. I started running it tonight and it's been running for 6 hours and filling up the log. Should I be concerned about this ? Also during the process it says



Examined 2670744 objects, deleting 1207821 backup and it's growing. Is it actually deleting objects or does the expiration process have to complete before it deletes these.

 

[M@rco]

If it's the first time you're running expiration, it takes extra time because there are probably a lot of expired files.

Files are deleted immediatly for as far as I know.

 

[heada]

Expire inventory removes the entries as they are expired. Also, it will start were it left off so if you cancel it, it will not hurt the process. Thirdly, you can use the "quiet=yes" option and it will not put anything into the activity log except the start and finish entries.



-Aaron

 

[Jcage]

<TABLE BORDER=0 ALIGN=CENTER WIDTH=85%><TR><TD><font class="pn-sub">Quote:</font><HR></TD></TR><TR><TD><FONT class="pn-sub"><BLOCKQUOTE>Expire inventory removes the entries as they are expired. Also, it will start were it left off so if you cancel it, it will not hurt the process. Thirdly, you can use the "quiet=yes" option and it will not put anything into the activity log except the start and finish entries.



-Aaron</BLOCKQUOTE></FONT></TD></TR><TR><TD><HR></TD></TR></TABLE>



I just checked and we do have quiet=yes. So that means it wasn't the expire inventory filling up the log ????



I think I know what's happening. We have the dataspace trigger for the database set at 85%, it's been sitting at 90% for a few days. Does it write to the recovery log once the dataspace limit has been exceeded ?

 

[Jcage]

Does it metter if I run expiration during the database backup ?

 

[cheffern]

Hi -

If you haven't run expiration in a while - this can take a long time like you are seeing. It does delete the files as expiration runs -
However, you won't reap the benefits fully until your reclamation process has run to regain the now blank tape space due to expiration and consolidate the leftover existing files.
So even though it is still going - it is best to let it run it's course.

Hope that helps.
-Chef.

 

[cheffern]

Examined 2670744 objects, deleting 1207821 backup and it's growing. Is it actually deleting objects or does the expiration process have to complete before it deletes these.

Hi -

If you haven't run expiration in a while - this can take a long time like you are seeing. It does delete the files as expiration runs -
However, you won't reap the benefits fully until your reclamation process has run to regain the now blank tape space due to expiration and consolidate the leftover existing files.
So even though it is still going - it is best to let it run its course.

Hope that helps.
-Chef.

 

[Jcage]

Hi -

If you haven't run expiration in a while - this can take a long time like you are seeing. It does delete the files as expiration runs -
However, you won't reap the benefits fully until your reclamation process has run to regain the now blank tape space due to expiration and consolidate the leftover existing files.
So even though it is still going - it is best to let it run its course.

Hope that helps.
-Chef.

Great thanks !

 

[Jcage]

One more thing, even though expired the inventory with quiet=yes it still seems to be writing to the log. Is there something else I should check to make sure it doesn't write to the log ?

 

[cheffern]

From a command line, run "dsmadmc -console"
or
"dsmadmc -id=login -password=passwd -consolemode"

So this will show you real-time logging -
And if you still see stuff related to your expiration - (if you used the quiet option, you shouldn't see anything but start and stop entries...)
Then stop the expiration and retry it with quiet mode again.

How much growth are we talking about in your log at this point, anyway?

-Chef.

 

[Jcage]

From a command line, run "dsmadmc -console"
or
"dsmadmc -id=login -password=passwd -consolemode"

So this will show you real-time logging -
And if you still see stuff related to your expiration - (if you used the quiet option, you shouldn't see anything but start and stop entries...)
Then stop the expiration and retry it with quiet mode again.

How much growth are we talking about in your log at this point, anyway?

-Chef.

ABout .1% every 45 seconds to a minute.

 

[Jcage]

It's been running 2 days non-stop, it's deleted over 5 gig worth of data. It's not possible I'm deleting valid data is it ?

 

[Jcage]

It's been running 2 days non-stop, it's deleted over 5 gig worth of data. It's not possible I'm deleting valid data is it ?

Any ideas guys ? We've now deleted 8 gig and I'm starting to wonder if I'm actually deleting stuff that is valid ? It will only delete expired data right ? If it's expired it's not accessible anyway is it ?

 

[rscherrer47]

Yes it Does

Think about what expiration is really doing...it is removing database entries, so to backup the database at the same time, if it comes down to doing a restore from that copy, you will have orphan entries.

And yes, if your data resides on tape, then reclaim will remove the final "data" from the system as it consolidates what is "still alive" and what just got deleted.

Rule of thumb says run it everday at a certain time...not during client backups if possible. Simple admin schedule expire inventory quiet=yes will do the trick.

In the dsmserv, set expiration to "quiet". This will save the entries from going into the activity log.

Hope this helps...

 

[Jcage]

Think about what expiration is really doing...it is removing database entries, so to backup the database at the same time, if it comes down to doing a restore from that copy, you will have orphan entries.

And yes, if your data resides on tape, then reclaim will remove the final "data" from the system as it consolidates what is "still alive" and what just got deleted.

Rule of thumb says run it everday at a certain time...not during client backups if possible. Simple admin schedule expire inventory quiet=yes will do the trick.

In the dsmserv, set expiration to "quiet". This will save the entries from going into the activity log.

Hope this helps...

OK, I understand why we shouldn't do it during database backup now that you've explained it but why not during client backups ? We only have a window between 2pm and 4pm where we're not doing client or database backups. It seems everytime I stop it it scans all objects again and a 2 hour window won't be enough to expire everything.

 

[cheffern]

Hi J -

During client backups you are adding entries for any particular client.
You don't really want to do that while it is in the middle of expiring data for a client... It can load the database as well as maybe even confuse it (the orphan issue).

I still think that once it does its initial run and really cleans things up for you - the next run is going to be faster. My expiration process takes about 10-20 min every day - longer if I have manually deleted anything it needs to catch up with. And my database is about 150 GB.

And obviously this is something you *have* to do. Otherwise you'll just be keeping everything forever-like and your db will keep growing and growing. You need to run this expiry process for a healthy database...

-Chef.

 

[mateinone]

Expire inventory removes the entries as they are expired. Also, it will start were it left off so if you cancel it, it will not hurt the process. Thirdly, you can use the "quiet=yes" option and it will not put anything into the activity log except the start and finish entries.



-Aaron

Can you confirm that as accurate? Because I am not sure that is completely accurate though I could very easily be wrong.

I believe if you cancel the process with a
can pr [1234]
then you will in fact kill the expiration task and not actually delete any entries in the database. When expirations starts again it will need to start again.

However if you cancel the process cleanly with a
can expiration
Then the entries marked for deletion are deleted and the process is stopped.

So if you have a job that kills all processes at a certain time each day (and expiration is one of those jobs) then you will in fact never complete expirations.

 

[cheffern]

Good catch -

Though the documentation on expire inventory doesn't state it this way -
Re: EXPIRE INVENTORY "This command creates a background process that can be canceled with the CANCEL PROCESS command."

Unfortunately, I don't see anything in the online docs that gives a statement about what actually happens when you issue CANCEL EXPIRATION vs a CANCEL PROCESS.

But surely there is a reason for having the CANCEL EXPIRATION command???

Here is something that should be of interest to you JCage -
"Task
Run the expiration process for two hours.

Command
expire inventory duration=120 "

Perhaps you can cancel the expiration that is running now and retry it for the 2 hours you need...maybe - if Mate is correct - then in a few days or so you may be cleaned out properly...

Good luck -
-Chef.