Backup Failing with "Server out of data storage space" Error

[RandomAdmin]

Hello all,

I'm a complete newbie to the TSM world and, as often happens, I was tasked to take care of the product without having proper training or know how transfer, anyhow I am trying my best to learn it so here I am.
I've already search the same issue among older posts but it seems I've hit a blocker and would need some help from the experts.

As per subject I have a server, more specifically an Exchange server, with failing backups due the above error message, TSM Server is 6.3 running on top of a Windows 2008 box all Exchange backups go to disk storage pool and then to tape library an IBM 3592.

Storage         Device          Estimated       Pct       Pct     High     Low     Next Stora-
Pool Name       Class Name       Capacity      Util      Migr      Mig     Mig     ge Pool
                                                                   Pct     Pct
-----------     ----------     ----------     -----     -----     ----     ---     -----------
TAPEPOOL_T-     3592_TL01       156,771 G      50.6      66.0       80      60
L01
TAPEPOOL_T-     3592_TL02       155,778 G      50.4
L02

DISKPOOLCO-     DISK              1,005 G      87.0      86.7       80      60     TAPEPOOL_T-
LL                                                                                 L01

From the above output I've removed archive pools which are not used as part of the backup to keep output short, from what I understand diskpool is 87% full but on tape pool I should have plenty of space but yet the backup fails.

My question would be how can I make certain there is enough space to write the data? Put differently how I can verify which of the stgpool does not have enough space? The message itself is self explanatory but not helping me nail the issue.

It has to be noted that we received an alert about not having scratch medias, could this be the issue? Is there a way I can check how many scratch medias I do have? I will try to build up a query but if you have a ready answer would be much appreciated.

Last but not least in many posts I've read people mentioning the actlog to check out TSM events, how can I check this log file?

Thanks in advance and sorry for the lengthy post full of questions as I said I'm noobie and trying to learn.

Thanks!

 

[LED888]

"Server out of data storage space"

Short version -
- Is the disk pool full? How large is the diskpool?
Is the disk pool large enough to contain all the data. If not increase the disk pool. If we are not able to increase the disk pool. Migrate all the data out of the disk pool prior the TDP backup. If the disk pool still not large enough and we can not increase the capacity.
Use a Storage Agent so that the data will go directly to the tape library.

- Is there a next storage pool to a tape library? If not only solution increase disk capacity.
- If there is a next storage pool to a tape library, why is the data not migrating to the next storage pool?
Are there plenty of scratch tapes in the library?
If not, add more scratch tapes.
If there are plenty of scratch tapes, did we reach the max scratch limit
( q stg pool_name f-d). If we did reach the max scratch limit, increase this value
(update stg pool_name maxscratch=##).

If we have not reach the maxscratch limit.
Lower the migration threshold and review the actlog for any errors.

Good Luck,
Sias

 

[RandomAdmin]

Long time no speak Led .

Trying to answer your questions:

- Disk Storage pool is ~1TB in size with PCT Util 87, unfortunately I cannot increase it as it is configured over a dedicated SAN with no disk space left to provision
- Next Storage pool is defined and it is the primary library
- Not enough scratch tapes are available, there 0, so I guess this is the main issue. I'm trying to check-in new tapes but I think something went wrong in the process of physically putting them in the library (I was not the one labelling and importing tapes) as I can see them physical library but when I try to check in them in TSM nothing happens, or at least the process run for a bit but the tape is nowhere to be found (Library is an IBM 3592)
- Migration threshold is set to 60 but due lack of scratch tapes the process fails

Two more questions as you mention them:

- How can I check the actlog?
- I'll check how I can use storage agent for TDP but if you have any pointer it would be appreciated.

*EDIT* The command I used to import tapes is:

checkin libvol library_name search=yes checklabel=barcode status=scratch

Again thanks and sorry for being such a burden.

 

[rmazzon]

Random,

Try "search=bulk" in your checkin command. And try "help query actlog" for the syntax of that command. If you still cannot see the tapes being checked in then log into the library web interface and determine if they are indeed in the library. Perhaps they were not inserted through the I/O door and an audit of the library is required.

As already noted the issue of your first questions mostly likely is a lack of scratch tapes. There may be 130 GB of space left in your disk pool but if the Exchange databases are larger than that they will automatically try to go to the tape pool since the client would have first checked to see how much space was available in the disk pool. Alternatively, perhaps you have set the "maximum size threshold" on your disk pool and that is sending the data to tape.

I personally would try to stay away from storage agents. If your disk pool is large enough to hold a night's backup and the backup performance is sufficient then you should be OK once you have sufficient scratch tapes and start monitoring your scratch levels.

 

[RandomAdmin]

Rmazzon,

thanks for taking the time to answer, yes I think the root cause of the issue is the lack of scratch tapes as definitely the space in the diskpool is not enough for the Exchange DBs which are growing too fast for my taste.

I'll now try your suggestion and see if I can get the tapes correctly recognized, as per the I/O door insertion, despite not having a full picture of the concept, I've read about it and checked the library where Virtual I/O is enabled, not sure if this makes any difference or could be the issue itself what I know for a fact tapes are indeed in the library and correctly recognized as I can see from the web gui.

I will now educate myself about the actlog and library audit and will report here once I know more.

It's 10pm over here but I've been challenged and want to solve this now All in all I'm enjoying the experience and learned more TSM in one day subscribed to this forum that reading lot of redbooks pages.

Thanks all for the invaluable help!