• Please help support our sponsors by considering their products and services.
    Our sponsors enable us to serve you with this high-speed Internet connection and fast webservers you are currently using at ADSM.ORG.
    They support this free flow of information and knowledge exchange service at no cost to you.

    Please welcome our latest sponsor Tectrade . We can show our appreciation by learning more about Tectrade Solutions
  • Community Tip: Please Give Thanks to Those Sharing Their Knowledge.

    If you receive helpful answer on this forum, please show thanks to the poster by clicking "LIKE" link for the answer that you found helpful.

  • Community Tip: Forum Rules (PLEASE CLICK HERE TO READ BEFORE POSTING)

    Click the link above to access ADSM.ORG Acceptable Use Policy and forum rules which should be observed when using this website. Violators may be banned from this website. This notice will disappear after you have made at least 3 posts.

Bottomless pit

zforray

ADSM.ORG Member
#1
Here is a new one.......

We turned off backing up SystemState last week. Now I am going through and deleted the Systemstate filesystems.

Since I wanted to see how many objects would be deleted, I did a "Q OCCUPANCY" and preserved the file count numbers for all Windows nodes on this server.

For 4-nodes, the delete of their systemstate filespaces has been running for 5-hours. A "Q PROC" shows:

2019-02-25 08:52:05 Deleting file space ORION-POLL-WEST\SystemState\NULL\System State\SystemState (fsId=1) (backup data) for node ORION-POLL-WEST: 105,511,859 objects deleted.

Considering the occupancy for this node was ~5-Million objects, how has it deleted 105-Million objects (and counting). The other 3-nodes in question are also up to >100-Million objects deleted and none of them had more than 6M objects in occupancy?

At this rate, the deleting objects count for 4-nodes systemstate will exceed 50% of the total occupancy objects on this server that houses the backups for 263-nodes?

I vaguely remember some bug/APAR about systemstate backups being large/slow/causing performance problems with expiration but these nodes client levels are fairly current (8.1.0.2 - staying below the 8.1.2/SSL/TLS enforcement levels) and the ISP server is 7.1.7.400. All of these are Windows 2016, if that matters.
 

mikeymac

ADSM.ORG Moderator
#2
Whoa! That's a new one by me! In the past, TSM would famously re-introduce old bugs in new versions of the software! I think those days are gone, though.

That's a ludicrous number of objects for SystemState. I'd get a PMR with IBM going.

Good luck, and let us know what they find!
 

zforray

ADSM.ORG Member
#3
Whoa! That's a new one by me! In the past, TSM would famously re-introduce old bugs in new versions of the software! I think those days are gone, though.

That's a ludicrous number of objects for SystemState. I'd get a PMR with IBM going.

Good luck, and let us know what they find!
Thanks for the confirmation that I am not the only one seeing it and wondering what is going on. FWIW, the deletes all failed/crashed with strange "unexpected error 4522 fetching row in table "Backup.Objects" (or Filespaces). The last "q proc" I recorded:

2,325 DELETE FILESPACE Deleting file space ORION-POLL-W2\SystemState\NULL\System State\SystemState (fsId=1) (backup data) for node ORION-POLL-W2: 119,442,593 objects deleted.
2,326 DELETE FILESPACE Deleting file space ORION-POLL-E2\SystemState\NULL\System State\SystemState (fsId=1) (backup data) for node ORION-POLL-E2: 116,621,727 objects deleted.

Then I see this in the logs:

2/25/2019 3:07:29 PM ANR1893E Process 2324 for DELETE FILESPACE completed with a completion state of FAILURE.
2/25/2019 3:32:53 PM ANR0106E imfs.c(8340): Unexpected error 4522 fetching row in table "Filespaces".
2/25/2019 3:32:53 PM ANR0106E imfsdel.c(2723): Unexpected error 4522 fetching row in table "Backup.Objects".
2/25/2019 3:32:53 PM ANR1893E Process 2325 for DELETE FILESPACE completed with a completion state of FAILURE.
2/25/2019 4:29:26 PM ANR0106E imfsdel.c(2723): Unexpected error 4522 fetching row in table "Backup.Objects".
2/25/2019 4:29:26 PM ANR1893E Process 2326 for DELETE FILESPACE completed with a completion state of FAILURE.
 

zforray

ADSM.ORG Member
#4
Whoa! That's a new one by me! In the past, TSM would famously re-introduce old bugs in new versions of the software! I think those days are gone, though.

That's a ludicrous number of objects for SystemState. I'd get a PMR with IBM going.

Good luck, and let us know what they find!
This is getting interesting. On another ISP server, a similarly named node (I think it is the same application), a similar problem. This is breaking all kinds of records - might hit 200M objects - which is interesting since the total objects on this server is 493M


2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL\System State\SystemState (fsId=1) (backup data) for node ORIONADDWEB: 189,078,701 objects deleted.
 

mikeymac

ADSM.ORG Moderator
#5
This is getting interesting. On another ISP server, a similarly named node (I think it is the same application), a similar problem. This is breaking all kinds of records - might hit 200M objects - which is interesting since the total objects on this server is 493M


2019-02-26 08:57:56 Deleting file space ORIONADDWEB\SystemState\NULL\System State\SystemState (fsId=1) (backup data) for node ORIONADDWEB: 189,078,701 objects deleted.
I'd definitely get that PMR going! This is obviously not normal behavior! o_O
 

moon-buddy

ADSM.ORG Moderator
#6
zforray,

You are not the only one that has seen this. I am running TSM Server 7.1.7.1 and I have one node that has 5 meg of data left. I had started deleting filespace and ran for days; the number of objects went pass 200 million and all was pointed to system state.
 

mikeymac

ADSM.ORG Moderator
#7
zforray,

You are not the only one that has seen this. I am running TSM Server 7.1.7.1 and I have one node that has 5 meg of data left. I had started deleting filespace and ran for days; the number of objects went pass 200 million and all was pointed to system state.
Did your filespace finish deleting?
 

zforray

ADSM.ORG Member
#9
I had to take multiple whacks at deleting systemstate filespaces across 3-days but it was finally deleted (kept failing with errors like:
2/26/2019 2:24:27 PM ANR0106E imfsdel.c(2723): Unexpected error 4522 fetching row in table "Backup.Objects".
2/26/2019 2:24:27 PM ANR1880W Server transaction was canceled because of a conflicting lock on table BACKUP_OBJECTS.
2/26/2019 2:24:27 PM ANR1893E Process 2399 for DELETE FILESPACE completed with a completion state of FAILURE.)

In the end, finally deleted 1.2B (that is Billion) systemstate objects from 4-nodes.

IBM did refer to an this article/webpage: https://social.technet.microsoft.co...s-is-filling-my-disk-space?forum=winservergen

and my OS engineer for these servers did say the \Crypto\RSA keys folder does exist on at least one of these machines.

The ISP server that ended up deleting 300M+ systemstate objects for 1-node dropped 2M total occupancy objects.

I am glad to have finally purged almost all systemstate backups (folks who manage AD servers requested keeping it for them since they have used TSM to restore AD objects in the past).
 

moon-buddy

ADSM.ORG Moderator
#10
This happened again; filespace deletion started three days ago.

How can a server, with only 2.5M objects as reported by the select command be deleting 1.8+ billion obejcts and still going?

While the delete filespace was running, we monitored the file objects. This time the objects were reducing in count but at a rate that is 10 to 20 times slower than the filespace delete.

This time, the TSM server was at 7.1.7.1; last time it was at 7.1.5.2. My conclusion is that the issue is not server bound per se but may be a Node + Server combination, and/or Windows version. The node is a Windows 2008.

Bottom line, the filespace finally deleted after three days.

We still have to find the root cause so a PMR with IBM is in order.
 
Last edited:

Advertise at ADSM.ORG

If you are reading this, so are your potential customer. Advertise at ADSM.ORG right now.

UpCloud high performance VPS at $5/month

Get started with $25 in credits on Cloud Servers. You must use link below to receive the credit. Use the promo to get upto 5 month of FREE Linux VPS.

The Spectrum Protect TLA (Three-Letter Acronym): ISP or something else?

  • Every product needs a TLA, Let's call it ISP (IBM Spectrum Protect).

    Votes: 15 18.1%
  • Keep using TSM for Spectrum Protect.

    Votes: 52 62.7%
  • Let's be formal and just say Spectrum Protect

    Votes: 9 10.8%
  • Other (please comement)

    Votes: 7 8.4%

Forum statistics

Threads
31,403
Messages
133,763
Members
21,511
Latest member
rcoelho
Top