Hey!
Okay, here's the situation. One of our locations is running a TSM 5.5 storing on two NAS boxes. The TSM instance files (like the volhist and such) are stored on one of these.
I'll try to piece the events together as well as I can, I wasn't there myself.
Last Saturday there was maintenance on the power lines - which meant total blackout for a while, servers where shutdown properly for all I know. The problem is, that the NAS boxes don't automatically come up when power ist restored - so TSM started up without its files and for a few times reported a sequence like this:
17.09.2011 14:25:24 ANR1401W Mount request denied for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS - mount failed.
17.09.2011 14:25:24 ANR1401W Mount request denied for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS - mount failed.
17.09.2011 14:25:24 ANR1401W Mount request denied for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS - mount failed.
17.09.2011 14:25:24 ANR1216E BACKUP STGPOOL: Process 3 terminated - storage media inaccessible.
17.09.2011 14:25:31 ANR1436E Device configuration file \\o1403N1\TSMDATA\server1\devcnfg.out cannot be opened.
17.09.2011 14:25:31 ANR1435E Server could not write device configuration information to \\o1403N1\TSMDATA\server1\devcnfg.out.
17.09.2011 14:25:31 ANR1436E Device configuration file \\o1403N2\TSMDATA\server1\devcnfg.out cannot be opened.
17.09.2011 14:25:31 ANR1435E Server could not write device configuration information to \\o1403N2\TSMDATA\server1\devcnfg.out.
17.09.2011 14:25:31 ANR2396E BACKUP DEVCONFIG: An I/O error was encountered in writing device configuration information to one or more of the defined device configuration files.
17.09.2011 14:25:31 ANR4511E Volume history file \\o1403N1\TSMDATA\server1\volhist.out could not be opened.
17.09.2011 14:25:31 ANR4510E Server could not write sequential volume history information to \\o1403N1\TSMDATA\server1\volhist.out.
17.09.2011 14:25:31 ANR4511E Volume history file \\o1403N2\TSMDATA\server1\volhist.out could not be opened.
17.09.2011 14:25:31 ANR4510E Server could not write sequential volume history information to \\o1403N2\TSMDATA\server1\volhist.out.
He did that several times (got the message 1 hour appart), six times is what I can get from the mails. Now inbetween that power was gone at least one more time.
Inbetween I got messages like
17.09.2011 21:10:16 ANR2121W ATTENTION: More than 263 MB of the database has changed and the last database backup was more than 24 hours ago. Use the BACKUP DB command to provide for database recovery.
Okay, now after that, I got several of these for a handful of volumes:
17.09.2011 21:25:45 ANR9999D_0536094596 (pvrfil64.c:1475) Thread: Error: positioning FILE volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS past EOF.
17.09.2011 21:25:45 ANR1411W Access mode for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS now set to "read-only" due to write error.
Every hour another "xxx MB of the Database have changed" messages. Then these started:
18.09.2011 08:18:31 ANR9999D_2358488677 (pvrfil64.c:2039) Thread: Insufficient bytes read: expected bytes (262144), actual bytes (0).
18.09.2011 08:18:31 ANR1083E Space reclamation is ended for volume \\O1403N2\TSMDATA\FILESERV\00002025.BFS. Excessive read errors were encountered.
That kept coming back every hour.
Now, I saw this yesterday (Monday after the power line maintenance). I checked the NAS Boxes - they are online, their RAID status is normal and the filesystem accessible.
TSM showed 4 volumes "read only" - two in primary, two in copy pools, split over both boxes. I doubt both boxes have failed so I'd say it's a TSM thing .. but what on earth happened?
I restored both primary volumes from the copy pool and hoped I was okay. Last night I got
19.09.2011 22:05:02 ANR9999D_0536094596 (pvrfil64.c:1475) Thread: Error: positioning FILE volume \\O1403N1\TSMDATA\DB-LOG-M-D\00002031.BFS past EOF.
19.09.2011 22:05:02 ANR1411W Access mode for volume \\O1403N1\TSMDATA\DB-LOG-M-D\00002031.BFS now set to "read-only" due to write error.
though ... which gives me chills since I am now afraid this is going to go on and sooner or later lead to some form of data loss.
Any of you got an idea what I can do here?! I had hoped these were corrupted because maybe power was simply cut while TSM was doing something but I am not sure anymore. Any way to check "it all" for consistency?! No tape library here by the way, only the two NAS boxes ...
Thank you!
*edit* I did an audit fix=no on that last volume and it reported no errors ... Any way to find out if other volumes might be affected or what's going on here?
Okay, here's the situation. One of our locations is running a TSM 5.5 storing on two NAS boxes. The TSM instance files (like the volhist and such) are stored on one of these.
I'll try to piece the events together as well as I can, I wasn't there myself.
Last Saturday there was maintenance on the power lines - which meant total blackout for a while, servers where shutdown properly for all I know. The problem is, that the NAS boxes don't automatically come up when power ist restored - so TSM started up without its files and for a few times reported a sequence like this:
17.09.2011 14:25:24 ANR1401W Mount request denied for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS - mount failed.
17.09.2011 14:25:24 ANR1401W Mount request denied for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS - mount failed.
17.09.2011 14:25:24 ANR1401W Mount request denied for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS - mount failed.
17.09.2011 14:25:24 ANR1216E BACKUP STGPOOL: Process 3 terminated - storage media inaccessible.
17.09.2011 14:25:31 ANR1436E Device configuration file \\o1403N1\TSMDATA\server1\devcnfg.out cannot be opened.
17.09.2011 14:25:31 ANR1435E Server could not write device configuration information to \\o1403N1\TSMDATA\server1\devcnfg.out.
17.09.2011 14:25:31 ANR1436E Device configuration file \\o1403N2\TSMDATA\server1\devcnfg.out cannot be opened.
17.09.2011 14:25:31 ANR1435E Server could not write device configuration information to \\o1403N2\TSMDATA\server1\devcnfg.out.
17.09.2011 14:25:31 ANR2396E BACKUP DEVCONFIG: An I/O error was encountered in writing device configuration information to one or more of the defined device configuration files.
17.09.2011 14:25:31 ANR4511E Volume history file \\o1403N1\TSMDATA\server1\volhist.out could not be opened.
17.09.2011 14:25:31 ANR4510E Server could not write sequential volume history information to \\o1403N1\TSMDATA\server1\volhist.out.
17.09.2011 14:25:31 ANR4511E Volume history file \\o1403N2\TSMDATA\server1\volhist.out could not be opened.
17.09.2011 14:25:31 ANR4510E Server could not write sequential volume history information to \\o1403N2\TSMDATA\server1\volhist.out.
He did that several times (got the message 1 hour appart), six times is what I can get from the mails. Now inbetween that power was gone at least one more time.
Inbetween I got messages like
17.09.2011 21:10:16 ANR2121W ATTENTION: More than 263 MB of the database has changed and the last database backup was more than 24 hours ago. Use the BACKUP DB command to provide for database recovery.
Okay, now after that, I got several of these for a handful of volumes:
17.09.2011 21:25:45 ANR9999D_0536094596 (pvrfil64.c:1475) Thread: Error: positioning FILE volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS past EOF.
17.09.2011 21:25:45 ANR1411W Access mode for volume \\O1403N2\TSMDATA\DB-LOG-C-D\0000202D.BFS now set to "read-only" due to write error.
Every hour another "xxx MB of the Database have changed" messages. Then these started:
18.09.2011 08:18:31 ANR9999D_2358488677 (pvrfil64.c:2039) Thread: Insufficient bytes read: expected bytes (262144), actual bytes (0).
18.09.2011 08:18:31 ANR1083E Space reclamation is ended for volume \\O1403N2\TSMDATA\FILESERV\00002025.BFS. Excessive read errors were encountered.
That kept coming back every hour.
Now, I saw this yesterday (Monday after the power line maintenance). I checked the NAS Boxes - they are online, their RAID status is normal and the filesystem accessible.
TSM showed 4 volumes "read only" - two in primary, two in copy pools, split over both boxes. I doubt both boxes have failed so I'd say it's a TSM thing .. but what on earth happened?
I restored both primary volumes from the copy pool and hoped I was okay. Last night I got
19.09.2011 22:05:02 ANR9999D_0536094596 (pvrfil64.c:1475) Thread: Error: positioning FILE volume \\O1403N1\TSMDATA\DB-LOG-M-D\00002031.BFS past EOF.
19.09.2011 22:05:02 ANR1411W Access mode for volume \\O1403N1\TSMDATA\DB-LOG-M-D\00002031.BFS now set to "read-only" due to write error.
though ... which gives me chills since I am now afraid this is going to go on and sooner or later lead to some form of data loss.
Any of you got an idea what I can do here?! I had hoped these were corrupted because maybe power was simply cut while TSM was doing something but I am not sure anymore. Any way to check "it all" for consistency?! No tape library here by the way, only the two NAS boxes ...
Thank you!
*edit* I did an audit fix=no on that last volume and it reported no errors ... Any way to find out if other volumes might be affected or what's going on here?
Last edited:
