Thanks to everyone who helped on this issue.
All your suggestions kept us moving down the right path and eliminated a lot of
speculation.
The symptom was a VE RESTORE VM that kept mounting tapes over and over and
over, but never completed and never failed. We even did a move nodedata for
the filespace of the VM back to disk, and still got the same results - kept
issuing zillions of tape mounts for something that shouldn't be on disk.
Fortunately, my customer was able to allocate enough disk to finally move all
the VE filespaces back to disk. THEN the restore attempt generated the error
message "ANS0323E (RC33) partialObjOffset value for partial object retrieve
is invalid ", which is indicative of IC91076.
Customer upgraded the server to 6.3.4 at direction of L2 support, restores
worked, VM's OK, all is well.
Many thanks to all the wonderful people who contribute to this list!
Wanda
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Prather, Wanda
Sent: Friday, June 21, 2013 11:52 AM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] Another VE mystery - restoring from tape - but shouldn't
be
Thanks Andy,
I'll ask the customer for that info.
W
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Andrew Raibeck
Sent: Wednesday, June 19, 2013 12:20 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] Another VE mystery - restoring from tape - but shouldn't
be
Hi Wanda,
Identify one of the tapes being mounted from TAPEPOOL2, then do a QUERY CONTENT
on it. See if there is anything for DC1. There has to be
*something* even if we haven't yet put our finger on it.
Best regards,
- Andy
____________________________________________________________________________
Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead | storman AT
us.ibm DOT com
IBM Tivoli Storage Manager links:
Product support:
http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager
Online documentation:
https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli
Documentation Central/page/Tivoli Storage Manager Product Wiki:
https://www.ibm.com/developerworks/mydeveloperworks/wikis/home/wiki/Tivoli
+Storage+Manager/page/Home
"ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu> wrote on 2013-06-19
11:53:07:
> From: "Prather, Wanda" <Wanda.Prather AT ICFI DOT COM>
> To: ADSM-L AT vm.marist DOT edu,
> Date: 2013-06-19 11:55
> Subject: Re: Another VE mystery - restoring from tape - but shouldn't
> be Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT vm.marist DOT edu>
>
> Richard,
> Here is the Q OCC result.
> The Q OCC shows all the data for FSID 86 is in VMCTLMC (seq disk),
> SLOWDEDUP (seq disk) or the COPYPOOL (tape COPY pool).
>
> But when the customer tries a restore, the tapes getting mounted are
> from primary pool TAPEPOOL2.
> How can a restore be calling for tapes in a pool where the filespace
> has no data according to Q OCC?
> I suggested they open a sev 1 with Tivoli. This can't be right.
>
>
> tsm: HCG-TSM-SERVER>q occ dc1 86 nametype=fsid
>
> Node Name Type Filespace FSID Storage
> Number of Physical Logical
> Name Pool Name
> Files Space Space
>
> Occupied Occupied
>
> (MB) (MB)
> ---------- ---- ---------- ----- ----------
> --------- --------- ---------
> DC1 Bkup \VMFULL-H- 86 COPYPOOL
> 8,928 99,057.72 99,062.73
> WDDMZOCU-
> LARX1
> DC1 Bkup \VMFULL-H- 86 SLOWDEDUP
> 4,464 - 98,733.95
> WDDMZOCU-
> LARX1
> DC1 Bkup \VMFULL-H- 86 VMCTLMC
> 4,464 315.14 315.14
> WDDMZOCU-
> LARX1
>
> -----Original Message-----
> From: Prather, Wanda
> Sent: Wednesday, June 19, 2013 11:03 AM
> To: Richard Cowen
> Subject: RE: Another VE mystery - restoring from tape - but shouldn't
> be
>
> >>Did the MOVE NODEDATA result in a zillion new volumes (.BFS's) ?
> No, we don't use scratch volumes in the seq disk pool
>
>
> >>Can you get the activity log for the time the process(es) ran ?
> Any chance you have query occupancy node=dcname filespace=victim1
> stgpool=<tape,disk> before and after move?
> If not, does the query for tape pool now show zero?
>
> Don't have them and right now I don't have access, but those are great
> ideas, will ask the customer for them.
>
> Thanks!
>
> Wanda
>
> -----Original Message-----
> From: Prather, Wanda [mailto:Wanda.Prather AT icfi DOT com]
> Sent: Tuesday, June 18, 2013 11:35 AM
> To: Richard Cowen
> Subject: RE: Another VE mystery - restoring from tape - but shouldn't
> be
>
> Hi Richard,
>
>
> >>Did you get a "zillion" tape mounts during the MOVE NODEDATA?
> Yes
>
>
> >>Do you know it finished without errors?
> Yes. And we ran another MOVE NODEDATA to verify there was no more
> data to move for that filespace.
>
> >>How, exactly, did the data go from sequential fast disk ->
> sequential slow disk - > tape ?
> Ordinary migration, at different times, as the pools hit migration
thresholds.
>
> >>I didn't think TSM would "migrate" more than one level, so maybe
> the last step was using a MOVE command?
> Yes, your migration hierarchy can have as many levels as you want, as
> long as you don't try to go from a sequential pool back to a disk
> (random) pool.
>
> >>What does a QUERY NODEDATA show for primary pools and copy pools?
> Would not be informative, as we only moved some of the filespaces, not
> all of them.
>
> >>Are you running aggressive reclamation on the tape pool?
> No, and it's not collocated. Which is why we decided to move the
> filespace back to the seq disk pool.
> I don't think it's odd the data was too spread out to make the restore
> from tape not work well.
> What is odd is that all the data from one VM is supposedly in one
filespace.
> We moved that filespace back to disk with move nodedata fsid= But
> still getting tape mounts on the restore.
>
> Thanks for your interest!
>
> Wanda
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> Of Prather, Wanda
> Sent: Monday, June 17, 2013 9:45 PM
> To: ADSM-L AT VM.MARIST DOT EDU<mailto:ADSM-L AT VM.MARIST DOT EDU>
> Subject: [ADSM-L] Another VE mystery - restoring from tape - but
shouldn't be
>
> TSM 6.2.3
> Backup done with TSM VE 6.4.0.0 and TSM client 6.4.0.0.
>
> VMCTLMC points to a dedicated sequential pool on fast disk. That pool
> has no NEXTSTGPOOL defined.
>
> VMMC points backup data to a deduplicated sequential pool on fast disk
> storage.
> After the server dedups it there, it migrates to a slower NAS-based
> deduplicated storage pool on disk.
> When that slower pool fills, data migrates out to tape.
> DEDUPREQUIRESBACKUP set to YES.
> (No client-side dedup used.)
>
> We have done full VM restores through the plug-in and file-level
> restores through the recovery agent in the past, including testing
> restores from tape, with no problems.
>
> Now one of the customer's VM datastores has met with an unfortunate
> accident in a dark alley.
> We need to restore 7 full VM's.
>
> From the dsmc command line, restore vm victim1 datastore=newhealthyone
> starts up OK, but was calling for a zillion tape mounts, and therefore
> was restoring at the rate of about 4GB per 24 hours.
>
> So, we did MOVE NODEDATA DCNAME FILESPACE=victim1 to bring the data
> from the tape back to the sequential disk pool.
> Cranked up again, same result - requesting a zillion tape mounts.
>
> So riddle me this:
> If the control information is on disk, and the filespace is back on
> disk, what are the tape mounts going after?
>
> (FWIW, tried upgrading VE to 6.4.0.1 and data mover to 6.4.0.4, no
> difference.) Signed, Confused by VE. Again.
>
> Wanda Prather | Senior Technical Specialist |
> Wanda.Prather AT icfi DOT com< mailto:Wanda.Prather AT icfi DOT com> |
> www.icfi.com<http://www.icfi.com> ICF International | 401 E. Pratt
> St, Suite 2214, Baltimore, MD
> 21202 | 410.539.1135 (o)
>
|
