Thanks, Andy.
We are continuing to troubleshoot. It is difficult since I can't replicate the
problem on every backup, and our end users are a bit uneasy being our "guinea
pigs". I will double-check, but pretty sure we have VE configured to force the
use of NBD, never Hotadd.
Thanks again for the info, I'll let the list know if we find the "smoking gun".
-steve
-----Original Message-----
From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf Of
Andrew Raibeck
Sent: Wednesday, March 11, 2015 3:19 PM
To: ADSM-L AT VM.MARIST DOT EDU
Subject: Re: [ADSM-L] VE 7.1.1.1 backup "freezing" a VM & question about
"megablocks"
Hi Steve,
No firm diagnosis or solution, but this could be something going on during
snapshot cleanup, when the volumes are being unmounted if you are using the
HOTADD transport. One possibility (but by no means the only one) is:
kb.vmware.com/kb/2010953
One way to try to isolate this is to manually create a snapshot, and leave it
in place for the same amount of time the backup normally runs for that VM. Do
you see the frozen VM issue occur? Next, remove the snapshot you just created.
Does the VM seem to freeze while the snapshot is being removed?
The big factor is how long the backup took and how busy the target VM's I/O
was. The longer the backup with high I/O, the larger the redo log becomes,
requiring longer time to consolidate the disks.
I'm not sure what those error log messages are about, but I cannot tie them to
the problem you are describing in this thread.
Best regards,
- Andy
____________________________________________________________________________
Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead | storman AT
us.ibm DOT com
IBM Tivoli Storage Manager links:
Product support:
http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/Tivoli_Storage_Manager
Online documentation:
http://www.ibm.com/support/knowledgecenter/SSGSG7/welcome
Product Wiki:
https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%20Storage%20Manager
"ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2015-03-09
10:08:31:
> From: "Schaub, Steve" <Steve_Schaub AT BCBST DOT COM>
> To: ADSM-L AT VM.MARIST DOT EDU
> Date: 2015-03-09 10:12
> Subject: Re: VE 7.1.1.1 backup "freezing" a VM & question about
"megablocks"
> Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
>
> Andy,
>
> These are the only odd messages I saw from these backups. Just to
> clarify, the backups didn't freeze, but the servers became
> unresponsive to the end user while the backup was running. As soon as
> the backup completed, the servers became responsive again.
> Looking at the Windows app event logs, there is a gap showing no
> activity for the duration of the backup. There were some entries in
> the system event log during the backup. It was like someone hit the
> "pause" button. Which I could understand if it looked like either VSS
> or VMWare Snapshot manager hung, but as far as I can tell they were
> successful.
>
> Thanks,
> -steve
>
> 03/05/2015 07:58:19 ANS9365E VMware vStorage API error for virtual
> machine 'XXXXXXXX'.
> TSM function name : VixDiskLib_Read
> TSM file : vmvddksdk.cpp (2766)
> API return code : 1
> API error message : NBD_ERR_DISKLIB
> 03/05/2015 07:58:19 ANS0361I DIAG: VmProcessExtent(): Retrying failed
> read: vddksdkRead() rc=4398, startSector=31955968,
numSectorsToRead=512
> 03/05/2015 09:14:51 ANS9365E VMware vStorage API error for virtual
> machine 'ZZZZZZZZ'.
> TSM function name : VixDiskLib_Read
> TSM file : vmvddksdk.cpp (2766)
> API return code : 1
> API error message : NBD_ERR_DISKLIB
> 03/05/2015 09:14:51 ANS0361I DIAG: VmProcessExtent(): Retrying failed
> read: vddksdkRead() rc=4398, startSector=24463360,
numSectorsToRead=512
> 03/05/2015 09:16:02 ANS9365E VMware vStorage API error for virtual
> machine 'XXXXXXXX'.
> TSM function name : VixDiskLib_Read
> TSM file : vmvddksdk.cpp (2766)
> API return code : 1
> API error message : NBD_ERR_DISKLIB
> 03/05/2015 09:16:02 ANS0361I DIAG: VmProcessExtent(): Retrying failed
> read: vddksdkRead() rc=4398, startSector=76280832,
numSectorsToRead=512
> 03/05/2015 09:19:05 ANS9365E VMware vStorage API error for virtual
> machine 'XXXXXXXX'.
> TSM function name : VixDiskLib_Read
> TSM file : vmvddksdk.cpp (2766)
> API return code : 1
> API error message : NBD_ERR_DISKLIB
> 03/05/2015 09:19:05 ANS0361I DIAG: VmProcessExtent(): Retrying failed
> read: vddksdkRead() rc=4398, startSector=82475520,
numSectorsToRead=512
> 03/05/2015 10:06:56 ANS9365E VMware vStorage API error for virtual
> machine 'XXXXXXXX'.
> TSM function name : VixDiskLib_Read
> TSM file : vmvddksdk.cpp (2766)
> API return code : 1
> API error message : NBD_ERR_DISKLIB
>
> -----Original Message-----
> From: ADSM: Dist Stor Manager [mailto:ADSM-L AT VM.MARIST DOT EDU] On Behalf
> Of Andrew Raibeck
> Sent: Monday, March 09, 2015 9:24 AM
> To: ADSM-L AT VM.MARIST DOT EDU
> Subject: Re: [ADSM-L] VE 7.1.1.1 backup "freezing" a VM & question
> about "megablocks"
>
> Hi Steve,
>
> Off-hand I am not sure what this is, but a couple of things:
>
> 1. Are there any anomalous messages in the error log during the
> timeframe of the backup that exhibits the problem?
>
> 2. Consider capturing a dump (*) of the TSM client backup process,
> e.g., dsmcsc.exe, when the backup is in the "frozen" state, open a PMR
> with TSM support, and send in the dump, the dsmerror.log, and the
> dsmsched.log.
>
> (*) In case you are not familiar with capturing a dump: start task
> manager, find the TSM client process that appears frozen, right- click
> on the process, and select "Create Dump File".
>
> Best regards,
>
> - Andy
>
>
____________________________________________________________________________
>
> Andrew Raibeck | Tivoli Storage Manager Level 3 Technical Lead |
> storman AT us.ibm DOT com
>
> IBM Tivoli Storage Manager links:
> Product support:
> http://www.ibm.com/support/entry/portal/Overview/Software/Tivoli/
> Tivoli_Storage_Manager
>
> Online documentation:
> http://www.ibm.com/support/knowledgecenter/SSGSG7/welcome
> Product Wiki:
> https://www.ibm.com/developerworks/community/wikis/home/wiki/Tivoli%
> 20Storage%20Manager
>
> "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU> wrote on 2015-03-06
> 11:28:19:
>
> > From: "Schaub, Steve" <Steve_Schaub AT BCBST DOT COM>
> > To: ADSM-L AT VM.MARIST DOT EDU
> > Date: 2015-03-06 11:29
> > Subject: VE 7.1.1.1 backup "freezing" a VM & question about
"megablocks"
> > Sent by: "ADSM: Dist Stor Manager" <ADSM-L AT VM.MARIST DOT EDU>
> >
> > First, many thanks to Wanda & others who have been so helpful in
> > answering my previous VE questions!
> >
> > We had a situation yesterday where 2 VE backups were causing the
> > VM's to go unresponsive. No response to ping, unable to RDP, etc.
> > As soon as the backup finished (or was killed in one case), the
> > servers picked back up where they left off. They never rebooted,
> > but you can actually see in the Windows event logs a gap where no
> > activity happens. Has anyone seen this behavior before? VE is at
> > 7.1.1.1, the Hosts are ESXi 5.0 U2, vCenter is 5.5, windows 2008R2.
> >
> > Secondly, while reading the docs, I ran across the idea of
> > performing periodic full backups in VE due to fragmentation of
> > "megablocks"? Is this needed? If so, how do you manage it (how
> > frequently, do you try to scatter the fulls across every day, how do
> > these interact with daily incrementals, etc)? If it matters, all
> > our backups land on a VTL.
> >
> > Thanks,
> >
> > Steve Schaub
> > Systems Engineer II, Backup/Recovery Blue Cross Blue Shield of
> > Tennessee
> > 423-535-6574 (desk)
> > 423-785-7347 (cell)
> >
> > -----------------------------------------------------
> > Please see the following link for the BlueCross BlueShield of
> > Tennessee E-mail disclaimer:
> > http://www.bcbst.com/email_disclaimer.shtm
> >
> -----------------------------------------------------
> Please see the following link for the BlueCross BlueShield of
> Tennessee E-mail disclaimer:
> http://www.bcbst.com/email_disclaimer.shtm
>
-----------------------------------------------------
Please see the following link for the BlueCross BlueShield of Tennessee E-mail
disclaimer: http://www.bcbst.com/email_disclaimer.shtm
|
