VMware data recovery troubleshooting

VMware data recovery troubleshooting

If the VDP backup fails , the following troubleshooting steps can be used

1. SSH to the the VDP appliance and browse to the /usr/local/avamarclient
2. Search for logs related to the VM :   grep -r -a "VM_NAME" ./*
3. If you suspect it is snapshot related issue : grep -r -a " VM_name" ./* | grep "FATAL"
4. To be more specific and to check messages for a certain date, try searching using the date : grep -r -a " VM_name" ./* | grep "2013-08-02"
5. Sometimes we could get very useful information from the "info" messages as well. Inorder to narrow down to the same, you can use the command: grep -r -a "VM_name" ./var-* | grep "2013-07-03"
6. The baove command will search only through the 'var-proxy' directories. It will display the entire log file. You can less it to view details for a specific date eg: less ./var-proxy-5/VMGROUP1-1378306800496-35fj52c29f48eeejef090b27edaeba3d868719e8-4016-vmimagew.log
   /2013-07-03 07:10:00

Error messages:

Message 1:

avvcbimage FATAL <16018>: The datastore information from VMX '[STORAGE-1] VMNAME_1/VMNAME.vmx' will not permit a restore or backup. 

Reason: The most common reason is that a snapshot file is present but it is not getting displayed in the snapshot manager.Inorder to resolve this, 

1. SSH to the esx hosting the VM 
2. Browse to the VM's datastore :cd /vmfs/volumes/datastore_name/VM_name/
3. Check if there are any delta files in it ie files with -delat in name or -00001 etc
4. Now check if any of these files are in use by checking the vmx file : grep "vmdk" ./*.vmx
5. If the files are not being referenced in the vmx, we can safely delete or move the delta files to a temp directory: mkdir 0ld-delta-files ; mv vm_name.000*.vmdk old-delta-files/
6. Confirm that the files have been deleted

Message 2:

avvcbimage FATAL <14688>: The VMX '[STORAGE-1] VMNAME_1/VMNAME.vmx could not be snapshot.

Reason:One possible reason is that you execute a backup and it overruns the scheduled backup in VDR

Message 3:
2013-07-03 17:00:57 avvcbimage Info <14642>: Deleting the snapshot 'VDP-137830742335fc52c29f98eeebef090b22edaeba3p868716e8', moref 'snapshot-17946'
2013-07-03 17:00:57 avvcbimage Info <0000>: Snapshot (snapshot-17946) removal for VMX '[STORAGE-1] VMNAME_1/VMNAME.vmx  task still in progress, sleep for 2
sec
2013-07-03 17:00:57 avvcbimage Info <0000>: Snapshot (snapshot-17946) removal for VMX '[STORAGE-1] VMNAME_1/VMNAME.vmx task was canceled.

2013-09-04 17:00:57 avvcbimage Info <0000>: Removal of snapshot 'VDP-VDP-137830742335fc52c29f98eeebef090b22edaeba3p868716e8' is not complete, moref 'snapshot-17946'

Reason:This is because VDP doesnt get enough time to delete the snapshots created during the backup operation.Solution is to change the timeout value to allow enough time for snapshots to commit.

To increase this timeout value:
1.Open an SSH session to the VDP server.
2.Change to the /usr/local/avamarclient/var directory using this command:
# cd /usr/local/avamarclient/var
3.Open the avvcbimage.cmd file using a text editor. For more information, see Editing files on an ESX host using vi or nano (1020302).
4.Add this entry to the file:
--subprocesstimeout=600
5.Restart the avagent service using this command:
# service avagent restart

Reference: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2044821

Common WinDBG Commands Reference

Command	Description from WinDBG Help (go there for detailed help!)
!	! Extension Commands
!address	displays information about the memory that the target process or target computer uses.
!analyze -hang	(hang) Generates !analyze hung-application output.
!analyze -v	displays information about the current exception or bug check.
!devstack	displays a formatted view of the device stack associated with a device object.
!drvobj	displays detailed information about a DRIVER_OBJECT.
!exqueue	(hang) displays a list of items currently queued in the ExWorkerQueue work queues.
!exqueue 2	(hang) display a list of threads and events associated with the work queue and their wait states.
!handle	displays information about a handle or handles that one or all processes in the target system own.
!irpfind	(hang) displays information about all I/O request packets (IRP) currently allocated in the target system, or about those IRPs matching the specified search criteria.
!irql	displays the interrupt request level (IRQL) of a processor on the target computer before the debugger break.
!locks	(hang) information about kernel ERESOURCE locks.
!memusage	displays summary statistics about physical memory use.
!pcr	(hang) displays the current status of the Processor Control Region (PCR) on a specific processor.
!podev	displays the power capabilities of the target computer.
!poolused	(hang) displays memory use summaries, based on the tag used for each pool allocation. Use !xpoolused if this command does not work.
!process	(hang) displays information about the specified process, or about all processes, including the EPROCESS block.
!pte	displays the page table entry (PTE) and page directory entry (PDE) for the specified address.
!ready	displays summary information about each thread in the system in a READY state.
!session	displays one or more user sessions, or displays a specified process running in multiple user sessions.
!stacks	displays information about the kernel stacks.
!stacks 2	(hang) displays the full parameters for all stacks, including those currently paged out and the current kernel stacks
!teb	displays a formatted view of the information in the thread environment block (TEB).
!thread	displays summary information about a thread on the target system, including the ETHREAD block.
!verifier –f	displays the status of Driver Verifier and its actions.
!vm	(hang) displays summary information about virtual memory use statistics on the target system.
!vm 20	(hang) display to include kernel virtual address usage.
!xpoolmap	(hang) displays a map of pool use
.	. Commands
.cxr	displays the context record saved at the specified address. It also sets the register context.
.imgscan	scans virtual memory for image headers
.kFrames	sets the default length of a stack trace display, i.e. 0n256 = 256 length
.reload	deletes all symbol information for the specified module and reloads these symbols as needed
.sympath	changes the default path of the host debugger for symbol search.
.trap	displays the trap frame register state and also sets the register context.
a	Standard Commands
dh	displays the headers for the specified image.
dps	The dds (double word), dps (pointer-sized), and dqs (quad-word) commands display the contents of memory in the given range.
dt	displays information about a local variable, global variable or data type
kvf, kvn	display the stack frame of the given thread, together with related information
lm	displays the specified loaded modules.
ln	displays the symbols at or near the given address
r	displays or modifies registers, floating-point registers, flags, pseudo-registers, and fixed-name aliases.
ub	displays an assembly translation of the specified program code in memory.
uf	displays an assembly translation of the specified function in memory.
uf /c	Displays only the call instructions in a routine instead of the full disassembly.
up	displays an assembly translation of the specified program code in physical memory.
vertarget	displays the current version of the Microsoft Windows operating system of the target computer.
x	displays the symbols in all contexts that match the specified pattern.

Troubleshoot Memory/Disk lantency/High i/o issue

VM memory ballooning (Amount of memory being used for ballooning in a VM at the request of VM tools rises above 10% configured memory).

Excessive memory ballooning on a VM is driven by tools, and can indicate memory contention on the host. Check the other VMs on the same host for possible memory contention. Additional memory or balancing of workloads across hosts and clusters may be required. Consider the active links below for further drill down.

VM memory swap.(Amount of memory being swapped for a VM by the hypervisor rises above 10%)

Excessive memory swapping for a VM often indicates memory contention on the underyling host, and is a last resort by the hypervisor to reclaim memory. Check the other VMs on the same host for possible memory contention. Additional memory or balancing of workloads across hosts and clusters may be required. Consider the active links below for further drill down.

Host to Datastore Latency(Fires when the total latency between a host and a datastore is above 20ms)

One or more of your hosts has high storage latency going to a particular datastore - consider using the active links below to narrow this down to time spent in the host vs the SAN to pinpoint the cause of the issue

VM Disk Latency(Read or write latency to any of this VM's VMDKs has gone above 20ms)

One or more VMs have high storage latency - consider using the active links to investigate IO loading on this VM's datastore(s)

Datastore high I/O delay(Latency spent by storage I/O requests in the datastore)

The datastore total latency is averaging above 20ms. This could indicate a storage IO bottleneck on the datastore. Examine the IO of VMs using this datastore for possible rebalancing of IO loads. Consider the active links below for further drill down.

VM Phantom Snapshot Files(Finds VMSN files on the datastore that aren't listed in a Virtual Machine's snapshots)

it is possible that when some of your snapshots were removed, all associated snapshot file were not cleaned up correctly. There may be opportunities to reclaim storage by removing these files

How to activate Windows server when we have KMS server installed

How to activate Windows server when we have KMS server installed

few of our business partner report that when they logon to the servers they see windows is not activated..

open the command prompt and type 
slmgr /ato with this servers gets activated.

if it is not activated try slmgr /skms "enter KMS servername":1688

this should work this time.

Convert snapshot file into memory dump

How to Convert snapshot file into memory dump?

Vmss2core is a tool to convert VMware checkpoint state files into formats that third party debugger tools understand. It can handle both suspend (.vmss) and snapshot (.vmsn) checkpoint state files .The vmss2core tool can produce core dump files for the Windows debugger (WinDbg).

1. When the Guest OS is Hung or unresponsive, Login to the vCenter Server or ESXi host using vSphere Client or vSphere Web Client

2. Take the Snapshot of the Virtual Machine which is hung or unresponsive state. Which will create .vmsn file in the virtual machine directory

Alternative to the snapshot, You can place the virtual Machine in Suspend State  by Right-Click the hung Virtual Machine -> Power -> Suspend. Which will create .vmss file in the virtual machine directory.

3. Select the .vmsn or .vmss file and download to  your local machine where you are going to  save the Vmss2core utility.

4. Download the Vmss2core  from VMware Fling Site and place the downloaded file (vmss2core_win.exe)  in your local drive (C:\ or D:\). Ensure  your Windows machine is installed with Visual Studio 2008 Service Pack 1 runtime. Place both the Vmss2core file and downloaded .vmsn or .vmss file in same location

5. Execute the below commands to generate core dump. The vmss2core tool can produce core dump files for the Windows debugger (WinDbg).

c:\>vmss2core_win.exe -W VM-1-Snapshot3.vmsn

It generates a file called *.dmp extension. You can share the crash dump files to your operating system vendor to identify the route cause of Hung or Crash.

Below are the list of command line switches available for vmsscore utility.

Options	Explanation
(none)	Without any options, produces linear views of memory (vmss.core<n>) one per virtual CPU.
-W	Creates aWinDbgfile (memory.dmp)of aWindowsvirtualmachinewithcommonlyusedbuildnumbers, 2195 for Win32 and 6000 for Win64.
-W<num>	Creates a WinDbg file (memory.dmp) with <num> as the build number,for example: -W2600
-WDDB<num>	Creates aWinDbg file (memory.dmp) with <num> as thedebugger data block address in hexadecimal,for example: -W12ac34de
-WSCAN	Creates aWinDbg file (memory.dmp) and scan all of memory forthe debugger data block, instead ofjust the lower 256 MB
-M	Creates a core file (vmss.core) with a physical memory view suitable forthe Gnu debugger gdb
-l <str>	Specifies the starting andending offsets ofLinux kerneldata structures for use by the -N and-P options, with <str> expressed as 0xHEXNUM,0xHEXNUM. Ignored when used with other options.
-N	Red Hat crash core file (vmss.core) for an arbitrary Linux version as defined by the -l option.
-N4	Red Hat crash core file (vmss.core) for Linux kernel version 2.4.
-N6	Red Hat crash core file (vmss.core) for Linux kernel version 2.6.
-P	Prints a list of processes running in the Linux virtual machine at checkpoint time
-P<pid>	Creates a core file (core.<pid>) forthe Linux process number <pid>. Itis likely thatprograms compiledwithsymboltables (notremoved)willyieldbetterdebug information
-X<nn-v>	Mac OS core dump with <nn-v> representing architecture and Darwin kernel version
-q	Quiet operation.

How to check list VCSA services status?

How to check list VCSA services status?

Connect to VCSA through putty and perform the following

shell.set --enabled True

shell

cd /bin

Run this command to list the vCenter Server Appliance services:

service-control --list

To check the status of a service runn the following command

service-control -status

To Start and Stop a service run the following commands

service-control -stop servicename
service-control -start servicename

To restart all the services

service-control -stop -all
service-control -start -all

Useful Esxi commands

When snapshot consolidation is getting failed.

To move the CTK files and consolidate snapshots:

Connect to the ESXi host that the virtual machine is running on using SSH. For more information, see Using ESXi Shell in ESXi 5.x and 6.0 (2004746).

Navigate to the virtual machine folder using this command:

cd /vmfs/volumes/datastore/virtual_machine/

List the contents of the directory using the ls command and look for .ctk files.

Create a temporary directory for the CTK files.

For example:

mkdir temp

Move the CTK files to this directory with this command:

mv *-ctk.vmdk temp/

Run the snapshot consolidation again. 

To restart management agents:

#/etc/init.d/hostd restart

#/etc/init.d/vpxa restart

or

To restart all agents at once

#Services.sh

Note: this cmd restart all agents so could take a long time to complete that is why its preferable to restart individual agent

To Power off / Reboot an unresponsive VM: 

Using vim-cmd,

#vim-cmd vmsvc/getallvms  This command will give you the VM Name and respective VM ID.

Now, to see the power state of VM,

#vim-cmd vmsvc/power.getstate VMID

To shutdown the VM,

#vim-cmd vmsvc/shutdown VMID

If the above doesn't work then, use fowwing command to power off the VM,

#vim-cmd vmsvc/poweroff VMID

Using esxcli:

To get VM name and respective wordNumber

#esxcli vm process list 

To power off a VM : There are three power-off methods available with esxcli. Soft is the most graceful, hard performs an immediate shutdown, and force should be used as a last resort.

#esxcli vm process kill -t [ soft,hard,force] -w WorldNumber

Reload a vmx file without removing the virtual machine from inventory: First get to VMID as described above, then 

#vim-cmd vmsvc/reload VMID

To get a list of running tasks on the host,

#vim-cmd vmsvc/task_list 

To get a list of tasks associated with a specific VM,

#vim-cmd vmsvc/get.tasklist VMID

To get information about the status of a particular task, run the command

#vim-cmd vimsvc/task_info task_identifier 

We can also use ps command to find VM related running processes.

#ps |grep VM_name    this command will give you process id and if you want to kill the process, then

#kill process_id   wait for some time, if process is still there then use

#kill -9 process_id

To ping Any host/VM etc: 

#vmkping ip_or_hostname

To enter maintenance mode using the command line interface

vimsh -n -e /hostsvc/maintenance_mode_enter      or   vim-cmd /hostsvc/maintenance_mode_enter or  esxcli system maintenanceMode set --enable true

To check if host is in maintenance mode

#vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode     or

vimsh -n -e /hostsvc/hostsummary | grep inMaintenanceMode or

esxcli system maintenanceMode get

To exit maintenance mode

#vimsh -n -e /hostsvc/maintenance_mode_exit    or  

vim-cmd /hostsvc/maintenance_mode_exit or

esxcli system maintenanceMode set --enable false

To install an update or any third party vib file stored in any datastore:

#esxcli software vib install -d /vmfs/volumes/..../downloaded.vib or downloaded_vib_bundle.zip

To list the software and drivers currently installed on the ESXi host: 

#esxcli software vib list 

To check the status of physical NIC connectivity, run this command:

# esxcfg-nics –l

To check installed Esxi version :

#vmware -vl

To lunreset :

vmkfstools -L lunreset /vmfs/devices/disks/naa.XXXXXXXXX