VMDK UUID Recovery

We had an incident a few weeks ago in which a technician restored a virtual server using vRanger Pro to a different name and data-store for the purpose of copying the restored server to our test lab.  No big deal, but one bad thing happened.  Unkbeknownst to the technician, the vmdk header files (the ones that point to the _flat.vmdk files) and .vmx file were still pointing to the production system.   So when he went to clean up and delete the restored virtual server, Virtual Center dutifully removed the .vmx and .vmdk header files from the REAL server.  It wasn’t able to delete the _flat.vmdk files (thankfully) because they were in use.  Also turns out we had a few disks marked Independent, Persistent so they weren’t being backed-up.

We recovered what we could but still had to manually recreate the vmdk header files.  Since I had no reference file, I took an existing one from a working virtual server, changed all the various pointers, and powered the guest up.  I got an error in regards to the vmdk UUID.  Whenever I have a disk problem, one of my “go to” tools is the VMware provided vmkfstools utility.  A quick man-page review and I found that it can create UUIDs.  Running “vmkfstools -J setuuid filename.vmdk” solved the problem.  The system now boots fine, but since I didn’t know all the details to put in the file, the drive shows up with incorrect information in Virtual Center and backups are still a problem.

A soon to be implemented solution is to create another drive with the correct parameters (via Virtual Center) and then copy everything from the bad drive into the good drive, do some drive letter changing, and Oila.  Another solution is a process change on our part.  No more deletions of this sort via Virtual Center.  Just going to remove the server from inventory and manually delete the files via the service console.

UPDATE: So what do I find in my VMware support RSS feed this morning?  An article detailing how to create a new vmdk header file to replacing a missing header file.   (Link VM KB) I like VMware’s solution better because it not only generates a new UUID (in the header file), it also adds the correct disk geometries.

Patching VMware

So here is my first post regarding an actual “day in the Life….”   Last night was VMware patching.  I wish I could say it was a pleasant experience, but I can’t.   The unpleasantness has nothing to do with VMware and all to do with my organization’s maintenance policies.

Our policy is to work on e-commerce systems on Sunday nights, from 10:00pm to 1:00am.   In my case, since the vm hosts in question host some e-commerce systems, I am stuck in the same maintenance window.  It doesn’t matter that I am running in a DRS/HA clustered configuration and that no downtime is experienced.  It’s the potential for downtime that causes worry in the ranks.  I can’t say that I disagree.  Nothing is 100%.  Problems do occur which sometimes result in an outage.

On-site at 9:45pm. Maintenance mode activated at 10:00pm, patches applied.  Repeat for other host.    I was home by 1am.    What makes it really unpleasant is that I am a terrible sleeper.  I’ll wake up at the slightest noise and/or light.  So home by 1am, in bed by 1:30am, up at 6:00am.  I am going to be dragging until Wednesday.  The older I get, the longer it takes to recover.