OK, so I wasn’t really on vacation. But it sure felt like it at times. By some quirk of fate, I was able to attend both Cisco Live and VMworld this summer. And I had a blast at both of them.
I was at Cisco Live solely to work in VCE’s booth. For four days, I spent open to close talking to folks as they wandered the expo near the VCE booth. I got to meet existing customers, potential customers, co-workers, past co-workers from other places where I’ve worked, and more.
Since I work on the Product Management team, I tried to get people to tell me their stories. I wanted to know what their daily IT life was like, how was their infrastructure working for them, what were their plans for the near & long-terms. I heard some doozies in regards to plans, but I am not sure they are appropriate for a technology blog ;)
Anyway….I heard a lot of recurring themes: need to do more with less, need better management tools, need to learn about cloud, need to learn how to operate (or operate better) in a virtual world. Excuse me? The last one threw me a bit, but after a little more digging I found that some folks thought virtualization would solve their operational issues.
Folks, you’ve read this before on numerous other blogs but I am going to repeat it: if you have bad operational practices in the physical world and you don’t change them when you enter the virtual world, then you still have bad operational practices. Fix your bad practices before virtualizing. It will save you a lot of heartache and finger pointing. /soapbox off/
What I heard a lot of was, “Please help me”. There is just so much change going on in our industry now that it can be quite daunting to know what to do and where to go. Do I go cloud? Do I not go cloud? What is cloud? Can I have my own infrastructure? Can I just get my feet wet? All good questions and all that have the same answer: It depends. It’s usually at this point I would bring in one of our vArchitects to help me. I can answer most of the questions, but when someone asks me how many switches will I need, or how much capacity needs to be reserved for sparing, it’s best to leave it to the more knowledgeable folks.
My highlight of Cisco Live was when a customer came to the VCE booth with a friend and then proceeded to try to sell his friend a Vblock. It got so far as whiteboarding, drawing designs, and then some. A few vArchitects were listening in to clarify statements when needed, but pretty much just left them alone. The customer was doing an amazing job and was so enthusiastic about his Vblock he just had to get his friend to buy one (or at least into the concept).
It’s one thing for an employee to sell and be enthusiastic about products, it something else when a customer does it.
VMworld was a different story. I got to go as a mighty ATTENDEE (cue angels singing). I spent most of my time either in sessions or on the expo floor checking out all the other products. There is a lot of interesting work going on out there. I was surprised a few companies were still around from last year given that VMware entered their niche with some of the new features in vSphere 5.0. But after talking to them, the surprise went away. Some of these niche products do one thing, but they do it very well compared to VMware’s implementation and that keeps the customers coming to them.
As for sessions, I focused on vCloud Director and storage. I hit about 10 sessions covering the two topics. A lot for me to learn there. I was decently versed in the storage side of vSphere, but wanted a primer on the new storage features of vSphere 5.1. When it came to vCloud Director, I was fairly ignorant. I’m still ignorant on this topic, just less so. It’s definitely an area I want to learn more about. Time to cozy up with a book or two….
While at VMworld, I decided to run an experiment and wear my official VCE logoed shirt during the sessions. I wanted to see if people would stop me to ask questions. You now what? They did. In almost all the sessions I attended, at least one person came up to me with questions about VCE and Vblocks. There was one session where I had four people (non-related) stop me to answer questions.
So what did I come away with? 2 Kindle Fires, an Apple TV, and the VMworld plague. Been sick almost a week now. Awful stuff.
What else did I come away with? Some knowledge of vCD, some new friends, and a change in perspective on how VCE and Vblocks are viewed. Good times indeed.
Disclosure: I work for VCE so make of it what you will.
I had lunch this weekend with a friend of mine that manages the server infrastructure for a local government entity. During the usual banter regarding business, he mentioned that a recent storage project (1+yr and still going) had suffered a series of setbacks due to outages, product compatibility issues, and other minor odds & ends.
The entity he worked at released an RFP that detailed quite a bit of what they were looking to accomplish, which in retrospect may have to been too ambitious given the budget available. After going through all the proposals, the list was narrowed down to two. On paper, both were excellent. All vendors were either Tier1 or Tier2 (defined by sales). Both proposals were heavily vetted with on-site visits to existing customers, reference phone calls, etc. Both proposals consisted of a storage back-end with copious amounts of software for replication, snapshotting, provisioning, heterogeneous storage virtualization, and more.
While each component vendor of the winning proposal was highly respected, the two together did not have a large installed base that mimicked the proposed configuration (combo of hardware & software). In hindsight, this should have been a big RED flag. What looked good on paper did not pan out in reality.
Procurement went smoothly and the equipment arrived on time. Installation went smoothly. It was during integration with the existing environment that things started to break down. The heterogeneous virtualization layer wasn’t quite as transparent as listed on paper. Turns out all the servers needed to have the storage unpresented and presented back. (Hmmm…first outage, albeit not a crash.)
Then servers starting having BSODs. A review by the vendors determined that new HBAs were needed. This was quite the surprise to my friend since the he provided all the tech info on the server & storage environment as part of the proposal and was told that all existing equipment was compatible with the proposal. (2nd, 3rd, 4th+ outages…crashes this time).
So HBAs were updated, drivers installed, and hopefully goodness would ensue. (planned outages galore).
Unfortunately, goodness would not last. Performance issues, random outages, and more. This is where it started to get nasty and the finger pointing began. My friend runs servers from Cisco (UCS) and HP. The storage software vendor started pointing at the server vendors. Then problems were attributed to both VMware and Microsoft. (more unplanned outages).
Then the two component vendors started pointing fingers at each other. Talk about partnership breakdown.
So what is my friend’s shop doing? They are buying some EMC and NetApp storage for their Tier 1 apps. Tier 2 and Tier 3 apps will remain on the problematic storage. They are also scaling back their ambitious goals since they can’t afford all the bells & whistles from either vendor. The reason they didn’t purchase EMC or NetApp in the first place was due to fiscal constraints. Those constraints still exist.
As I listened to the details, I realized that he really needed one of the Vblock ™ Infrastructure Platforms from VCE.
First, his shop already has the constituent products that make up the majority of a Vblock – Cisco UCS, EMC Storage (small amount), and vSphere. This makes transitioning to a Vblock ™ easier since less training is needed and a comfort level already exists for those products.
Second, the hardware and software components in the winning proposal were not widely deployed in the proposed configuration. At the time my friend’s shop put out the RFP, there were more Vblocks in production use in the United States than there were of the winning proposal’s configuration world-wide.
Third, at VCE, all the hardware/software components in a Vblock ™ are thoroughly tested together. It’s VCE’s job to find any problems so a customer doesn’t. People think that if you take three items listed on an HCL somewhere and put them together, then everything will work fine. It just isn’t true. In fact, a number of patches put out by the parent companies are the result of testing by VCE’s engineering and quality assurance teams.
And finally, probably the biggest reason why my friend needs a product from VCE is to get rid of all the finger pointing. When VCE sells a product, every component is supported by VCE. There is no saying “It’s not our problem, call the server vendor”, or “call the storage vendor”, or “call the software vendor”. I’m not saying that all vendors finger point. Also, your mileage with a set of particular vendors will vary, but if you are CIO/Manager/whatever, you have to admit that “one call, that’s all” is quite compelling. You can either have your staff spend time managing vendors or you can have your staff spend time moving your business forward.
I’ve put the bug in the ear of my friend about going Vblock in the future. It won’t happen any time soon since his procurement cycle isn’t conducive to purchasing all his infrastructure components in one fiscal year. It usually takes five years to get through all three major components. But who knows? Maybe his recent experiences will accelerate the cycle.
A few days ago, both Cisco Press and VMware Press announced a few opportunities to win a few goodies via Facebook. Details below:
VMware Press Launches Sweepstakes!
VMware Press, the official publisher of VMware books and training materials, has launched a 60 day Facebook sweepstakes beginning May 1 and running through June 30th. Prize offerings include a $100 Amazon gift card and three VMware Press books of the winner’s choice; nine second prize winners will win an eBook of their choice.
Cisco Press Offers Free Trip to Cisco Live!
The official publisher of Cisco launched the annual Cisco Press Facebook sweepstakes today, offering free to Cisco Live 2013 including travel ($1,000 American Express gift card) and registration and a choice of three Cisco Press print or eBooks! Nine second prize winners will also win three print or eBooks of their choice for a total of 10 winners in all. The Cisco Press Sweepstakes begin May 1 and run through June 30th http://ow.ly/aBv08.
So for my first book review of 2012, I am going to start with Administering VMware Site Recovery Manager 5.0 by Mike Laverick. Many of you already know Mike (the guy is everywhere) or are at least familiar with his blog (RTFM-ed).
Let me start out by saying that I did not like this book after first reading it. I felt that it was missing something; something that I couldn’t quite put my finger on. Then it hit me. Concepts are not necessarily discussed as concepts. Yes, there are one/two page discussions on concepts, but most often they are discussed as working knowledge. This should not have been a surprise to me because Mike clearly states (multiple times) that he expects his readers to have read the various manuals for detailed concept and background info on vSphere, Site Recovery Manager (SRM), your storage array, etc. He can’t teach everything needed to get SRM working so you have do some work on your own. In other words, RTFM. Once I came to grips with this, I re-evaluated the book in a new light and have decided that I like it.
As for the book itself, it has an interesting layout. You get a little bit of history concerning vSphere’s DR and HA features and what SRM is, and is not. Then comes a little detour into setting up a number of different storage arrays from Dell, EMC, HP, and NetApp. This detour does serve a purpose in that it sets a baseline storage configuration for installing and configuring SRM, albeit in the most simple configuration possible. It’s actually a smart move on his part because he is able to show how he setup his lab. It also prompts the reader to go check various items in order to ensure a successful install of SRM.
Then you get to the good stuff: installing, configuring, and using SRM. There are plenty of screenshots and step-by-step instructions for doing a lot of the configuration tasks. In fact, you could think of this book along the lines of a cookbook. Follow along and you should end up with a usable (in a lab) install of SRM.
It’s clear that after reading this book, Mike knows SRM. Peppered throughout the chapters are the various problems and errors he encountered as well as what he did to fix them. In a few cases, he does a mea culpa for not following his own advice of RTFM. If he had, a few problems would have been avoided.
Mike also hits home on a few simple truths. For those involved with Active Directory in the early days, there was a truth that went something like this: “The question is irrelevant, the answer is DNS”. In the case of SRM, substitute “Network and storage configuration” for “DNS”. So many problems that may be encountered are the result of a network or storage configuration issue. vSwitches need to be setup correctly, hosts need to see storage, vCenter needs to see hosts, etc.
I especially liked the bits of wisdom that he shares in regards to doing what I call “rookie maneuvers” (others call them stupid mistakes). For example, once you have SRM up and running, it’s too easy to hit the button without realizing what it all really entails. Mike warns you about this many times and prompts you to think about your actions ahead of time.
The later chapters of the book introduce customizations, scripting, more complex configurations, and how to reverse a failover. There is a lot going on here and worth re-reading a few times. A surprising amount of this info can be applied to basic disaster recovery principals regardless of whether or not SRM is in the picture.
Lastly, Mike walks you through upgrading from vSphere 4.1 to vSphere 5 and from SRM 4.1 to SRM 5. Upgrading vSphere may sound a bit odd, but not when you take into account that it’s required in order to upgrade SRM.
All-in-all, this book is a worthy read and should be in your library if your shop uses (or plans to use) SRM.
Early January marked my one year anniversary with VCE. I was hired to be a Program Manager on the virtualization team and my first project to lead was bringing vSphere 5 to the world of Vblocks. I didn’t think this would be as difficult as it turned out to be. I knew I would be herding cats, but I didn’t plan on herding cats from outside the herd. About midway through the project, both Cisco and EMC informed us that they weren’t going to certify vSphere 5 on older levels of firmware. In the case of Cisco, this meant the we were going to move all the Vblock platforms to UCS 2.0. For EMC, it meant upgrading the firmware for all our supported storage arrays. In essence, I was actually leading a project to upgrade all the components in a Vblock platform.
If I do say so myself, I did a great job. But it wasn’t just me. I worked with a great team of engineers, tech writers, product managers, trainers, and more. This was truly a cross-functional project and involved over 50 staff across three companies by the time the project completed.
In this same one year, VCE had gone through tremendous change. When I hired on, VCE was basically a startup. About midway through the year, a certain level of operational maturity was needed. We had achieved significant growth, both in sales and in head count. Thus began a series of reorgs. There was basically one large reorg and a series of refining reorgs. In my case, I went through two refinements.
The first reorg moved the virtualization team in with the rest of the Product Management team. It also moved the bulk of our engineering staff into one engineering group. This was a smart move as it removed barriers to introducing new product. Unfortunately for me, all Program Managers were moved into a formal Program Management Office. While I did a great job on the vSphere 5 project, I found that this wasn’t the position for me. Luckily, my managers recognized my talents and kept me on the virtualization team, which was now part of the Product Management group.
As the vSphere 5 project was winding down, the virtualization team was disbanded and we moved into the direct chain of Product Management. Again, not a bad idea but it did leave me in a bit of limbo since Product Management does not have a need for a Program Manager. Again, I got lucky. The director of Product Management recognized my abilities in the areas of process management, barrier breaking, and general mayhem. A product management operations team was created and I was assigned to it. Our charter is simple: keep things moving. Think of us a “fixers”. If a project is in trouble, we show up and get it back on track. If someone is not getting things done in a timely manner, we will. We are also developing various policies, processes, and procedures for the Product Management team as well as working with other teams inside of VCE to develop company-wide policies and processes.
It’s been interesting to me because I am being exposed to areas of the business that I have not had previous exposure to. For example, I am working with the marketing group on website redesign and developing launch materials. I am also working with our supply chain managers on setting appropriate stocking levels.
I’ve had an exciting first year. I’m betting the second is going to be even better.
There’s a saying in the medical profession that goes something like, “Doctors make the worst patients”. It due to them thinking they know what’s wrong with them or them thinking that nothing is wrong with them. It really should say, “Medical professionals make the worst patients”. Case in point: my mother. She’s a retired nurse that is DOWN to a pack of cigarettes per day. She has this cough that is so bad I swear that she’s going to hack up a lung on of these days. She says she’s fine and refuses to seek treatment.
So how does this relate to IT? Well, back in the 90′s I worked for a IT consultancy firm. You wouldn’t believe how bad the internal systems were. You would think that with all the fancy certifications and brain power that my local branch had, we would have a working network and such. Not so. It was really a simple choice: fix our own infrastructure or be out in the field and generate revenue. Revenue won.
The same can sometime happen in one’s own house. How? Let me regale you with a tale of woe.
Sometime around VMworld (can’t remember if before or after), I noticed my house lights flickering. My UPS/surge protector started making some funny noises for a few moments and then went back to normal. Things were good, so I thought.
A few hours later I noticed that the lower level of my house was quite warm even though the A/C was running. I turned off the A/C and called the repair company. The next morning when the automatic schedule kicked in, the A/C ran fine. The repairman thought that some of my attic insulation had clogged the A/C unit’s drip pan/pipe and that the water level in the drip pan rose to the level where it triggered the auto shutoff. Simple enough. I have a split system: The compressors is outside, but the air handler is in the attic. What I thought was a functioning A/C system was really just the air handler circulating air.
Over the next week or two I experienced my first blue-screen in two years. Then my UPS would randomly start beeping. Nothing like a 1am BEEP! BEEP! BEEP! to scare the crap out of you. Other oddities would pop up every now and then until finally, I went to wake my computer from sleep mode and it wouldn’t wake. I did the turn off/on trick and no video, no beeps, no nothing. After a lot of manual reading, troubleshooting, and the occassional sacrifice to the gods, I finally determined it was the CPU that died.
I’ve never had a CPU die. I’ve had them arrive DOA, but I’ve never had one just go bad on me. Thankfully, my Intel CPU carried a 3yr warranty. I played the 20 question game with Intel and got it replaced. Guess what? System still wouldn’t come up. So I took it to a local computer shop and asked them to run diagnostics on everything. They got my system up and running, but in the process they reset the BIOS back to factory defaults. That really sucked.
I run an ASUS motherboard that has built-on RAID. Resetting the BIOS set the drive controller back to standard IDE mode. Since this entire process of troubleshooting, a short vacation, and replacing parts took over 30 days, new Windows patches had been released. I run with “automatic updates” turned on so it had downloaded a few patches and installed them. Upon reboot, I got the dreaded “No boot device detected” message. Seems the combination of losing the RAID setting and patching screwed up the boot loader. “No problem”, says I, “I have my Win7 DVD so I’ll just boot to it and do a repair”.
DUMB! DUMB! DUMB!. Windows warned me that the repair process could take over an hour so I walked away and let it ran. I checked it the next morning and it said it was done. I rebooted to find that I no longer had anything installed on my hard drive except Windows. Everything was gone…iTunes: gone. Other Apps: gone. All my data: gone.
OK, I lost everything. Thankfully, I really didn’t have a lot that I couldn’t replace or rebuild (virtual machines). Largest loss was photographs. I can recover about 10% them from various web sites that I’ve shared them on. The rest are lost. My iTunes library consists of about 3000 CDs. I own them all on physical CDs so I can re-rip them. The other major loss was years of personal emails.
To prevent this from happening again, I went out and bought another drive and a copy of Ghost. I also turned on the backup feature of my Synology DS211. Yes, I ‘ve had a backup system at hand for over six months and never used it. I bought the DS211 for iSCSI and NFS storage capabilities for my home lab. Now I back up to my DS211 every night and Ghost once a week to the new drive.
As an IT Pro, I should have known better. How many times have we expressed to our employeers, clients, and whomever else will listen, the importance of backups? If we make claims to our customers regarding best practices, shouldn’t we follow them ourselves? Are we “doctors” when it comes with diagnosing our own IT issues?
By the way, I had another A/C failure a week ago and a different technician was sent to fix it. He found that the electrical connection on my A/C compressor had melted somewhat. Hmm…flickering lights, A/C outage, UPS issues, CPU dieing..I’m betting that I took a massive hit and my UPS didn’t do it’s job of protecting my equipment. Or it did, but it took some damage and eventually passed it on. Maybe the beeping was a hint.
So I bought another UPS. Like the extra drive and Ghost, it’s cheap protection in the grand scheme of things.
I’m also still experiencing random wierdness. I’m going to hazard a guess and say that whatever took out my UPS and CPU also may have damaged either my RAM or motherboard. Looks like I may be making my way back to the part store in the next week or two for some replacements.
People go to VMworld for many reasons. Some go because it’s their job to ”man the booth”. Others go to party. And still others go “just because”. However, the most common reason why people go to VMworld is to learn about VMware products and its ecosystem. If I were still in the position of IT Architect, that would have been my primary reason too. This year is different. I changed jobs at the beginning of 2011 and went from an IT position that held responsibility for the care and feeding of the virtual infrastructure platform to a Product Management position. As such, my VMworld focus has changed from learning about VMware products to learning about VMware’s customers.
One of the basic tenets of Product Management/Development is to build products that customers want/need to buy. So how does one go about finding out what customers want and/or need? Simple. Ask them. I’ll be roaming the Solutions Exchange talking to attendees about their jobs, roadmaps, challenges, and desires (within the context of the datacenter). I want to gather as much information as I can to help me excel in my new”ish” position. I want to collect contact info so that I can reach out to folks later and see how things change as time passes. I want to know if your efforts are successful or not. Basically, I want to “know” and “learn” about you.
So if you happen to see me, introduce yourself. Tell me about your company, your datacenter challenges, and more. Help me develop a better product.
If you can’t find me, send a me a tweet - @ITVirtuality – and let’s schedule a time to meet.