Practice What You Preach
There’s a saying in the medical profession that goes something like, “Doctors make the worst patients”. It due to them thinking they know what’s wrong with them or them thinking that nothing is wrong with them. It really should say, “Medical professionals make the worst patients”. Case in point: my mother. She’s a retired nurse that is DOWN to a pack of cigarettes per day. She has this cough that is so bad I swear that she’s going to hack up a lung on of these days. She says she’s fine and refuses to seek treatment.
So how does this relate to IT? Well, back in the 90′s I worked for a IT consultancy firm. You wouldn’t believe how bad the internal systems were. You would think that with all the fancy certifications and brain power that my local branch had, we would have a working network and such. Not so. It was really a simple choice: fix our own infrastructure or be out in the field and generate revenue. Revenue won.
The same can sometime happen in one’s own house. How? Let me regale you with a tale of woe.
Sometime around VMworld (can’t remember if before or after), I noticed my house lights flickering. My UPS/surge protector started making some funny noises for a few moments and then went back to normal. Things were good, so I thought.
A few hours later I noticed that the lower level of my house was quite warm even though the A/C was running. I turned off the A/C and called the repair company. The next morning when the automatic schedule kicked in, the A/C ran fine. The repairman thought that some of my attic insulation had clogged the A/C unit’s drip pan/pipe and that the water level in the drip pan rose to the level where it triggered the auto shutoff. Simple enough. I have a split system: The compressors is outside, but the air handler is in the attic. What I thought was a functioning A/C system was really just the air handler circulating air.
Over the next week or two I experienced my first blue-screen in two years. Then my UPS would randomly start beeping. Nothing like a 1am BEEP! BEEP! BEEP! to scare the crap out of you. Other oddities would pop up every now and then until finally, I went to wake my computer from sleep mode and it wouldn’t wake. I did the turn off/on trick and no video, no beeps, no nothing. After a lot of manual reading, troubleshooting, and the occassional sacrifice to the gods, I finally determined it was the CPU that died.
I’ve never had a CPU die. I’ve had them arrive DOA, but I’ve never had one just go bad on me. Thankfully, my Intel CPU carried a 3yr warranty. I played the 20 question game with Intel and got it replaced. Guess what? System still wouldn’t come up. So I took it to a local computer shop and asked them to run diagnostics on everything. They got my system up and running, but in the process they reset the BIOS back to factory defaults. That really sucked.
I run an ASUS motherboard that has built-on RAID. Resetting the BIOS set the drive controller back to standard IDE mode. Since this entire process of troubleshooting, a short vacation, and replacing parts took over 30 days, new Windows patches had been released. I run with “automatic updates” turned on so it had downloaded a few patches and installed them. Upon reboot, I got the dreaded “No boot device detected” message. Seems the combination of losing the RAID setting and patching screwed up the boot loader. “No problem”, says I, “I have my Win7 DVD so I’ll just boot to it and do a repair”.
DUMB! DUMB! DUMB!. Windows warned me that the repair process could take over an hour so I walked away and let it ran. I checked it the next morning and it said it was done. I rebooted to find that I no longer had anything installed on my hard drive except Windows. Everything was gone…iTunes: gone. Other Apps: gone. All my data: gone.
Sigh.
Sigh, again.
OK, I lost everything. Thankfully, I really didn’t have a lot that I couldn’t replace or rebuild (virtual machines). Largest loss was photographs. I can recover about 10% them from various web sites that I’ve shared them on. The rest are lost. My iTunes library consists of about 3000 CDs. I own them all on physical CDs so I can re-rip them. The other major loss was years of personal emails.
To prevent this from happening again, I went out and bought another drive and a copy of Ghost. I also turned on the backup feature of my Synology DS211. Yes, I ‘ve had a backup system at hand for over six months and never used it. I bought the DS211 for iSCSI and NFS storage capabilities for my home lab. Now I back up to my DS211 every night and Ghost once a week to the new drive.
As an IT Pro, I should have known better. How many times have we expressed to our employeers, clients, and whomever else will listen, the importance of backups? If we make claims to our customers regarding best practices, shouldn’t we follow them ourselves? Are we “doctors” when it comes with diagnosing our own IT issues?
By the way, I had another A/C failure a week ago and a different technician was sent to fix it. He found that the electrical connection on my A/C compressor had melted somewhat. Hmm…flickering lights, A/C outage, UPS issues, CPU dieing..I’m betting that I took a massive hit and my UPS didn’t do it’s job of protecting my equipment. Or it did, but it took some damage and eventually passed it on. Maybe the beeping was a hint.
So I bought another UPS. Like the extra drive and Ghost, it’s cheap protection in the grand scheme of things.
I’m also still experiencing random wierdness. I’m going to hazard a guess and say that whatever took out my UPS and CPU also may have damaged either my RAM or motherboard. Looks like I may be making my way back to the part store in the next week or two for some replacements.
Sigh.
Why I Am Going to VMworld
People go to VMworld for many reasons. Some go because it’s their job to ”man the booth”. Others go to party. And still others go “just because”. However, the most common reason why people go to VMworld is to learn about VMware products and its ecosystem. If I were still in the position of IT Architect, that would have been my primary reason too. This year is different. I changed jobs at the beginning of 2011 and went from an IT position that held responsibility for the care and feeding of the virtual infrastructure platform to a Product Management position. As such, my VMworld focus has changed from learning about VMware products to learning about VMware’s customers.
One of the basic tenets of Product Management/Development is to build products that customers want/need to buy. So how does one go about finding out what customers want and/or need? Simple. Ask them. I’ll be roaming the Solutions Exchange talking to attendees about their jobs, roadmaps, challenges, and desires (within the context of the datacenter). I want to gather as much information as I can to help me excel in my new”ish” position. I want to collect contact info so that I can reach out to folks later and see how things change as time passes. I want to know if your efforts are successful or not. Basically, I want to “know” and “learn” about you.
So if you happen to see me, introduce yourself. Tell me about your company, your datacenter challenges, and more. Help me develop a better product.
If you can’t find me, send a me a tweet - @ITVirtuality – and let’s schedule a time to meet.
.
What about Tintri?
I attended the Phoenix VMUG meeting this week. The two main sessions were about vSphere5 and Tintri’s VMstore. While vSphere5 is interesting, I have been working with it for over 5 months now so it wasn’t a “must see” presentation for me. I was actually at the event to see Tintri and I have to say that the Tintri VMstore product intrigues me quite a bit. For those who haven’t heard of this product, think of it as a purpose built storage appliance for your VMware environment. This “appliance” is roughly 8.5TB (usable) and is only accessed via NFS. The entire device presents itself as one large datastore to your hosts. If you think about it, this really does simplify things quite a bit. There is no zoning, no LUN creation, no disk grouping, etc. Basically, all of your standard storage creation tasks have been removed. Time to add capacity? Just add another appliance and add it to your vCenter as another datastore. It’s that simple.
Management of the appliance is performed through a web interface and via a vCenter Plug-in. The bulk of what you expect in a management tool is there with a few notable exceptions (discussed later in this post).
One of the VMstore design goals is performance. To that end, Tintri has equipped the VMstore with 1TB of SSD storage. Through their own internally developed magic, the bulk of “hot” data is kept in SSD. The rest is stored on SATA disks. You can imagine the kind of IOPS possible given the heavy use of SSD. BTW, the SSD is deduped so you get more bang for your buck.
The folks at Tintri gave the standard “who we are” and “why we are different” presentation that we all expect at open events like this. After talking about the product and walking us through the mgmt. interface the Tintri folks took questions from the audience. All-in-all, a good showing.
There were no hard questions asked at the VMUG, but the after meeting was completely different. I am also a member of Advanced Technology Networking Group (ATNG) and we met up with the Tintri folks a few hours later. ATNG consist of hardcore techies and since many of our members are responsible for acquisitions and managing data centers, our meeting with vendors tend to be “No holds barred”, but in a friendly way. Our goal is to get to know the product (warts and all) as much as we can during our meetings.
We questioned a lot of design choices and where the product is going. One are of particular interest to me was the use of SATA drives. Yes, the appliance uses RAID6 and has hot spares, but that did not alleviate my concern. Drive quality continues to improve so only time will tell if this was a good design choice or not.
Another area questioned was the use of a single controller. The general rule of enterprise storage is to have two controllers. VMstore currently has one. Notice that I say “currently”. Future product will have two controllers.
There were a few questions and suggestions regarding the management interface. One suggestion was to rename the VMStore snapshot function. It is not the same snapshot feature as in vCenter. vCenter has no visibility into VMstore native snapshots and vice-versa. This can be a source of confusion if you consider that the target audience for this product is VM admin.
The lack of some enterprise features also came up in our discussions. Notably, the lack of SNMP support and the lack of replication support. The only way to get notified of something going wrong with the appliance is to either receive an email alert or see something in vCenter. As for replication, the only option available is to perform a standard vm backup and restore the data to another appliance or storage device of your choice.
However, all is not doom and gloom. Tintri is working on updates and improvements. SNMP support, replication capabilities, and more are coming soon. Keep in mind that Tintri recently came out of stealth mode and is on 1.0 of their product. For a 1.0 product, it’s pretty good. Just to give an idea of the performance and quality of VMstore, Tintri has a reference customer that will attest that they have been running a beta version since November 2010 without any issues. In fact, that customer is still on the beta code and has not upgraded. That’s a pretty good reference if you ask me.
So what do I think of VMStore? I think Tintri is on the right track. Purpose built storage for VMware is a great concept. It shows a laser like focus on a particular market and it lets the company focus on capabilities and features that are specific to that market. Generic storage has to cater to many masters and sometimes gets lost in the process.
I am going to predict that Tintri will either be copied by other storage vendors or that they will be acquired by one of them. The product/concept is just too unique and spot-on that it can’t be ignored.
Links of interest:
Latest Thoughts on Training
Let’s talk training again. I recently had the opportunity to attend two classes. One was self-paced using pre-recorded content; the other was an online instructor led class. Before I get into what I think about them, I want to define a few items:
Tutorial: good for single item topics and are generally short. Tutorials should teach how to do a task, not provide comprehensive knowledge.
Instructor led, classroom environment: This is the traditional training setup. With this style of training, you drive to some facility to sit in a classroom with a bunch of people you may, or may not know, and tell war stories to each other all week. . All the while you hope the training facility has some good restaurants around it for lunch. Instruction gets in the way of all the kibitzing, but you find that you actually learned a lot when the class is finished.
There are two variants of the Instructor Led, classroom environment type that are becoming very popular with the training providers. They are:
a. Instructor Led, classroom environment, equipment somewhere else and
b. Instructor led, online environment, equipment somewhere else.
Self-paced: This is where you download videos, watch slideware, and more often than not, find yourself bored almost to the point of falling asleep. This type of instruction can be so boring that a class that normally takes 40 hours might take two months to complete.
Now you may not agree 100% with my definitions and that’s fine. But for the sake of this post, just pretend to agree with me.
.
A little over a year ago, I wrote about a Cisco UCS class that I took. It was of the Instructor Led, classroom environment, equipment somewhere else variety. One of my chief complaints about it was the lab environment and all the problems we had accessing it. Well, my recent experience with an Instructor led, online environment, equipment somewhere else type class shows me that the access issues are still prevalent.
Not a day went by where we didn’t have problems with either the presentation tool or accessing the lab servers. I can understand why training providers like the equipment somewhere else concept. It means fewer dollars spent buying equipment. It also means that you can get better utilization of the equipment that is purchased. However, this introduces a dependency on remote access systems, your network, and the availability of folks in some equipment room/data center to troubleshoot when problems arise. In my case, the requirement was for a perfect network. Any slight hiccup and you got kicked out of the presentation software. If you were unlucky enough to have this occur twice in one day, you were SOL. The only fix was for the instructor to kick everyone out and reset the class.
I will say this though: my training partner learned a lot about communication tricks during our labs. Towards the end of the second day of class, I got kicked out of the presentation software (which also acted as a softphone) and could not communicate with my lab partner. Rather than having the instructor reset everyone, I just used various little tricks to send him messages. Tricks such as changing the Message of the Day in vCenter, opening notepad on our vCenter server to write him messages, and using the old “Net send” command from a command prompt. It worked, but was not very efficient.
Even if there were no technical difficulties, I can definitely say that I am not a fan of the Instructor led, online environment, equipment somewhere else delivery method. More specifically, I am not a fan of the online environment component. I thrive on all the interaction that takes place in the classroom. I typically learn more from the other students than I do from the instructor and official content. (As people are apt to say, nothing beats real world experience.) With an online class, it is very hard to interact with the other students. I can’t really describe it, but it’s hard to carry on conversations. There are no facial cues; it’s hard to get people’s attention, etc.
I also missed out on the troubleshooting opportunities. In a classroom environment, when someone has a problem, everyone will huddle around their screens and work together to solve the problem. Not easy to do in an online class.
Unfortunately, I foresee even more training occurring online. Why? $$$. It’s cheaper to have an employee sit at home or his/her office space than it is to send them to a physical classroom. This becomes more evident if the class is held out of state. I only hope that training providers get more resilient software and other infrastructure. Otherwise we’ll have ended up going backwards.
.
As for my second class, it was a self-paced network training program. All I can say about this class was that it was extremely boring. It was so boring that it took me almost two months to finish a 40hr class. The class was basically slideware that read itself to me. One peculiar oddity to note: If I accidentally clicked “Next” before the audio completed, it would pick up physically where it left off from the previous slide on the new slide. Huh? Yes, if the audio was in the middle of a slide and I clicked “Next”, it would then proceed to read to me from the middle of the new slide. It acted as if it was a screen scraper of some sort. You won’t believe how happy I was when I found I could turn off the audio.
At that point, I turned into a speed reader and went at a more comfortable pace. I think I finished about 30hrs of instruction (according to student guide) in about 10 hours. It’s amazing how much that audio slows you down without adding value. I think I learned more with the audio off than I did with it turned on.
Remember what I said up above about interacting with fellow students? Well, forget about it with the self-paced model. What really killed it for me was the inability to ask questions and get answers in a timely manner. Yes, there was an “ask a question” link, but I had to wait up to 24hrs for a response. What should I have done while waiting for a response? Continue? Wait? Talk about a momentum killer.
I also have to add that I thought the content was fairly light. It seemed to have a fair amount of business driver/marketing type slides as opposed to technical information. It was also fond of rehashing them. While there is value in having this info, I would have preferred more technical related content.
.
I know times are tough and everyone is looking for ways to save money. But maybe we need to rethink how training is provided. If the goal is to prepare the employee, then maybe saving a few bucks isn’t so cost effective. While I personally feel that most training is overpriced (come on, $500+ per day for many classes), I don’t think saving a grand is worth it in the long run. I wonder how much more effective I would be at my job if I got more out of the training classes? Would it be worth that extra grand in two months? How about three months? Could the payback be even a month? Think about it.
.
Book Review: Enterprise Network Testing
For my next trick, I am going to review another Cisco Press book titled “Enterprise Network Testing”. I think I can sum this up in two sentences: “Holy Crap!” and “This book has PLENTY of cowbell!”.
Now I am not currently a network guy by profession, but if I was, this book would be on my desk with copies on my teammates’ desks too. It is literally THE blueprint for how to test your network.
The journey into network testing begins with a discussion on why you need to test your network. Most people only think of one or two reasons. This book provides a few more to help you make your business case. BTW, the authors make it very clear that testing your network is not a one-time event. Testing should be done whenever changes are made, for compliance, introduction of new technologies, etc. In other words, plan on testing regularly.
One area where this book and I completely agree is where testing should first take place: in the lab. There is whole chapter devoted to lab strategy. Topics covered include staffing, facilities planning, test methodologies, power, and more. I must say that I was surprised at how good this chapter turned out to be. Most books give basic guidance on lab setup, but like I said at the beginning of this review, this book has plenty of cowbell.
So now you have your lab setup, what are you going to do? Simple, read this book because it provides guidance for “crafting the test approach” (actual chapter title). Briefly, this chapter discusses several reasons/objectives for testing and how to craft your strategies to set you up for success. This includes setting your test targets, what tools are you going to use, writing a test plan, allocating resources, etc. It’s a very well thought out approach.
Business case approved? Check. Lab resources allocated? Check. Test plan created? Check. Great, now go execute your plan. Need help? No problem, this book will walk you through a sample lab setup, finding the appropriate tools, and a few different methodologies for measuring different network characteristics. This is the point in the book where the authors stress the need to understand what you are testing, the tools you are using, and how to interpret the results. In other words, if you don’t know what you are doing you will not be successful.
Speaking of knowing your tools, this book does a credible job discussing network toolsets that are available for free and for purchase. Even non-Cisco products are covered which is something I am not used to seeing in a Cisco Press book. Usually, these books are oblivious to other companies’ products. Kudos to the authors for being thorough.
The next six chapters are where you will find plenty of test case examples. There are individual chapters devoted to six types of testing. They are: Proof of concept testing, network readiness testing, design verification testing, migration plan testing, new platform and code certification testing, and network ready for use testing. They are written in a case study format and are quite readable.
Nerdgasm time. This is where the book gets hairy…Are you too lazy to develop your own plans from scratch? You want to cheat? Just borrow the DETAILED test plans that are in the next seven chapters. There is enough meat here that Cisco Press could copy & paste into a shorter book to sell. We are talking over 200 pages of test plans covering seven areas. That’s a lot of cowbell!
The book ends on a high note. Since you went through the trouble of setting up a lab, why not use it for training/learning purposes. Step-by-step instructions are provided to setup a lab. This chapter may not be useful to a large number of folks since the equipment covered is pure Cisco, including UCS. In fact, many of the directions provided center around setting up a UCS environment. I happen to like this chapter because one of my last major implementations before joining VCE was installing UCS for the organization at which I worked. Sort of brings back memories.
To sum this review up: If you are in the network field, you need this book.
