Archive for the ‘Philosophy’ Category

Practice What You Preach

October 29, 2011 Leave a comment

There’s a saying in the medical profession that goes something like, “Doctors make the worst patients”. It due to them thinking they know what’s wrong with them or them thinking that nothing is wrong with them. It really should say, “Medical professionals make the worst patients”. Case in point: my mother. She’s a retired nurse that is DOWN to a pack of cigarettes per day. She has this cough that is so bad I swear that she’s going to hack up a lung on of these days. She says she’s fine and refuses to seek treatment.

So how does this relate to IT? Well, back in the 90’s I worked for a IT consultancy firm. You wouldn’t believe how bad the internal systems were. You would think that with all the fancy certifications and brain power that my local branch had, we would have a working network and such. Not so. It was really a simple choice: fix our own infrastructure or be out in the field and generate revenue. Revenue won.


The same can sometime happen in one’s own house. How? Let me regale you with a tale of woe.

Sometime around VMworld (can’t remember if before or after), I noticed my house lights flickering. My UPS/surge protector started making some funny noises for a few moments and then went back to normal. Things were good, so I thought.

A few hours later I noticed that the lower level of my house was quite warm even though the A/C was running. I turned off the A/C and called the repair company. The next morning when the automatic schedule kicked in, the A/C ran fine. The repairman thought that some of my attic insulation had clogged the A/C unit’s drip pan/pipe and that the water level in the drip pan rose to the level where it triggered the auto shutoff. Simple enough. I have a split system: The compressors is outside, but the air handler is in the attic. What I thought was a functioning A/C system was really just the air handler circulating air.

Over the next week or two I experienced my first blue-screen in two years. Then my UPS would randomly start beeping. Nothing like a 1am BEEP! BEEP! BEEP! to scare the crap out of you. Other oddities would pop up every now and then until finally, I went to wake my computer from sleep mode and it wouldn’t wake. I did the turn off/on trick and no video, no beeps, no nothing. After a lot of manual reading, troubleshooting, and the occassional sacrifice to the gods, I finally determined it was the CPU that died.

I’ve never had a CPU die. I’ve had them arrive DOA, but I’ve never had one just go bad on me. Thankfully, my Intel CPU carried a 3yr warranty. I played the 20 question game with Intel and got it replaced. Guess what? System still wouldn’t come up. So I took it to a local computer shop and asked them to run diagnostics on everything. They got my system up and running, but in the process they reset the BIOS back to factory defaults. That really sucked.

I run an ASUS motherboard that has built-on RAID. Resetting the BIOS set the drive controller back to standard IDE mode. Since this entire process of troubleshooting, a short vacation, and replacing parts took over 30 days, new Windows patches had been released. I run with “automatic updates” turned on so it had downloaded a few patches and installed them. Upon reboot, I got the dreaded “No boot device detected” message. Seems the combination of losing the RAID setting and patching screwed up the boot loader. “No problem”, says I, “I have my Win7 DVD so I’ll just boot to it and do a repair”.

DUMB! DUMB! DUMB!. Windows warned me that the repair process could take over an hour so I walked away and let it ran. I checked it the next morning and it said it was done. I rebooted to find that I no longer had anything installed on my hard drive except Windows. Everything was gone…iTunes: gone. Other Apps: gone. All my data: gone.


Sigh, again.


OK, I lost everything. Thankfully, I really didn’t have a lot that I couldn’t replace or rebuild (virtual machines). Largest loss was photographs. I can recover about 10% them from various web sites that I’ve shared them on. The rest are lost. My iTunes library consists of about 3000 CDs. I own them all on physical CDs so I can re-rip them. The other major loss was years of personal emails.

To prevent this from happening again, I went out and bought another drive and a copy of Ghost. I also turned on the backup feature of my Synology DS211. Yes, I ‘ve had a backup system at hand for over six months and never used it. I bought the DS211 for iSCSI and NFS storage capabilities for my home lab. Now I back up to my DS211 every night and Ghost once a week to the new drive.

As an IT Pro, I should have known better. How many times have we expressed to our employeers, clients, and whomever else will listen, the importance of backups? If we make claims to our customers regarding best practices, shouldn’t we follow them ourselves? Are we “doctors” when it comes with diagnosing our own IT issues?


By the way, I had another A/C failure a week ago and a different technician was sent to fix it. He found that the electrical connection on my A/C compressor had melted somewhat. Hmm…flickering lights, A/C outage, UPS issues, CPU dieing..I’m betting that I took a massive hit and my UPS didn’t do it’s job of protecting my equipment. Or it did, but it took some damage and eventually passed it on. Maybe the beeping was a hint.

So I bought another UPS. Like the extra drive and Ghost, it’s cheap protection in the grand scheme of things.

I’m also still experiencing random wierdness. I’m going to hazard a guess and say that whatever took out my UPS and CPU also may have damaged either my RAM or motherboard. Looks like I may be making my way back to the part store in the next week or two for some replacements.



Latest Thoughts on Training

Let’s talk training again.  I recently had the opportunity to attend two classes.  One was self-paced using pre-recorded content; the other was an online instructor led class.  Before I get into what I think about them, I want to define a few items:

Tutorial:  good for single item topics and are generally short.  Tutorials should teach how to do a task, not provide comprehensive knowledge.

Instructor led, classroom environment:  This is the traditional training setup.  With this style of training, you drive to some facility to sit in a classroom with a bunch of people you may, or may not know, and tell war stories to each other all week.  .  All the while you hope the training facility has some good restaurants around it for lunch.  Instruction gets in the way of all the kibitzing, but you find that you actually learned a lot when the class is finished.

There are two variants of the Instructor Led, classroom environment type that are becoming very popular with the training providers.  They are:

a.      Instructor Led, classroom environment, equipment somewhere else and

b.     Instructor led, online environment, equipment somewhere else.

Self-paced:  This is where you download videos, watch slideware, and more often than not, find yourself bored almost to the point of falling asleep.  This type of instruction can be so boring that a class that normally takes 40 hours might take two months to complete.

Now you may not agree 100% with my definitions and that’s fine.  But for the sake of this post, just pretend to agree with me.


A little over a year ago, I wrote about a Cisco UCS class that I took.  It was of the Instructor Led, classroom environment, equipment somewhere else variety.  One of my chief complaints about it was the lab environment and all the problems we had accessing it.  Well, my recent experience with an Instructor led, online environment, equipment somewhere else type class shows me that the access issues are still prevalent.

Not a day went by where we didn’t have problems with either the presentation tool or accessing the lab servers.  I can understand why training providers like the equipment somewhere else concept.  It means fewer dollars spent buying equipment.  It also means that you can get better utilization of the equipment that is purchased.  However, this introduces a dependency on remote access systems, your network, and the availability of folks in some equipment room/data center to troubleshoot when problems arise.  In my case, the requirement was for a perfect network.  Any slight hiccup and you got kicked out of the presentation software.  If you were unlucky enough to have this occur twice in one day, you were SOL.  The only fix was for the instructor to kick everyone out and reset the class.

I will say this though: my training partner learned a lot about communication tricks during our labs.  Towards the end of the second day of class, I got kicked out of the presentation software (which also acted as a softphone) and could not communicate with my lab partner.  Rather than having the instructor reset everyone, I just used various little tricks to send him messages.  Tricks such as changing the Message of the Day in vCenter, opening notepad on our vCenter server to write him messages, and using the old “Net send” command from a command prompt.  It worked, but was not very efficient.

Even if there were no technical difficulties, I can definitely say that I am not a fan of the Instructor led, online environment, equipment somewhere else delivery method.  More specifically, I am not a fan of the online environment component. I thrive on all the interaction that takes place in the classroom.  I typically learn more from the other students than I do from the instructor and official content. (As people are apt to say, nothing beats real world experience.)  With an online class, it is very hard to interact with the other students.  I can’t really describe it, but it’s hard to carry on conversations.  There are no facial cues; it’s hard to get people’s attention, etc.

I also missed out on the troubleshooting opportunities.  In a classroom environment, when someone has a problem, everyone will huddle around their screens and work together to solve the problem.  Not easy to do in an online class.

Unfortunately, I foresee even more training occurring online.  Why?  $$$.  It’s cheaper to have an employee sit at home or his/her office space than it is to send them to a physical classroom.  This becomes more evident if the class is held out of state.  I only hope that training providers get more resilient software and other infrastructure.  Otherwise we’ll have ended up going backwards.


As for my second class, it was a self-paced network training program.  All I can say about this class was that it was extremely boring.  It was so boring that it took me almost two months to finish a 40hr class. The class was basically slideware that read itself to me.   One peculiar oddity to note: If I accidentally clicked “Next” before the audio completed, it would pick up physically where it left off from the previous slide on the new slide.  Huh?  Yes, if the audio was in the middle of a slide and I clicked “Next”, it would then proceed to read to me from the middle of the new slide.  It acted as if it was a screen scraper of some sort.  You won’t believe how happy I was when I found I could turn off the audio.

At that point, I turned into a speed reader and went at a more comfortable pace.  I think I finished about 30hrs of instruction (according to student guide) in about 10 hours.  It’s amazing how much that audio slows you down without adding value.  I think I learned more with the audio off than I did with it turned on.

Remember what I said up above about interacting with fellow students?  Well, forget about it with the self-paced model.   What really killed it for me was the inability to ask questions and get answers in a timely manner.  Yes, there was an “ask a question” link, but I had to wait up to 24hrs for a response.  What should I have done while waiting for a response?  Continue? Wait?  Talk about a momentum killer.

I also have to add that I thought the content was fairly light.  It seemed to have a fair amount of business driver/marketing type slides as opposed to technical information.  It was also fond of rehashing them.  While there is value in having this info, I would have preferred more technical related content.


I know times are tough and everyone is looking for ways to save money.  But maybe we need to rethink how training is provided.  If the goal is to prepare the employee, then maybe saving a few bucks isn’t so cost effective.  While I personally feel that most training is overpriced (come on, $500+ per day for many classes), I don’t think saving a grand is worth it in the long run.  I wonder how much more effective I would be at my job if I got more out of the training classes?  Would it be worth that extra grand in two months?  How about three months?   Could the payback be even a month?  Think about it.



Categories: Philosophy Tags: ,

My Thoughts on Our Cisco UCS Sales Experience

August 31, 2010 2 comments

This is a topic that when I think about it, I jump around in my head from subtopic to subtopic.  To make things easier on myself, I am going to write a bunch of disjointed paragraphs and tie them together in the end.

Disjoint #1

I’ve never worked on Cisco gear in the past.  Everywhere I worked where I had access to network/server equipment, Cisco was not a technology provider.  I don’t know why, other than I’ve heard Cisco had the priciest gear on the market.  I’ve also heard/read that while Cisco is #1 in the networking gear market, their products are not necessarily #1 in performance, capacity, etc.  Throw in the perception of the 800lb gorilla and you get a lot of negative commentary out there.

Disjoint #2

When I was 19, I started my career in the technology field as a bench tech for a local consumer electronics store.  The owner (Ralph) was a man wise beyond his years. He saw something in me and decided to take me under his wing, but because I was 19, I did not understand/appreciate the opportunity that he was bestowing upon me.

While I learned some of the various technical aspects of running a small business, I did not do so well on the human side of it.  I was a brash, cocky 19yr old who thought he could take over the world.  However, there is one thing Ralph said that I remember very well and that is, “If no one has any problems, how will they ever find out what wonderful customer service we have”.

It’s not that he wanted people to have problems with the equipment they purchased.  He knew that by selling thousands of answering machines, telephones, T.Vs, computer, etc there would be some issues at some point and he felt that he  should do his best to make amends for it.

Ralph truly believed in customer service and would go out of his way to ensure that all customers left feeling like they had been taken care of extremely well.  If there was poster child for exemplary customer service, it would be Ralph.

Disjoint #3

A number of vendors with broad product lines have somehow decided that the SMB market does not need robust, highly available (maybe even fault tolerant) equipment.  Somehow, company size and revenue have become equated with technical needs.  Perceptions of affordability have also played into this, meaning, if you can’t afford it, then you don’t need it.

Why do I bring this up?  Way back in one of my earlier posts, I mentioned that we had a major piece of equipment fail and received poor customer service from the vendor.  The vendor sales rep kept saying that we bought the wrong equipment.  We didn’t buy the wrong equipment, we bought what we could afford.   In hindsight it wasn’t the equipment that failed us, but the company behind it.


Tieing all this together…

When we first started looking at UCS, some folks here had trepidations about doing business with Cisco.  There were preconceived notions of pricing and support.  Cisco was also perceived to have a reputation of abandoning a market where they could not be number one in sales.

I must also admit that there are technical zealots in my organization that only believe in technical specifications.  These folks try to avoid products that don’t “read” the best on paper or have the best results in every performance test.

However, my team diligently worked to overcome these objections one by one and we couldn’t have done it without the exceptional Cisco sales team assigned to us.

In the early part of the sales process, we pretty much only dealt with the Product Sales Specialist (PSS) and her System Engineer (SE).  The rest of the account team entered the picture a month or so later.

These two (PSS and SE) had the patience of Job.   The sales team took copious amounts of time meeting with us to explain how UCS was different from the other blade systems out there and how it could fit into our environment and enable us to achieve our strategic goals.  All questions were answered thoroughly in a timely manner.  Not once did I ever get the feeling that they (Cisco) felt they were wasting their time.

When the infamous HP-sponsored Tolly report (and other competing vendor FUD) came out, Cisco sales took the time to allay our concerns.   As we read and talked about other competing products, not once did they engage in any negative marketing.  Cisco took the high road and stuck to it.

We had phone calls with multiple reference accounts.  We had phone calls with product managers.  We had phone calls with the Unified Computing business unit leaders.   We had phone calls with…you get the idea.  Cisco put in a great amount of effort show us their commitment to be in the server business.

On top of all this, there was no overt pressure to close the sale.  Yes, the sales team asked if they could have the sale.  That’s what they are supposed to do.  But they didn’t act like car salesman by offering a limited duration, once in a lifetime deal.   Instead, they offered a competitive price with no strings attached. (Disjoint #1)

Needless to say, we bought into UCS and have transitioned to the post sales team.  This means we now interact more with our overall account rep and a generic SE rather than the PSS and her SE.  I call our new SE generic because he is not tied to a particular product but represents the entire Cisco product line.  He’s is quite knowledgeable and very helpful in teaching the ways of navigating Cisco sales and support.

So has everything gone perfectly?  No. We’ve had a few defective parts.  If you have read of my other posts, you know that we have had some integration issues.  We’ve also found a few areas of the management system that could use a bit more polish.  So in light of all this, do I regret going with UCS?  Not at all.  I still think it is the best blade system out there and I truly think the UCS architecture is the right way to go.

But with defective parts, integrations issues, etc…”Why do I still like Cisco?” you ask.  For starters, I don’t expect everything to be perfect.  That’s just life in the IT field.

Second, go re-read Disjoint #2.   Cisco must have hired Ralph at some point in time because their support has been phenomenal.    Not only do the pre and post sales teams check in to see how we are doing, any time we run into an issue they ask what Cisco can do to help.  It’s not that they just ask to see if they can help, they actually follow through if we say “yes”.  They are treating us as if we are their most important customer.

Finally, to tie in Disjoint #3, any time we run into something where other vendors would say we purchased the wrong equipment, Cisco owns the issue and asks how they can improve what we already have purchased.   It’s not about “buy this” or “buy that”.  It’s “How can we make it right?”, “What can we do to improve the product/process/experience?”, and “What could we have done differently?”   These are all questions a quality organization asks themselves and their customers.

I don’t know what else I can write about my Cisco sales experience other than to say that it has become my gold standard.  If other vendors read this post, they now know what standard they have to live up to.

To other UCS customers: What was your sale experience like?


Do you defrag your virtual servers?

April 20, 2010 Leave a comment

There have been some recent comments on Scott Drumond’s site (and others)  regarding defragging of virtual servers.  What do you think?  Do you defrag your virtual servers?  I’m personally torn.  I can see the value, but I am not sure if the costs are justified,  and I am not just talking about the monetary costs either.

We try to have a “what we do for one, we do for a thousand” mentality when it comes to our standard server builds.  Simply, this means that if we are going to declare a piece of software as a part of our standard base image all servers get it.  So far, it means that all servers get anti-virus protection, log monitoring, and a few others as a default.  In terms of dollars, it adds up to quite a pretty penny.  Virtualization makes it more expensive because I  have more server instances in my environment than I would have if every server was physical due to physical server costs.  Since a project doesn’t have to pay for hardware, it’s easier to ask for a server.  Some people call this sprawl, but I wouldn’t.  Sprawl connotates lack of control and we have well-defined controls in place.  No servers get deployed without adequate licensing and other resources.

Another cost is resource utilization.  If a server is busy defragging, it’s using CPU and disk resources.  Does this impact other virtual servers?  I would say yes, but I can’t say how much.  Your mileage will vary.  Yes, I can quantify direct resource utilization, but if my customers don’t notice the difference, does it really matter.    A 5% increase in CPU may have no impact on customer experience.  Fine then.  But what if they do notice a difference?  What if transaction times go up?  All the sudden that $xxx license may have just tripled in cost due to lost productivity.

Don’t forget to throw in the costs of environmentals.  If the host is busy, it’s generating heat.  If the host is busy, it’s using more electricity than it would be at idle (definitely true on Nehalem CPUs with power mgmt active).

Long story short, it’s not so simple as saying “defragging will improve vm performance by x”.  You need to figure out all the other ramifications.    My personal belief is to defrag those systems that clearly need it.  You’ll know which ones because you will either already have seen a significant performance degradation in them, or if you actively monitor your systems, you’re watching them begin to degrade.


  • As an aside..most of the performance results being posted these days are based on running newer CPUs, newer storage, etc.  Older equipment will not fare the same.  Example, just because Virtual Center shows 50% CPU available doesn’t mean it’s really available.  An additional 5% load can be noticed by other virtual servers.  We’ve experienced it in my organization on 3yr old servers.  It’s not a problem on new equipment, but something we have to take into consideration when deploying guests onto older hosts.
Categories: Philosophy Tags:

Patches! We don’t need no stinkin’ patches!

April 13, 2010 Leave a comment

It’s time again for our monthly visitor.  No, not that one.  I’m talking about Microsoft patches.   We’re pretty aggressive here in applying them.   How aggressive?  We like to play Russian Roulette by patching all servers within 48hours of patch release.   Strangely enough, we’ve been really lucky.  I think we’ve had only five outages in the seven or so years of patching.  I would prefer more time to test, but that’s not my call.  However, if something breaks, my team has to pick up the pieces.

How about you?  When do you patch?

Categories: Philosophy Tags: