Test Management

This category contains 9 posts

Hard Lessons Learned as a Senior Manager – The Monsanto Challenge

The following is a series of personal thoughts and reflections on eight or so years of learning my way around what it means to be a senior quality assurance  or engineering manager. As some of you know I have recently found myself once again looking at the future of my professional career and wondering what it holds. The man who made the decision to change the entire organization of my department this week gave me some great advice. In our long and unofficial exit interview he challenged me to write down my thoughts on things I can personally take away from my successes and failures over a decade of different management roles.

Since I over share (according to everyone who loves me) this is the medium I’ve chosen to use. No names will be changed since I have no desire to name names or otherwise talk about specific instances, people, or companies.

Entry the First: The Monsanto Challenge

In which our hero finds himself experiencing blowback for not recognizing his full customer chain

A while back I worked for a company that was purchased by Monsanto. Monsanto meant business when they purchased our admittedly small little outfit in downtown San Francisco. They wanted everyone to feel like they were special and to know they purchased us because they had no idea how to do what we did and thus had no interest in telling us how to do what we do or interfere in any other manner. They purchased us because they wanted us and our data and our data models all to themselves. They spared no expense instilling the impression that both we were still going to be us but that we were also now in the “big boy pants” leagues. They chartered a jet and flew all of us out to three different facilities throughout the middle parts of the United States. We were given access to top research scientists, laboratories, executives, and even had an intimate dinner with the C-level executives and several board members. I got to meet men and women in one day who did everything from build Maker Faire sort of wild machines designed to strip genetic information from seeds without destroying them to people who advised Presidents and Prime Ministers on issues of national importance. Everyone I met on this magical mystery tour was very nice. They were to a person cheerful and helpful but most of all every single person from the assistants to the front desk personnel to the geneticists were all passionately committed to the idea of solving the world’s current and looming hunger crises. These were not Jurassic Park scientists playing with expensive toys. These were dedicated botanists, geneticists, engineers, and explorers who sought out, cataloged, studied, and genomed every single variety of crops like corn and wheat in existence looking for answers to complicated problems. When they couldn’t find answers in the species they realized they had to look outside the species. That’s when GMOs came on the scene.

Since I had what was essentially a captive audience who claimed to be willing to talk about anything, I asked the President of Monsanto directly why he thought his company’s public image was so wildly divergent with what I personally experienced just chatting with everyone. He told me it was one thing they completely screwed up a long time ago and didn’t even try to correct until lately. They didn’t look into anyone except the farmer’s buying their seeds and herbicides. The stopped at the primary connection in their value chain.

You see as much as a lot of people hate Monsanto, farmers around the world love love love them. Monsanto has done an amazing job of working with farmers, listening to their problems, and rapidly (in terms of seed) changed course several times to address their pain points. In most ways their movement into GMOs was a direct request from farmer’s who’s crops were being destroyed by climate change, water contamination, rapidly evolving pests & diseases, and the regulation of most herbicides out of existence. For decades Monsanto thought that was enough. When the suddenly felt the pain of people reacting emotionally to the idea of consuming what to them felt like “Franken-food” Monsanto reacted like the scientists and engineers they are by simply telling people to look at the science and research. To which what seemed like an entire planet replied with the phrase “Nuts.” Monsanto got hammered and is still being hammered. They’ve become the symbol of everything people don’t like about our current society and a repository for all the fears we have for what seems to be looking more and more like a bleak future. They ended up trying to enter a very pernicious and embedded opinion based on a decade or more of rumor armed only with facts.

Looking back at his now I wish I’d learned it years ago and had remembered it recently.

When you’re a manager (and especially a senior manager) you are responsible for a lot of things. Your team can feel like the most important thing that need concern you. You have reports and meetings with your direct manager who may be a director, vice president, or even chief officer of some sort. You have customers you need to keep happy be they external to your organization or internal to it. You can fall into the trap of making sure you do the best job possible, your team is chugging along fantastically, and your customers think you’re amazing. You want to believe that is more than enough to secure your reputation. Every company says it’s a “no drama zone” or something like it. Technology companies in particular like to say they’re data driven and that the rely on things they can measure. The reality is while you’re intensely focused on your primary relationships you are neglecting your secondary and tertiary relationships. These relationships are vitally important for the long term health and opinion of you and your team.

There is a tool consultants use called an influencer map. They use it to rapidly assess everyone on a project who has some sort of influence over the success or failure of that project and how they perceive it. People with direct influence are given priority but importantly people with indirect or even outlier influence are also mapped. The different people are mapped around who they can influence rather than where they occur in an org chart. Your job as a consultant is to be aware of people with influence who can have an effect on your outcome. You then turn around the ones you can to be supporters or even interested in your work. If you can’t change their stance you try and isolate them so they can’t scuttle your work through their attitude. As a manager you need to be aware of everyone in your organization in the same manner. it’s not something you need to address all the time but you don’t do it at the risk of how people perceive your work.

You see perception is often completely at odds with facts. People don’t trust facts or they selectively pick the ones that will support their perception. If a manager or director or a vice president forms a negative or even questionable perception/opinion of you or your team they will start to notice things supporting that viewpoint. You can present them with lots of Key Process Indicators (KPIs) showing you are doing great but they’ll come away remembering the one thing that may not have been awesome. At that point you are fighting a losing battle since you are actively working against an opinion formed from nothing. If you have enough time you can mitigate this issue with personal intervention and salesmanship but it’s a holding action until the opinion changes. If something like a re-organization or change in management happens before you’re turned that around you will find yourself on the list to be changed.

The best way to avoid The Monsanto Challenge is to never find yourself in it. Your first month as a manager should be spent personally meeting every other manager in your organization and every upper level manager in the company. You need to meet with them regularly and let them form their own impression of you, an impression you can hopefully control. Take the time to talk about the great things your team is doing and showcase their success. Ask them for advice or go to them with intelligent questions they can answer. Controlling your image in this manner provides some inoculation against other influencers who may be spreading a message you don’t want spread. It won’t prevent you from having your budget cut or being laid off or having your project shut down. These things happen all the time even in large companies.

It will remove one of the ways things like lay offs can happen in spite of success.

It’s a lesson I’ve paid the price for not following now at least twice and perhaps three times. I did not get on a plane and go meet with people face to face (the value of which I will talk about in another blog post). I didn’t control the message about myself and my team. In all of these occasions there was a price to pay; sometimes it was a very dear price indeed.


In Defense of a Software Testing Manager

Hey there, reprobates and iconoclasts!  Sorry I’ve been away for a while but I’ve been busy with work and playing a very interesting video game, Elder Scrolls V: Skyrim.  The video game itself is interesting after a fashion and I logged more than a few hours inside it when it first came out on PC.  I went back to it because the PC version has this amazing symbiotic relationship with their customers.  Bethesda Games released a creation engine (basically a rich set of level design tools) for free.  These tools were released for several very important reasons.  Allowing gamers access to design their own experiences and (more importantly) share them with their peers creates a rich set of downloadable content at zero cost to the production studio.  This “DLC” greatly extends the life of a game oftentimes by years.  The peer review nature of these downloads generates a lot of really valuable user data for the production company.  By watching what’s popular, what’s being discussed, and what’s being downloaded they can more easily design a richer and more rewarding gaming experience the next time they want to release another installment or even a brand new intellectual property.  The final things they gain are an instant hiring pool from which to draw and a free set of dedicated and very intense testers.  Some of the first DLCs produced by the people using these tools are usually what are called “unofficial patches,” which means they went through and fixed all the bugs and weirdness and outright insanity left in the game by the studio.  Of course after those you’ll start to see the usual nudity and chicken guns (this is the internet after all) but soon enough a lot of visual enhancements, physics tweaks, skeleton models, and even entire side quest worlds are submitted and essentially voted on by your customers.  The most successful of these designers are now working at Bethesda or were at least offered a high paying job if they wanted it.  So that’s why I’m back playing this game after a three year(ish) hiatus.  The mods and DLC I can now download have made it into a gorgeous, rich, engaging, and altogether brand new experience … and it only cost me the time it took to download the content and tweak out the conflicts.

So what does this have to do with test managers, you may well ask?  Ha ha … you must be new.

Video game companies are notorious for treating testers poorly.  Thankfully that has been changing for a lot of the new kids on the block, but at the old guard it’s still very much a job reserved for teenage monkeys who would rather follow a written script in a dimly lit room than work at the Hot Topic again this summer.  The other thing they have not usually done all that well is design games for someone else since “the customer” is some vague notion of a blob that sort of resembles them since they like to play games.  This is one of the reasons girls didn’t start gaming until relatively recently.  A giant majority of the people making games were boys and man-boys and they liked to design games they’d want to play.  This is why most video games have giant men with bulging muscles and strange satires on women who could never exist in nature even with surgical help and a team of assistants helping them get dressed.  To combat this problem of myopia, video game companies began creating open spaces for anyone who wanted to join.  These were originally places where people could post their game hacks but some smart people decided to nurture them by assisting them to make things.  They essentially backed off and admitted they might not know everything there was to know about gaming and gamers.  Rather than forcing their market into conformity with their assumptions, they stepped back and let their market tell them what to do.

This is why you need a software manager even if you are one of those full stack team organizations companies seem to love these days since everyone wants to be sexy and lean startups are sexy.  The almost universal hierarchy of a software development company has developers and architects at the top with everyone else following behind them.  If you want to get seed funding for your startup, you’d best be a developer or be partnered with a developer.  If you want to rise above a management role you’d best have the last few stints as a developer in your CV if you started out in a different profession or discipline.  Usually this is not a bad thing, but it does cause conflicts when this person attempts to manage other professions … by which I mean testers.  The average software developer/coder does not really know anything about software testing.  It’s not their fault.  Software testing does not excite them.  They don’t read articles on software testing unless they feel they have to since they’re now managing testers.  They don’t attend software testing conferences.  They don’t follow software testers on Twitter.  They don’t attend software testing meetups or participate in weekend testing events.  In short, they have zero qualifications when it comes to helping drive the direction and practices of the testers working under them.  They are also horribly unqualified to advise these people on their long term  professional growth should they want to stay in software testing.  In places with this sort of setup and expectation the testers will usually find themselves either managed and directed as if they were software developers or worse, ignored and simply told what to do when the time comes to tell them.

There is a different way and it’s the same way some video game houses have followed.  The software Sr Director/VP/Major Domo can simply let go and admit they don’t know everything there is to know about all things software related.  They can find the person or people in their company who are deeply engaged in the professional knowledge they lack and let them pursue their passion as they see fit.  Even in a self-sustaining team there is a lot of cross team interplay at healthy companies.  Someone needs to nurture that interaction and keep the momentum going when people stumble onto something awesome.  That person is usually called a “manager.”  It’s their primary job.  They make sure the people they’re nurturing have the tools, resources, and backing to pursue the things their experience and passion say are worth pursuing.  They help get people to understand when something is just a little failing and when it’s a flaming disaster.  They make sure people feels safe to do so.  You cannot accomplish this if you lack the authenticity of actual interest.

Let me say that again.  You cannot inspire passion in people if you do not feel passionate yourself.  Period.  Mike drop.

If testing is important to you and you want passionate testers, you need someone with just as much or more passion leading them.  When you get that you’ll soon see a massive return on your investment, just like the makers of Skyrim found out when they went against industry canon and helped people hack their game.

That’s a software testing manager and that’s why you need one.

Diversity Isn’t a Question of Biology

I keep hearing the topic of diversity raised at regular intervals since moving to the SF Bay area.  Having moved to what is unarguably still the center of cutting edge innovation in software development, it’s impossible to avoid references to “this thing we do.”  I imagine this must be what it’s like to live in Los Angeles or Mumbai and work in the film industry.  There are other industries around and plenty of people making money in other ways … but  what you do is so vitally important to the local economy that industry insiders are local celebrities.  All that was by way of giving context for why I can’t drive into work or watch television or attend an evening lecture without standing a better than average chance of being subjected to a talk on why there are so few [fill in blank] in the software industry.  It’s gone from interesting to amusing to noise.  Mainly because I think the entire premise is wrong.

To explain what I just said, I’ll refer you to a seminal work authored by Psychologist Howard Gardner back in 1993.  The title of the work is Frames of Mind and Multiple Intelligences and the concepts if not the work itself should be required for any manager working with any team.  The basic theory proposed by the work is that everything you know about intelligence is wrong or at least very narrowly defined.  Dr Gardner first asked the question that had plagued psychologists and behavioral analysts since the adoption of the “intelligence quotient” in 1912 when the German psychologist William Stern first proposed using it.  The problem was namely one of description.  If the intelligence quotient (IQ) was a good descriptor of intelligence and cognitive ability, why did some people who scored high never seem to do well while others who scored lower seemed to excel?  The attempt to reconcile this seeming problem lead people to such ideas as “cognitive maturity,” “fluid intelligence,” and a host of others all premised on the idea that IQ was not a flawed theory in itself but rather had a problem with a lack of tools to help explain it.

Dr Gardner took another approach.  He proposed the whole concept of how we looked at “intelligence” as a hierarchical structure with certain cognitive abilities at the top was essentially wrong.  The problem as he stated it was that there were/are approximately seven distinct types of cognition rather than one.  The intelligence tests and point system were designed to only measure and grade one type, meaning people who excelled in the others were ranked as having low or lower cognitive reasoning ability.

The seven types proposed by Dr Gardner as as follows:

  • Visual-Spatial – think in terms of physical space, as do architects and sailors. Very aware of their environments. They like to draw, do jigsaw puzzles, read maps, daydream.
  • Bodily-kinesthetic – use the body effectively, like a dancer or a surgeon. Keen sense of body awareness. They like movement, making things, touching.
  • Musical – show sensitivity to rhythm and sound. They love music, but they are also sensitive to sounds in their environments. They may study better with music in the background.
  • Interpersonal – understanding, interacting with others. These students learn through interaction. They have many friends, empathy for others, street smarts.
  • Intrapersonal – understanding one’s own interests, goals. These learners tend to shy away from others. They’re in tune with their inner feelings; they have wisdom, intuition and motivation, as well as a strong will, confidence and opinions.
  • Linguistic – using words effectively. These learners have highly developed auditory skills and often think in words. They like reading, playing word games, making up poetry or stories.
  • Logical -Mathematical – reasoning, calculating. Think conceptually, abstractly and are able to see and explore patterns and relationships. They like to experiment, solve puzzles, ask cosmic questions.

(Source: http://www.tecweb.org/styles/gardner.html)

Those of you who work in software development or IT in general can probably see where I am going with this article now.  Those of you who don’t or can’t, let me spell it out clearly.

The software industry is almost completely dominated by people with deeply evolved logical-mathematical cognitive abilities.  It is nearly impossible for anyone else to even enter this work force, much less rise to a position of authority.  Over the past few decades I’ve seen a lot more people with highly evolved cognitive abilities in linguistic cognition start to make inroads, but they are still a deep minority.

And to the people who correctly for some reasons want to see more [fill in the blank] represented in the software industry, I say go for it but stop saying hiring more of said person will result in “diversity of thought.”  If your entire culture is geared toward training, recruiting, promoting, and rewarding only one sort of intelligence then the only solutions or innovations you will generate are the ones available to that sort of intelligence.

It’s also an issue when you start looking at who you can make a team lead or a manager.  You can’t train someone to have high levels of interpersonal cognition with a few weekend seminars anymore than you can teach them to be a mathematician with the same … and yet that is exactly what we do in the software industry because we cannot conceive of anyone except a logical or possibly linguistic thinker being of any worth.

So if you want our industry to move into the 20th century you should stop trying to make everyone just like you and start looking for people who are not like you at all, but still have something to offer.

And in closing, here is the Ted talk by Temple Grandin that started congealing these ideas in my head back in 2010.


Your Review Process is Hurting Performance

“Q: What are the three signs[ of a Miserable Job]?

  • The first is anonymity, which is the feeling that employees get when they realize that their manager has little interest in them a human being and that they know little about their lives, their aspirations and their interests.
  • The second sign is irrelevance, which takes root when employees cannot see how their job makes a difference in the lives of others. Every employee needs to know that the work they do impacts someone’s life–a customer, a co-worker, even a supervisor–in one way or another.
  • The third sign is something I call “immeasurement,” which is the inability of employees to assess for themselves their contribution or success. Employees who have no means of measuring how well they are doing on a given day or in a given week, must rely on the subjective opinions of others, usually their managers’, to gauge their progress or contribution.”

from an interview with Patrick Lencioni

So what does the above quote have to do with our performance review, you might be asking.  Well hopefully that’s what you’re asking.  You might be asking yourself what an essentially useless activity you and your team are forced to conduct semi or annually solely for the purpose of arguing about who should get more, less, or the same amount of money has to do with performance.  If you are, please stop reading.  You’ve just answered the question for yourself.  In your case performance is harmed because you and your team are being asked to stop all productive work just so you can devote a week to two weeks (or possibly more) filling out forms and completing online surveys about why you aren’t more productive and why the guy sitting next to you is driving you insane because he keeps popping his knuckles.

In your case just stop doing the performance reviews all together and move on with your life.  Base any bonuses or promotions on what you’ve actually done for the company and call it a day.  If the team has a kick ass year, hand out a few checks.  Or better yet, have a kick ass party somewhere to thank everyone.  At one company we did an amazing job with our launched product.  It was more successful than anyone had ever imagined.  That year upper management used some of the extra money to rent an entire private beach front hotel in Mexico and stock it with all the food and drink we could consume for a week.  Was that a good bonus in comparison to a check?  Let me put it this way, that happened about a decade ago and I still talk about it.  The bonus checks and stock awards are soon out of mind and out of sight nearly as soon as they’re awarded.

Yeah … lying on a beach in Mexico with a whole passel of computer geeks just as hung over, flabby, and sun-burned as me.  Good times.

So for everyone else, what’s wrong?  You are supposed to be doing this activity which helps people become better and move forward in their careers.  You honestly want to do this because you care.  It also helps retain employees who know they’ve got somewhere to go.  You’ve purchased the software.  You’ve read the books.  You’ve attended the training about “SMART goals” and “360 reviews” and “stack ranking” but it still feels like a chore to you.  It seems like every year you pull up the documents you painstakingly wrote the year before only to feel that sinking realization we all feel.  Most of what’s listed there is either out of date, unrelated to the work now, or makes absolutely no sense to you now.  So now you have to map out the things your team did this year that “rocked” and the things that were “lame” (you can take the boy out of the Seattle grunge scene, but you can’t take the grunge scene out of the boy it would appear) and somehow map them to what you promised.  That sinking feeling you experience is partly the knowledge of this task ahead, but also the absolute certainty you are going to run smack into the Dunning-Kruger effect or the self-serving attributional bias even if you are not a manager or lead.  If you’re even mildly empathetic you’ll probably also be aware of how seemingly purposeless this whole exercise seems.  The only thing people argue with passion seems to be whether or not they deserve a 5% or an 8% bonus or raise.  It will often feel like this is an excuse for the employee to demand more money and the employer to demand less money.  In short it feels like the initial salary negotiation has been converted into an annual event, sort of a masochistic bank holiday.

Why is this?  Does it have to be this way?

No.  No it doesn’t.  However the alternative is very very hard.  I’m not kidding.  It’s hard because it involves a radical shift in how you view your work and your employees.  Most people and companies don’t want to make this shift because … well … not to put too much emphasis on this, but it’s hard.  The answer directly addresses the three states of being listed by Lencioni in the quote I listed above and in his excellent book, The Three Signs of a Miserable Job: A Fable for Managers (And Their Employees) which I also linked above.

I’m bringing this up because of this blog post by Mario Moreira.  I really hate to pick on him but I can’t help it.  He just makes it too easy.  He seems to recognize there is a problem with trying to define someone as a role and base their performance goals on that alone.  He wants to somehow personalize the process.  Unfortunately he appears both in his article and in our subsequent chats on LinkedIn to fall short because … it’s hard to do it the right way.

So in honor of Mario and his compatriots who want to do more of the same because what they’re doing now isn’t working, get ready for another multi-part entry from yours’ truly.


Why I think test certification is wrong … right at this moment

I’ve really nothing witty to say about this subject as a prologue to my thoughts, because its a serious topic.  As you read these words, know that somewhere out there a classroom of people have paid anywhere from $50 USD to $250 USD for the privilege of taking a test.  If they pass this test, they will receive a paper certificate saying they have passed and be granted the right to put a certain grouping of capital letters after their name no professional business cards, signatures, and online profiles.   For some this is a lifetime award, and for others they’ll need to again pay for the privilege every few years if they want to keep tagging said letters to their name.  The bodies conducting these tests are not affiliated with any accredited institution of higher learning nor do they have any sort of license or charter from a governmental oversight board.  The testing bodies generally fall under the umbrella of “professional organizations” which can be defined as a collection of people who’ve formed a group based solely on a common understanding of their trade and its work.  Since there can be many different understandings of their trade and its work, there can be  many different professional organizations.  Sometimes the differences can be apparently slight to the uninitiated, but it does not matter.  Professional organizations exist to serve the initiated and direct the uninitiated.  Sometimes these differences can have large consequences such as the one between the American Medical Association and dentists.  Since they would not be allowed into the AMA, dentists formed the ADA and to this day there are still pockets of violence and mayhem if one member (intentionally or otherwise) infringes on territory tagged by the other.  Stories of drive-by flossing are horrible, so there’s no need to repeat them here.

But what, you may ask, do these people sitting in a room (usually in Europe or India) have to do with me?  Everything.  According to the word on the street, if by “street” you mean various blogs and message-boards, these people represent the end of software testing as we know it.  After reading the passionate debate on both sides, I’ve some to conclusion the Mayans were actually talking about this conflict and not the end of the physical world.  So look for 2012 to be the year Software Testing as a job ends.  Update your resumes.

But in all seriousness, what’s really going on here?  Quite simply there are two forces at work.  The first is a natural tendency of trades to eventually evolve into professions.  And why do trades naturally evolve into professions?  To some extent its to keep up with advances in techniques and knowledge formerly kept private to protect your livelihood.  Another reason is the movement to a profession involves a codification of first principles and a body of knowledge which allows higher level discourse and development to occur.  It’s hard to move the conversation along if you’re spending all your time trying to figure out what someone means by a “pony wall” in a technical paper.  But mostly the trades move toward professions because once the uninitiated buy into the idea you are a profession, you can charge a lot more for your services.  You see when you work at a tradesman level you are required to obtain specialized skills and knowledge, but its seen mostly as an obtainable task requiring only a certain amount of study and a little practice.  For proof, witness the number of weekend tradesmen who decide they can re-tile their bathrooms in a weekend because they took a three hour course at Home Depot.  A profession because of the existence of professional bodies, bodies of knowledge, and seemingly arcane jargon give the impression of unique abilities to the uninitiated.  The outsiders then feel as thought they must compensate the initiated at greater levels if they want to take advantage of their rare and unique abilities.  For proof, how many people do you know who feel comfortable filling their own cavities after taking a weekend course at Home Dental?  Do you even know where the dental supply stores are in your town?  How about the hardware stores?

I rest my case.

So what are the identified steps toward becoming a profession?  According to Ask.com (the Wikipedia of the 90’s except with … you know … vetted contributors), they are as follows:

  1. The establishment of the activity as a full-time occupation
  2. The establishment of training schools and university links
  3. The formation of a professional organization
  4. The struggle to gain legal support for exclusion
  5. The formation of a formal code of ethics

These steps look good enough for me to throw out there as a working theory so look at them and see if you can figure out where software testing currently lives.  Have we established ourselves as an activity with a full time occupation?  Yes, with some debate still remaining amongst Luddites who persist in thinking nothing can occur without a coder dipping their beak.  But those people are to most extents doomed to the recycle folder of history as Software Test has pretty much established a beach head, moved into the dunes, and erected rides and concession stands along the piers.  So it looks like right now we’re attempting to formulate some professional organizations to help … but wait.  What happened to step number two?  Do we have recognized and accredited training schools offering degrees in software testing out there?

No.  The closest we currently come are a few classes offered as addendum at Colleges and Universities offering a more generalized Computer Science (ie: coding/architecture and sysadmin) degree.  Some lower level (two year colleges and trade schools) and unaccredited or “for profit” colleges are offering coursework in software testing but that doesn’t count.  So if step two does not exist, why are people attempting to jump straight into steps three and beyond.

The reason is the second (and possibly third) force at work … greed coupled with the desire not to seem like a fool.  Since institutions of higher learning do not currently offer comprehensive coursework on the subject or software testing, the professional organization seeking to collect people together can find a rich source of income filling that void for people seeking entry.  This is why certification by medical boards is NOT a multi-million dollar international industry, unlike software testing certification.  The business of these certification bodies is to make themselves as attractive to potential revenue sources as possible, primarily through numbers.  It’s a pyramid scheme to some extent.  The more people who get certified by a particular body, the more attractive it becomes to others.  The accrual of numbers is in essence lending it an air of authority.  McDonald’s used to attempt the same when it assured potential customers they shouldn’t feel guilty about purchasing a giant ball of fat and grease that tasted like cardboard because millions of other people had made the same choice.  The certification bodies are simply doing the same.  ” A million billion testers can’t be wrong.”  Unfortunately the cycle is becoming self-perpetuating for many because people with these certifications are now coming into positions of hiring authority.  Either because they don’t want to admit they were gullible and spent money on something that proved of little value or they actually believe in the system because of the air of authority imparted by the illusion of popularity, they have started making these letters after someone’s name a requirement for employment.  This is opening up even more avenues of revenue because now the certification bodies can loop in people who just want to get a job and find they can’t, even though they suspect it’s all “a racket.”

So is this self-perpetuating pyramid scheme the end of the testing industry, or even putting it in danger?  I don’t believe so.  I’ve been around long enough to know people who saw something similar happen with coding, and that seems to be doing okay as an industry.  Why did it stop?  Because step number two eventually asserted itself.  Four year colleges and accredited Universities began offering majors in computer science distinct from mathematics and/or electrical engineering.  Once that began happening, the natural tendency of the college academic is to publish their thoughts and ideas in order to subject them ACTUAL peer review.  Most crackpot or misinformed opinions cannot stand up to actual peer review, so all of the generalized “programmer certifications” went away.  They’d survived before only because they were usually published in an echo chamber where only those espousing similar beliefs would “challenge” them.  I see similar things happening with a lot of testing “academic journals” and other publications.  Rigorous academic review and healthy debate will soon enough silence them, once the colleges catch up to us.  In the end the only people harmed by these groups are the ones who may have spent more money than they could afford betting on a body of knowledge that later becomes discredited.  It’s an individual choice, however.  You’re free to roll the dice and spend your money how you would like.  You’re also free to perpetuate the cycle as a hiring manager as long as you would like, in spite of any evidence to the contrary.

But my advice to all of you is to weigh your choices carefully.  We’re not yet a profession as we just barely became a trade perhaps 30 years ago.  Spend money educating yourself and developing your trade craft and do the smart thing … wait it out.  In the meantime, attach yourself to the smartest and best tradesmen (and women) you know like a sucker fish cleaning off the shark.  You may guess correctly about where the industry will emerge in the decades to come, but you may not.  Can you afford to gamble with the money you’re being asked to put on the table?

Attracting and Retaining Good People

During my “sabbatical” I’ve had a lot of time to read and chat with a lot of people from different walks of life, or at least a lot of people who are team managers, recruiters, product managers, and generally high level muckety-mucks.  The rabble have tended to remain quiet, which is good for rabble.

In any event, there is a common refrain amongst these otherwise perspicacious group (look it up … rabble) that I find either surprising or funny, depending on my mood.  The common refrain is more of a complaint, really.  The complaint has to do with the fact these people all seem to be having a devil of a time attracting, recruiting, and retaining the sort of employees you normally want as a manager.  The people they (and the rest of us) dream of attracting and managing are intellectually curious, skilled, experienced, passionate about their career, and self-motivated.  In other words, the people you don’t actually need to manage except in acting as a conductor who directs and helps enliven the work.

As some one who’s had a small amount of success building out teams apparently made up of these sorts of people, I sometimes am asked about my “secrets” or “techniques” as such.  I’ve been hesitant to divulge my secrets for recruiting really good people, until now.  I will now give you my primary secret/principle for attracting and recruiting top notch talent.

I try and be someone for whom I would want to work every day I’m at work.  The rest of the stuff like training budget, high salaries, beer parties, and the like are largely either out of my control or somewhat superfluous.  Here are the actual things you need to implement and practice if you want to bring talented people to you like ants to spilled ice cream.

Respect – Assume everyone on your team and on other teams (often more difficult) are reasonable, talented adults who can be trusted to do their work efficiently and capably.  If you have reason to believe to the contrary, first check your beliefs.  It’s more likely you are just using a different definition of “efficiently” and “capably” than they.

Openness – This one comes in two parts.  You need to be somewhat explicit in your success expectations.  You also need to be somewhat current and explicit your expectations are not met.  Since we can assume we’re all adults (see above) we can also assume most adults really do want to do a good job.  If you are open to them about how they can do better, they’ll usually surprise you by exceeding your expectations.

Honesty – Being honest usually goes hand in glove with being open, but it also implies consistency.  No one likes aiming at a moving target or being told they messed up sometime in the past, when they’re no chance to correct it.  Honesty is about communicating successes to be celebrated and failures to be corrected.  It’s about engendering a reputation of trust so people feel confident enough to come to you with any problems or victories.

Interest – Everyone likes to think they matter to someone or something.  For most people, this translates to the workplace even if only due to the amount of time we spend there.  A good manager shows interest in their employees’ lives, professional ambitions, and daily toil.  They also show interest in their own role and industry.  Nothing attracts talented people in a certain industry more than the chance to work with others who share their interests.  Nothing keeps talented people around more than knowing they’re valued and appreciated on a daily basis.

Availability – Finally you need to be available to your team as much as humanly possible.  You never know when someone will need help resolving a thorny issue, overcoming a seemingly meaningless bureaucratic hurdle, or just want to kvetch in general about the injustice of it all.  You also never know when someone will want to show off a brilliant solution to a problem or run a new idea by you to see what you think.  You need to do paperwork and attend meetings, but your primary job is to manage a team of people … not resources.  People (even adults) will stop coming to you if they feel they can’t count on you being available.

The other thing to remember is you often don’t have to spend a lot of time recruiting talented people.  You can encourage your current staff to become talented.  In order to do that, you may have to fire or transfer away anyone not willing to themselves practice the rules of respect, openness, honesty, interest, and availability.  That’s the downside of building an excellent team, but there’s no way I can see around it.  Some people do not have the personality or desire to be excellent.  Aptitude is something you can work with.  Personality is not.  You can’t make some one excellent in spite of themself, but you can bring some one down.

In the end, I suppose that’s the one over-arching rule for success as a manager.  Create an environment where people know they are expected to exceed and will be given every tool they need to get there.  If you let them take charge of their own path to excellence, they will always surprise you with their capabilities and growth.  If you force them down your path, they will always disappoint.

Test Management: The job description should already be written

Recently a curious soul put out an open question to the general LinkedIn test manager community at large about whether they, as test managers, or some one else writes job descriptions for openings about to be posted publicly or internally.  As with most internet questions posted under the false guise of anonymity, there was more than likely an ulterior motive.  Since we can assume there is one, let’s just move on as its probably something as banal as they’re miffed about a job they accepted having little to nothing to do with the posted description to which they applied.  I can answer that concern with a simple statement, no one does the job listed on the description for which they applied.  Even the person hired as a ditch digger will more than likely be someday soon asked to dig a trench, a drain, or a trough.  The medieval poetry master’s candidate then has three choices: they can shrug their shoulders and think about rondelles while they dig; they can walk off the job because they’re a union certified ditch digger and if the company wanted a trench they should have hired some one from the local 147 branch of International Trench Excavators; or they can dig the hole and complain loudly to anyone with no actual authority to fire them about the injustice of it all.

Guess which one the people complaining about job descriptions usually fall into?

Of course there could be an actual issue here, which is a failure to match up stated expectations with the actual work some one performs on a daily basis.  The issue of work mismatch is a lot more common than you would think.  You can easily check on whether you suffer from it by doing a simple test.  Do you know if you’re doing a good job or meeting the expectations of your boss without needing to go ask them?  If you didn’t need to think about the answer to this question, you’re either a cocky bastard or you are working in a position with clearly defined expectations and success criteria relevant to your daily work.  In either case you’re lucky because either you can spend your time focusing on actually doing work with a clear idea of how to advance (if you should want) or you’ll probably do well in life anyway if you can back it up because cocky bastards usually do if they can.

For the rest of us, a lack of understanding usually indicates we’re working at a position where our expectations and success criteria are either ill defined, subjective, conflicting, or completely at odds with our actual work.  There are several reasons for this but the most common I’ve seen is revulsion at the thought of doing “busy work.”  Most people at most companies consider things like role definitions and such to be odious tasks best left to others with no actual work to perform.  Real workers are seen as too valuable to waste their time on bureaucratic nonsense that will soon be outdated soon in any matter.  The trouble is this hurts companies in two ways.  First it hurts morale because workers don’t have a clear idea of what they are expected to be doing, how to judge their own performance, and what steps they need to take in order to get that raise, promotion, or bonus.  Second it hurts recruitment because each job opening is then treated as a special snowflake to be treated as an individual miracle or they’re treated as identical cookies cut from the same form with a few differences in toppings for decoration.

As an example, without a clearly defined set of expectations for a role the hiring manager, recruiter, and agent are all forced to guess what is now the difference between a senior development engineer and just a regular development engineer.  Taking it even further, perhaps there are no senior development positions even though your needs cover a wide range of experiences and skill sets.

I know your company is different.  You’re showing the world what a ragtag group of misfits can accomplish without all those titles and rules, man.  Our founders call themselves “Head Necromonger” and “Chief Gelfling” respectively.  I would then submit that come time to hand out bonuses or stock options or whatever you want to call it, there are going to be people who wonder why one gelfling received a larger share of stock than the gelfling working across the cubicle wall.  The recruiter you hire will also undoubtedly wonder what the hell a “necromonger” is and which Monster group should they post the opening into.  Is it computer software or hardware or is it a religious thing and should therefore go into nonprofits?

If you spend the time to accurately document the roles and their expectations within your organization, you will find your workload as a manager eased more than you can now comprehend.  Performance reviews then become a fun activity where you can work positively with some one to advance their careers or shore up weaknesses preventing them from getting the recognition.  Conflict management within the team becomes so much easier since most of these issues fall into the “not my problem” category.  But most of all, when you finally get the okay to hire that new team member you’ve been following the director into the bathroom to lobby for … you’ll actually have most of  a really good job description completed.  Just grab the role description and add the fun bits about specific skills or needs and run it through HR to make sure nothing you don’t want public will get out.

And then its done and you can then turn your attention to sorting through the 500 resumes on your desk because it turns out there really is a job called “necromonger” in Utah and the primary employer for that industry just announced a staff reduction of 30%.

Good luck, my little gelflings.

Learning Development: The role of QA during disruptive innovation

Modern (post-industrial age) sociology has created a few useful fictions or stereotypes which have found their way into the general parlance.  True, most times the general understanding of these fictions is completely inappropriate or outright incorrect when compared against the actual usage meant when coined.  But one thing is certain when it comes to concepts being taken out of context, no one really cares except the people who originally formed them.

I bring this up because one of those useful fictions is causing me some issues lately, and I’d like to address it.  The fiction is the supposed tension between conservative and liberal elements within a society.  Before knickers are bunched and assumptions are brandished like weapons, let’s address the terms themselves.  Conservative elements those elements resistant to change while liberal elements actively promote change.  Supposedly there is tension within both micro and macro societies between elements attempting to defend the status quo and those seeking to alter it, the ascribed motivations for each usually being a function of one’s particular belief system when it comes to sociological or political or economic theory.

The issues I’ve been having lately are warring camps who have decided to stake claims within the quality assurance community around sequential versus iterative development methodologies.  Sides have apparently been chosen and mental territory is being defended with the vehemence of children on a playground.  Accusations ranging from luddism to anarchism being tossed back and forth with wild abandon.  Of course like most dogmatic debates based on theoretical constructs, these completely miss the point.

So what is the point, you may ask?  The point is appropriateness.  Clayton M Christesen’s book The Innovator’s Dilemna (required reading in my opinion) raises the very interesting point that established companies are successful because they have developed processes which drive or facilitate their success.  In the case of innovation in alignment with these processes, he deomonstrates established companies have a remarkable track record for staying ahead of the curve and leading their market in both quality and innovation.  It’s only when the market changes due to what he terms “disruptive innovations” (ie: 3.5 inch disc drives, hydraulic excavation tools, dirt bikes, etc) that established companies fail while smaller companies succeed.

Mr Christensen’s (paraphrased) explanation for this is simple, the processes in place at successful companies cannot be drastically altered in the face of disruptions.  Successful companies can bend and adapt processes, but only up to a point.  What do I and he mean by that?  Let’s take a look at solid state disc drives.  The movement from 8 inch to 5 inch disc drives was disruptive because it reflected a radical change in the user market with different needs and requirements (personl workstation vs PC).  Does the shift from magnetic to solid state drives represent a similar shift?  No.  The solid state drives currently being marketed are used in the same manner as a magnetic drive, by the same customers, and represent a significant leap in performance (but not stability).  A more interesting disruptive technology is the use of SD cards, which with the advent of SDXC have the possibility of storing as much data as a normal hard drive with the added benefit of an incredibly compact size and amazing portability.

The follow up point Mr Christensen makes about disruptive technologies is the creators nearly always get it wrong at first when predicting markets.  Xerox got it wrong with the mouse, and so gave it to Macintosh.  Honda got it wrong with their small 150cc motorcycles, but got it right later when outdoor enthusiasts in California began buying them for off road fun.  The lesson to be taken from these examples is a company working in a disruptive technology must practice learning development and not commit their resources to a particular path.  They must work with the markets, experimenting with offerings, and altering their designs and assumptions at the requests of the market rather than assumptions or market research projects.  If a car company wants to purchase your 3.5 inch disc drive for a navigational system and cares more about durability than write times, start making them more durable and stop working at increasing capacity and write times  Does that sound familiar to any of you?  It should because that learning as you go model is capital “A” Agile defined for all industries.

So what does a quality assurance professional do when there are no defined processes to review as the company is learning what those should be?  Plenty.  The quality assurance professional in a disruptive company should act as the opposite of a process improvement expert.  Companies looking to build out a disruptive market cannot afford to define their best practices until they have established themselves as successful.  To do so is to miss market opportunities, allowing another company to grab them.  So the role of a disruptive QA is to constantly act as an iconoclast while making sure all market opportunities are explored, even if they at first do not seem to be in alignment.

So to summarize, there is no one way to do things for all companies.  Processes like CMMI ad Six Sigma are highly effective in established companies seeking to lead through sustaining innovation (ex: solid state drives for the PC).  These same processes are death to a company seeking to lead through disruptive innovation (ex: SD cards with embedded operating systems) as they will inevitably result in missed opportunities and inflexibility.  The role of QA in sustaining innovation is to constantly work at improving processes by looking for inefficiencies, waste, duplication, or other sources of “muda.”  The role of QA in disruptive innovation is to constantly work at making sure processes are supporting the learning development rather than defining it.  In a disruptive company there is no “muda” as there is no frame of reference for waste.

Consulting the oracles didn’t help Laius

I’ve been getting an education lately in something called “test oracles” by a couple of people I know and have come to respect as far as those things go.  Initially I was surprised I wasn’t aware of the term before now, but then I wasn’t.  Chances are I’d heard it and then promptly forgotten it.  I don’t think I’ll be able to clear out the clutter this time, so I might as well defend my decision to shift-delete the thing if I can.

The context of this re-education came about during a discussion about negative testing which, as usual when I’m involved, quickly branched out faster than the Mississippi delta into a wide range of topics from the appropriateness of negative testing to the use of testing itself.  The subject of test oracles came up when I posited the role of testing was to check and validate assumptions about the functioning of a product.  This caused quite a stir, probably because I used the words “assumption” and “validate” within the same sentence.  I defended my stance by making the claim we could not know if a test was “passing” or “failing” if we did not have some expectation of assumption of how we expected the system to function.  If the point of a test is to actually learn what something does, then any behavior up to and including smoke and fire leaping out of the housing are not failures.  Of course neither are they passes.  Experienced behaviors in a learning system are all simply experienced.  They live in the weird transitional state until some one makes a determination, labeling them as either a pass or fail.  This is or course what I meant to say.  But since the conversation was proscribed into 120 characters, it lost something in the translation.

I was then told/scolded by several people that we most certainly do not use assumptions when conducting testing.  We use something called oracles, and only oracles.

My initial reaction to this was something akin to shock.  My education and interest in classical Greek mythology as well as world religion and anthropology made me perk up my ears at this use of a very specific term.  You see oracles were (and are) very specific roles within societies spanning the globe.  Oracles received instructions from divine sources directly and were thought to relay them directly to humans.  People within a society visited an oracle when they wanted to ask a question of the divine directly, with no mucking about using entrails, star positions, runes, or other nonsense.  Visiting an oracle was something not to be taken lightly, by any means.  It was generally thought the divine didn’t appreciate being disturbed with questions about a lost ring or your neighbor’s donkey getting into your garden.  Some cultures even believed consulting an oracle would then bring you to the personal attention of the divine, which was not generally considered to be a good thing.  Finally the pronouncements of the oracles were often mysterious and prone to misinterpretation.  Tragedy in Greek literature, for instance, is primarily based on humans attempting to interpret oracular pronouncements and either seeking to avoid their supposed fate or just outright getting it wrong.  A good example of this would be the father of Oedipus, Laius.  The oracle warned him if he fathered a son, his son would murder him later in life.  Laius then set to kill his newborn son by leaving him out in the open, exposed to the elements.  Long story short, Oedipus survived and later killed his own father not knowing it was his biological father.  The tragic element of this story was had Laius kept the baby in his house, he would have known his father and probably not killed him  In other words, the statements of the oracle while true were not useful, informative, or even enlightening in any way.  In fact they were the opposite as if the king had never received word from them, he would probably have simply raised Oedipus as his son and heir and gone on to live a long(ish) and happy life.

Are the statements of these people really what you want to base the determination of your testing on?  Really?

Then I began to dig a little deeper and realized the term does not really imply an oracle in the classic sense.  A test oracle is “a mechanism for determining whether the program has passed or failed a test” according to Cem Kaner.  In and of itself I don’t have an issue with this statement, but it’s so broad as to defy any reasonable dissension.   And that’s the issue I have with the term “test oracle.”  It’s such a broad term as to be effectively useless since it not only uses itself to define itself, it confuses function and definition.  I dug further into Mr Kaner’s definition where he outlines three capabilities of a testing oracle, and they’re all like that.  It’s like if some one asked you to define a stop sign and you simply said, “It’s a sign that tells us to stop.”  So any sign telling us to stop is a stop sign, even if it could reasonably be called something else such as a stop light or a crossing sign.

I stand by my statement about assumptions because the oracular definition of itself contains unstated requirements for assumptions.  Mr Kaner’s definition, for instance, of the three capabilities continually references “the predicted behavior” but does not specify from where the predictions are generated.  There has to be an initial assumption of behavior for a prediction to be made.  Reading through other definitions from Hoffman to Memon, they all keep using the phrase “expected results.”  It looks like everyone has decided using the (to my opinion) more appropriate/descriptive term “assumptions” is wrong so the term “expected” will be used in its stead.

I still wish to suggest we not use the phrase “test oracle” as it does not really describe what we’re attempting to accomplish with the overly broad term.  I’d like to suggest we instead use any of the following phrases since they are generally more appropriate to what we are meant to be doing.

Testing augurs – Augury was a form of divination that identified and ranked the likelihood of several possibilities using a physical medium such as entrails or the flights of sacred birds

Testing sortes – Sortilege was a form of divination relying on the imposition of order on seemingly random occurrences such as the pattern of shuffled playing cards or cast bones

Testing omens – This form of divination simply required the recording and interpreting of unexpected or unusual events, the birth of unusual livestock being the most popular

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 1,077 other followers