In the interest of salacious name dropping, I was recently deeply entrenched with Peter Farrell-Vinay over on a LinkedIn discussion about KPI’s (Key Performance Indicators) for quality in a software release. The all too familiar debate broke out between the objectivists and subjectivists when defining quality. It’s not actually that interesting to me as a debate but here are the two arguments in a nutshell with gross simplification and lots and lots of specific cases not being met assumed.
- The objectivists believe software quality is intrinsic to a product and is therefore measurable using objective criteria over a period of time
- The subjectivists believe software quality is a function of market/customer perception and therefore cannot be objectively measured since the criteria over time will radically change as conditions change
To put it into even more succinct terms, the objectivists almost always want to start measuring things like defect density and code coverage while the subjectivists almost always start using the “how do you measure happiness or fun” simile to define their position. It’s an uninteresting discussion to me as it seems to devolve into quickly into and argument of paradigms that will never really be resolved. You are either some one who thinks the only solvable problems are those that can be expressed in quantitative equations and discreet parts or you are some one who thinks the world is “fuzzier” than that and we should be addressing issues from the standpoint of complex systems and interactions. Just like I can’t talk some one into Judaism or Islam, I can’t talk an objectivist into a subjectivist and vice versa.
Like I said, it’s frustrating, pointless, and ultimately boring. But that’s not what this entry is about. Peter asked me an interesting question about customer satisfaction surveys and how they can be useful. He specifically asked me about the typical “on a scale of one to five, how useful did you find our last release” sort of questionnaire we can’t seem to jettison from our tool box. It then struck me this is how most of us view customer satisfaction measuring activities. It’s a shame but not totally expected based on the fact most of us have little to no interest in pursuing ideas or concepts outside our small little areas of excellence (see my earlier grumpy rants about hiring almost exclusively for technical competence and expecting other competencies to be “easy” or “trainable” for my views on this subject) since most of our industry thinks “cross-discipline” means some one who has knowledge of cloud computing and mobile device platforms.
Customer satisfaction gathering is a traditional marketing skill set some one looking to move more toward a QA analyst role should add to their toolbox. The first step is usually a survey but its only the first step and is generally thought of more as a sampling tool than anything that will generate useful satisfaction statistics. If you’d like more information about how to write a decent customer survey, I can’t recommend Priscilla Saint’s and Don Dillman’s book How to Conduct Your Own Survey enough. It’s the first book I read on creating my own surveys and its still useful for me as a quick reference when I need to remind myself of what sort of survey might get me the information I need based on my needs.
But what about the darn cows, you’re probably asking. I’m getting to that. Cows are a good reference to me because of my history. My grandparents used to run a small ranch that these days would be wildly successful since they ran purebred Hereford cows that were mountain grass fed and completely free range. The important bit here is “mountain” when you read that statement. These cows were not the sort you could just saddle up a horse and find by scanning the tall grass lands visible in all directions until the horizon met the skies. Nope. These cows were the devilish mountain cows that could hide an entire herd in a thick stand of trees and bushes a hundred yards from the mystified cowboy. The only way to find these cows was a two prong assault using your knowledge of cow behavior and the local geography (ie: where is the fresh water, where are the sweet grasses, etc) and trickery in this case in the form of a ten year old boy that was (and still is) and excellent mimic. The mimic’s job was to ride around and bellow like a calf and later a cow. If the mimic was decent enough you’d eventually hear an answering call which I can only assume was cow for “Your accent is atrocious” or “You sound like a mountain lion has you by the throat, are you okay?” The calls could then be used to triangulate the rest of the herd since cows are not likely to go wandering off alone.
In this way, the cows is like your markets. Clayton Christensen in The Innovators’ Dilemma talked about actual customer behavior once you move beyond the strange little group of innovators most small companies usually target. These later adopters are more like cows in that they find safety in numbers. This makes an immense amount of sense if you think about it. A large group moving in a certain direction usually means better support, a wider knowledge community, and better interoperability and compatibility for other things you may want to use later. If you can find out where these groups (herds) are heading, you can extrapolate how you are viewed. So your job is to find the “chatty” cows moving with this herd. This is where the customer satisfaction survey comes in handy.
You want to look for the respondents that seem to follow the herd but are liable to give you more than just a check in the “1 to 5” rating index. You want to find the cows that are bellowing for your cow calls but are not the ones likely to wander off to see if there’s something interesting over the next ridge. This means you’ll need to follow up with these answering calls to check and see if they’re likely to be a second stage adopter or a first stage adopter not as likely to represent the larger market. Once you’ve rated their likelihood you can verify it with deeper contact and interaction. It still relates because an experienced cowboy will often develop a knowledge of their herd to the point where they can sometimes tell the difference between cows based only on their vocalizations.
Your marketing people and your quality people need to develop a similar knowledge of your markets, it not your specific customers.