I’ve been getting an education lately in something called “test oracles” by a couple of people I know and have come to respect as far as those things go. Initially I was surprised I wasn’t aware of the term before now, but then I wasn’t. Chances are I’d heard it and then promptly forgotten it. I don’t think I’ll be able to clear out the clutter this time, so I might as well defend my decision to shift-delete the thing if I can.
The context of this re-education came about during a discussion about negative testing which, as usual when I’m involved, quickly branched out faster than the Mississippi delta into a wide range of topics from the appropriateness of negative testing to the use of testing itself. The subject of test oracles came up when I posited the role of testing was to check and validate assumptions about the functioning of a product. This caused quite a stir, probably because I used the words “assumption” and “validate” within the same sentence. I defended my stance by making the claim we could not know if a test was “passing” or “failing” if we did not have some expectation of assumption of how we expected the system to function. If the point of a test is to actually learn what something does, then any behavior up to and including smoke and fire leaping out of the housing are not failures. Of course neither are they passes. Experienced behaviors in a learning system are all simply experienced. They live in the weird transitional state until some one makes a determination, labeling them as either a pass or fail. This is or course what I meant to say. But since the conversation was proscribed into 120 characters, it lost something in the translation.
I was then told/scolded by several people that we most certainly do not use assumptions when conducting testing. We use something called oracles, and only oracles.
My initial reaction to this was something akin to shock. My education and interest in classical Greek mythology as well as world religion and anthropology made me perk up my ears at this use of a very specific term. You see oracles were (and are) very specific roles within societies spanning the globe. Oracles received instructions from divine sources directly and were thought to relay them directly to humans. People within a society visited an oracle when they wanted to ask a question of the divine directly, with no mucking about using entrails, star positions, runes, or other nonsense. Visiting an oracle was something not to be taken lightly, by any means. It was generally thought the divine didn’t appreciate being disturbed with questions about a lost ring or your neighbor’s donkey getting into your garden. Some cultures even believed consulting an oracle would then bring you to the personal attention of the divine, which was not generally considered to be a good thing. Finally the pronouncements of the oracles were often mysterious and prone to misinterpretation. Tragedy in Greek literature, for instance, is primarily based on humans attempting to interpret oracular pronouncements and either seeking to avoid their supposed fate or just outright getting it wrong. A good example of this would be the father of Oedipus, Laius. The oracle warned him if he fathered a son, his son would murder him later in life. Laius then set to kill his newborn son by leaving him out in the open, exposed to the elements. Long story short, Oedipus survived and later killed his own father not knowing it was his biological father. The tragic element of this story was had Laius kept the baby in his house, he would have known his father and probably not killed him In other words, the statements of the oracle while true were not useful, informative, or even enlightening in any way. In fact they were the opposite as if the king had never received word from them, he would probably have simply raised Oedipus as his son and heir and gone on to live a long(ish) and happy life.
Are the statements of these people really what you want to base the determination of your testing on? Really?
Then I began to dig a little deeper and realized the term does not really imply an oracle in the classic sense. A test oracle is “a mechanism for determining whether the program has passed or failed a test” according to Cem Kaner. In and of itself I don’t have an issue with this statement, but it’s so broad as to defy any reasonable dissension. And that’s the issue I have with the term “test oracle.” It’s such a broad term as to be effectively useless since it not only uses itself to define itself, it confuses function and definition. I dug further into Mr Kaner’s definition where he outlines three capabilities of a testing oracle, and they’re all like that. It’s like if some one asked you to define a stop sign and you simply said, “It’s a sign that tells us to stop.” So any sign telling us to stop is a stop sign, even if it could reasonably be called something else such as a stop light or a crossing sign.
I stand by my statement about assumptions because the oracular definition of itself contains unstated requirements for assumptions. Mr Kaner’s definition, for instance, of the three capabilities continually references “the predicted behavior” but does not specify from where the predictions are generated. There has to be an initial assumption of behavior for a prediction to be made. Reading through other definitions from Hoffman to Memon, they all keep using the phrase “expected results.” It looks like everyone has decided using the (to my opinion) more appropriate/descriptive term “assumptions” is wrong so the term “expected” will be used in its stead.
I still wish to suggest we not use the phrase “test oracle” as it does not really describe what we’re attempting to accomplish with the overly broad term. I’d like to suggest we instead use any of the following phrases since they are generally more appropriate to what we are meant to be doing.
Testing augurs – Augury was a form of divination that identified and ranked the likelihood of several possibilities using a physical medium such as entrails or the flights of sacred birds
Testing sortes – Sortilege was a form of divination relying on the imposition of order on seemingly random occurrences such as the pattern of shuffled playing cards or cast bones
Testing omens – This form of divination simply required the recording and interpreting of unexpected or unusual events, the birth of unusual livestock being the most popular