Test Automation Planet

In my Don’t Give Test Cases To N00bs post I tried to make the argument against writing test cases as a means to coaching new testers.  At the risk of sounding like a test case hater, I would like to suggest three contexts that may benefit from detailed test cases.

These contexts do not include the case of a mandate (e.g., the stakeholder requires detailed test cases and you have no choice).

1. Automated Check Design: Whether a sapient tester is designing an automated check for an automation engineer or an automation engineer is designing the automated check herself, detailed test cases may be a good idea.  Writing detailed test cases will force tough decisions to be made prior to coding the check.  Decisions like: How will I know if this check passes?  How will I ensure this check’s dependent data exists?  What state can I expect the product-under-test to be in before the check’s first action?
2. Complex Business Process Flows:  If your product-under-test supports multiple ways of accomplishing each step in its business process flows, you may want to spec out each test to keep track of test coverage.  Example: Your product’s process to buy a new widget requires 3 steps.  Each of the 3 steps has 10 options.  Test1 may be: perform Step1 with Option4, perform Step2 with Option1, then perform Step3 with Option10.
3. Bug Report Repro Steps: Give those programmers the exact foot prints to follow else they’ll reply, “works on my box”.

Those are the three contexts I write detailed test cases for.  What about you?

In which the conversation about heuristics and oracles continues…

Tony’s brow furrowed as he spoke. “No oracle comes with a guarantee that it’s giving you the right answer. That’s what you said. But surely there are some oracles that are reliable,” he said. “What about pure math?”

“Pure math? All right. Here’s an example: what’s 61 plus 45?”

“Duh. 106.”

“Well,” I said, “for many computer systems prior to the year 2000, if you added 45 to the year 61, you’d get 6. That is, if you looked at a printout or a screen, you’d expect to see “06″ in the year field. And for those systems, that would have been the right answer.”

“But that was wrong! Y2K was a problem. They called it ‘the Y2K problem‘, didn’t they?”

“True,” I said. “But until the late ’90s, it wasn’t a problem—or to be more accurate, people didn’t perceive it as a problem. On the contrary, it was a solution to a problem: memory and storage were expensive. You could work around the “problem” with a combination of clever code and trust that people would interpret the output appropriately. Remember, a problem is a problem to some person at some time. Programmers and designers in the 1960s had one set of problems to solve, and programmers at the end of the ’90s had another set. The point is that one oracle (regular math) would give you one right answer, and another oracle (what the programmers and designers wanted) would give you another. Listen: no oracle can give you the right answer. An oracle can give you a right answer—a plausible answer that might be right for its context. But changing the context can flip that right answer into a wrong one—or a wrong answer into a right one.”

“Oracles are heuristic,” I continue. “There’s this terrific book, Discussion of the Method, by Billy Vaughan Koen. He’s an engineer, but he’s also a philosopher of engineering. In the book, he makes the argument that all decision-making, all problem-solving is heuristic.”

Tony looked quizzical. “Wait… Even algorithms? ‘Algorithm’ is the opposite of ‘heuristic’—didn’t you say that?”

“Not exactly. Algorithms are robust; they tend to produce very reliable results. But Koen says that even algorithms are heuristic. After all, if you apply an algorithm in the wrong way, to solve the wrong problem, or in the wrong context, it will fail.”

“Aaargh,” Tony said. “Where does that leave us? How can we ever know when a program’s correct?”

“That’s the interesting part,” I said. “We can’t. A program can appear to be working in all kinds of ways, but the program and your oracles can fool you.. Think of a calculator program. Yep: one plus one gives the answer ’2′. That’s looks correct to you, right?”

“Sure.”

“And yet if the calculator is in binary mode, the answer should be ’10′. You might be applying the wrong oracle for a given problem. Even if the program isn’t in binary mode and ’2′ is right, the program could be tying up the processor so your machine is unusable. Or the program gives you the right answer—in white text on a white background. Or the program clobbers the contents of the clipboard. And you don’t notice these things unless you’re looking for them, or unless you happen to notice them. That is, there might be a problem for which you don’t have an oracle..”

“So nothing can tell us that a program’s working right? We can’t ever tell whether a program is giving us the right answer?” Tony asked doubtfully. “That doesn’t sound… right.”

“Working right, yes, but only in the sense that it appears to be fulfilling some requirement to some degree. A right answer, yes, but the right answer only in context, and not a complete answer. Correctness is a human notion, and things are only correct in some context. As testers, we can’t know for sure the deep truth about any observation. Any right answer that we see in computer software is only right for now, this time, for some purpose, on this machine. We can’t reliably project our observations into the future. We can use an oracle to give us as a strong inference that the answer will be the same next time, but we don’t get a guarantee. What we see might be right based on what we’re observing, but there’s all this stuff that what we’re not observing too. Cem Kaner and Doug Hoffman describe that stuff really thoroughly. You’ve heard that complete testing is impossible, right?”

“Of course.”

“Well, part of that is the coverage problem; we can’t test every possible input to a program in a finite amount of time. But part of it is the oracle problem, too. We can’t see a problem unless we have an oracle for that problem: that is, a principle or mechanism for recognizing that problem. All our oracles are heuristic, fallible—and in software, the potential for problems is limitless.”

“So how do we get around that?” Tony asked.

“The first thing is to recognize that oracles don’t give us the right answer, but every oracle may be able to point us to some problem. Over the years, we we’ve studied oracles, we’ve come up with a bunch of principles and mechanisms for them, and we keep discovering more. Since there are infinite numbers of possible problems, we need a wide variety and diversity of oracles to spot them. But there is one principle that seems to prevail overall.”

“What?”

“It seems to us that oracles are founded on the idea of consistency.”

(to be continued)

For the past several months I’ve been involved with, but unable to publicly discuss, an exciting project called RubyMotion. My friend Laurent Sansonetti, creator of MacRuby, has now officially launched a serious alternative to Objective-C for iPhone and iPad development.

RubyMotion is not a bridge; it actually generates native applications that are as fast as the ones written in Objective-C . You can check out Ars Technica’s exclusive article for more details on the project and watch the Pragmatic Studio’s 50 minute free screencast about it to get a better idea of how RubyMotion works in practice.

For a limited time you can buy RubyMotion at an early bird discount rate of 25% off. It’s such a productive and affordable solution, that I’m sure it will quickly become popular among Ruby developers who are interested in iOS development and testing, as well as among existing iOS developers.

Many of the questions you may have can likely be found in the FAQ section, but feel free to ask any additional questions here. Enjoy it.

Possibly related posts:

1. Is the Enterprise world Rails ready?
2. Would you use a Mac mini as your development machine?
3. Mobile Development Marketshare Among Hacker News’ Participants

• Learn practical knowledge about designing experiential exercises.
• Expand my understanding of what participants experience during experiential exercises.
• Unlearn things that interfere with effective experiential learning.
• Help to expand my "big picture" about this topic.
• Link to other knowledge to help increase my effectiveness.
• Figure out if students are really learning.

In which the conversation about heuristics and oracles continues…

“So what’s the difference,” I asked my tester friend Tony, “between an oracle and a heuristic?”

“Hmm. Well, I’ve read the Rapid Testing stuff, and you and James keep saying an oracle is a principle or mechanism by which we recognize a problem.”

“Yes,” I said. “That’s what we call an oracle. What’s the difference between that and a heuristic?”

“An oracle helps us recognize a problem, but it’s not a method for solving a problem, or for making a decision.” He suddenly paused.

“Wait,” he said. “There’s that question you say testers should always be asking—Is there a problem here? An oracle does help us make a decision: it helps us to decide whether there’s a problem in the product we’re testing. And oracles can fail, too. So an oracle’s not different from a heuristic; an oracle is a heuristic. They’re the same.”

“Okay,” I said. “But that’s like saying ‘an iPhone isn’t different from a smartphone; an iPhone is a smartphone. They’re the same.’”

“But? But what? What’s the problem with that? Aren’t all iPhones smartphones?”

“Well, I’d say so,” I replied. “But let me ask you: are all smartphones iPhones?”

He paused for a second. “Oooh. Oracles are heuristic, but not all heuristics are oracles. An oracle is a heuristic, but it’s a specific kind of heuristic. Okay, let me see if I’ve got this: tossing a coin is a heuristic for making a decision. A heuristic approach for making a decision, I mean. You’d use the Coin Toss heuristic in some contexts—random decisions, or unimportant decisions, or… or intractable decisions, or decisions that you want to be fair. The approach can fail. It might not be a fair coin. Or it might be a high-stakes decision that shouldn’t be left to chance. So the Coin Toss heuristic might work, it can fail.”

“Right,” I said. “Tossing a coin is a heuristic approach for making a decision.”

“But it’s not an oracle,” Tony said, “because tossing a coin doesn’t help us to recognize a problem. So tossing a coin is a heuristic, but it’s not an oracle.”

“All right. What does an oracle do for us?”

Tony started confidently. “An oracle is something that gives us the right answer, so that we can compare it to the result the product gives us. If there’s a difference between the oracle’s answer and the product’s result, there’s a problem. If the product’s answer is the same as the oracle’s answer, then there’s no problem.”

“Are you sure about that?” I asked. “Is a specification an oracle?”

“Yes. The specification tells us how the product is supposed to behave.”

“And how reliable are the specifications where you work?”

Tony paused, and then he grinned. “Okay. They suck, to be honest with you,” he said. “They’re ambiguous. They’re unclear. They’re incomplete; they usually miss a bunch of requirements. They contradict each other, sometimes on the same page. So we have to talk about them a lot to clear them up—and then when we sort things out, the job of updating the written spec usually gets left for last, if it ever happens at all.”

“Still,” I said, “if you see an inconsistency between the spec and the product, you at least suspect a problem, don’t you?”

“Well, yeah. When the spec and the product disagree, there’s usually a problem somewhere—either with the product, or with the spec. Or both. When we’re not sure, the program manager is usually the one who clears things up. Sometimes the programmers fix the product. Sometimes the the product turns out to be right, and it’s the spec that’s wrong—but then we know at least the BA’s ought to fix the spec, even if they don’t get around to it right away.”

“So if you use a specification as an oracle, it’s somewhat reliable, but it’s not guaranteed to be right. What does that sound like?”

He paused again. “It’s a heuristic. An oracle is a special kind of heuristic. An oracle is a heuristic principle or mechanism by which we recognize a problem.”

“That’s the way I like to say it these days, yes,” I replied. “For one thing, having the word ‘heuristic’ in the defintion of ‘oracle’ seems to help people recognize that there’s some kind of distinction to be made between heuristics and oracles. But for another, I think it’s important to emphasize that oracles help us to learn things. And that, since they’re heuristics, oracles are fallible and context-dependent. No oracle comes with a guarantee that it’s giving you the right answer. An oracle can only point you to a possible problem.”

Tony’s brow furrowed again.

To be continued…

People sometimes seem to struggle with a concept that’s central to testing, the concept of “oracle”. In the three-day Rapid Software Testing class, we define an oracle as

a principle or mechanism by which we recognize a problem.

Sometimes I like to emphasize that oracles are fallible and context-dependent. When that’s so, I say that an oracle is

a heuristic principle or mechanism by which we recognize a problem.

That means that an oracle can work but might fail in helping us to recognize a problem. A heuristic is something that helps us learn; it’s a fallible method for solving a problem or making a decision. So an oracle, as a special kind of heuristic, helps us to make a particular decision in answer to the question, “Is there a problem here?” If the answer is Yes, it’s because an oracle is enabling us to recognize a problem.

For some time, I’ve been surprised at how tricky a concept this seems to be for some people. Possibly, I thought, it was because people simply weren’t used to thinking about oracles; for many it’s a new word and a new concept. Maybe the problem was with “heuristic”, or “principle”, or “mechanism”. Recently, though, I began to wonder: perhaps the difficulty comes from the fact that people aren’t used to thinking deeply about problems. I tested that idea by asking some people, and I found that they struggled in describing how they thought of problems. So what is a problem, anyway?

Here are two ways of thinking about problems that I’ve found useful.

1) A problem is “a difference between what is perceived and what is desired.” (Dewey, J. (1933), How We Think: A Restatement of the Relation of Reflective Thinking to the Educative Process) I first learned this definition of “problem” in a session led by Don Gray at the Amplifying Your Effecctiveness Conference a few years ago, and I believe it shows up in several places in Jerry Weinberg‘s work.

2) A problem is “an undesirable situation that is significant to and maybe solvable by some agent, though probably with some difficulty.” (G.F Smith, “Towards a Heuristic Theory of Problem Structuring”, quoted in Weick, Karl E. Sensemaking in Organizations. Sage Publications, Inc, 1995.)

Both definitions, I think, point to the same thing: problems are based on desires, perceptions, and situations. But both definitions have a minor problem for me, which is that they do not make explicit something I think to be very important: desires, perceptions, and situations are centred around people and context. Problems are subject to the Relative Rule:

For any abstract X, X is X to some person, at some time.

People often talk of problems (bugs, defects, issues, and so forth) as though they were attributes of a product or of a situation. But problems aren’t attributes as such. They’re relationships between some person(s), the product, and the system that includes them, and the situation that encompasses it all. A problem is a problem for some person, at some time. Differences, perceptions, and desires are like that too; they’re relative. If I were to expand out Dewey’s notion of “problem”, it would look like this:

A problem is a difference (according to some person) between what is perceived (by some person) and what is desired (by some person) (all at some point in time).

There are several implications here.

• The problem might be in the difference, or in the perception, or in the desire.


• Different people will have different notions of differences, different perceptions, different desires.


• A problem may also be influenced by timing; something that’s a problem today might not be a problem tomorrow, and something that isn’t a problem today might be a catastrophe tomorrow.


• Since, as testers, we’re in the business of finding potential problems, we must be alert to the enormous variety of people who may be affected by the product and the project.


• Our mission as testers typically includes the task of identifying possible problems. As such, we should resist the temptation to dismiss something as “not a problem”, because…


• Something that we might dismiss as a trivial problem for some people might be a serious problem for other people and…


• Something that we might see as a serious problem in our perception might appear as a trivial problem for other people so…


• We’re obliged as testers to identify a possible problem in terms of its most serious consequences for people who might matter to the product owner. However…


• We’re not in the business of deciding whether something is a problem or not, so for a given problem, our testing clients or the project owners decide on its ultimate significance, and on their response to it.

So the bottom line is that problems are slippery. A problem is not a thing in a product or in the world, but a relationship between some situation and some person.

Smith’s emphasis on an ability to solve a problem may matter too. People often accept things that they perceive as beyond anyone’s ability to solve. For example, I can’t do anything about a meteor hitting my computer in the next few minutes, so I’m not going to treat the possibility that it might happen as a problem. Yet again, I should be careful to suppress any of my own prejudices that nothing can be done with respect to a given problem. That’s a decision for those who build and those who own the product, since they may have information and more resources of which I am unaware.

One of the fundamental questions of testing is “Is there a problem here?” Oracles are media, in the McLuhan sense. Media are tools, extensions of ourselves that enhance, enable, accelerate, or intensify our capacity to do things. McLuhan pointed out, though, that media are agnostic about what they extend. If our concept of “problem” is limited, our oracles will extend and accelerate our ability to recognize a limited set of problems. In other words, with too narrow an idea of what a problem might be, oracles may only help us to recognize too narrow a set of possible problems.

So: one key to understanding and applying oracles skilfully is to recognize the richness and diversity of what we might mean by problem. What we’re testing is not simply source code or a collection of compiled object code files. We’re testing products or services that are systems, sets of things in meaningful relationship to each other. Those systems are related to other systems. Products don’t stand on their own. Every product is part of a system that includes people who build it, people who support it, people who maintain it, people who buy it, people who use it directly, and people who are affected by it. That’s an incomplete list of people who might experience problems related to the product or system. Try asking these questions:

• What are the elements of system that I’m testing?


• Who might have a relationship to that system, or to its elements?


• Who else might have a relationship to it?


• What desires might they have that are related to the system?


• What aspects of the system might provide value to those people by fulfilling their desires? What might threaten that value by dashing their desires?


• How might people perceive the system? What might they see, hear, smell, touch, taste, or feel as they use it or interact with it?


• What might influence, magnify, sharpen, or distort their perceptions?


• What differences might they experience between their perceptions and their desires?


• Could someone, or something—some change in the situation—help them to solve or otherwise deal with such differences?


• What might change in the system over time? How might people’s perceptions, desires, or notions of differences change over time?


• What ideas, tools, or conversations might help me to recognize perceptions, desires, and differences?

Reflecting on these questions periodically, even briefly, may help you to expand your notion of what a problem is. That in turn may extend your ability to use oracles to recognize potentional problems in the system you’re testing.

This blog post was inspired and sharpened by conversations with Anne-Marie Charrett, Peter Walen, and James Bach. Thanks to them.