Test Automation Planet

Tester Pride Without Bug Discovery

2012-02-08T18:09:00+00:00

You don’t need bugs to feel pride about the testing service you provide to your team. That was my initial message for my post, Avoid Trivial Bugs, Report What Works. I think I obscured said message by covering too many topics in that post so I’ll take a more focused stab at said topic.

Here is a list of things we (testers) can do to help feel pride in our testing when everything works and we have few to no bugs to report. Here we go…

Congratulate your programmers on a job well done. Encourage a small celebration. Encourage more of the same by asking what they did differently. Feel pride with them and be grateful to be a part of a successful team.
If you miss the ego boost that accompanies cool bug discovery, brag about your coolest, most creative, most technical test. You were sure the product would crash and burn but to your surprise, it held up. Sharing an impressive test is sometimes enough to show you’ve been busy.
Give more test reports (or start giving them). A test report is a method of summarizing your testing story. You did a lot. Share it.
Focus on how quickly whatever you tested has moved from development to production. Your manager may appreciate this even more than the knowledge that you found a bunch of bugs. Now you can test even more.
Start a count on a banner or webpage that indicates how many days your team has gone without bugs.
If the reason you didn’t find bugs is because you helped the programmer NOT write bugs from the beginning, then brag about it in your retrospective.
Perform a “self check”; ask another team member to see if they can find any bugs in your Feature. If they can’t find bugs, you can feel pride in your testing. If they can find bugs, you can feel pride in the guts it took to expose yourself to failure (and learn another test idea).

What additions can you think of?

Do Not Close This Window (Or Click The Back Button)

2012-02-07T16:48:47+00:00

Here’s a classic case of poor design and user experience. Most of us have seen something like it. It happened to my wife yesterday. It will happen to you again soon, probably.

You’re making an online payment for some product or service.

You press a button that says something like “Submit Payment”.

A web page appears that says something like “Your payment is being submitted. Please do not close this window or click the Back button on your browser.” And that’s all the page says.

The page stays on your screen forever—or until you wince and close the browser window despite the specific instructions on the screen.

Here are some questions that a tester could ask when presented with this design, or with this experience:

“Or else what?” “Please do not close this window or click the Back button on your browser.” Or else what? What Bad Thing might happen? What Good Thing might fail to happen? This should lead directly to…

“What if…?” What if the sequence of actions doesn’t go as planned? What if a conversation between a server and a client is interrupted? (Note: the connections between any two systems are at best somewhat reliable. If you believe otherwise, a travelling testing consultant has two words for you: hotel WiFi.) At what points might interruptions happen (quick answer: all of them.) How is the state of the conversation being managed? Have we considered interruptions in our design? Have we tested for them? How does the system handle and recover from delayed or interrupted transactions?

“What should the customer reasonably expect?” It’s not hard to imagine a good deal of variance in the performance of a system, especially when its end nodes might be dozens of network hops apart from each other. Still, how long should a customer reasonably expect the transaction to take? At what point might it make sense for the customer to bail out?

“How would the customer know when it’s time to bail out?” If you can put a message on the screen, and if you know how long it would be reasonable to wait before bailing out, should the customer have to look at her watch? Might a countdown timer be helpful?

“Is there another way?” Is there another way for the customer to see that the transaction has completed successfully, or has failed? Does your design and the message you display make that option clear?

“What emotions might come up?” How might a customer feel uncertain, confused, frustrated, annoyed, mystified, impatient, surprised, helpless—or confident, impressed, reassured, or delighted—by what she sees and experiences? How might we use those potential feelings to help us guide our search for problems?

“Who can help?” If the transaction fails, who can help the customer out? How does the customer get in touch with that person? Is there a means of contacting customer support on that “Please wait…” screen?

“What meta-information is available?” I’ve worked with companies that have said, “We can’t put a customer support telephone number on that screen; customer support would be swamped!” What does that statement tell you about the system, about people’s impressions of its reliability, and about risk?

“How do we raise awareness of problems?” When a transaction on our site fails or is subject to an unreasonable delay, how do we get to find out? Is someone alerted immediately? Are failures aggregated? Buried in a log file somewhere? Who looks for problems, and how often do they look? Who hears about problems? How does that information get relayed to the people who design, maintain, and update the system? How might that information—or parts of it—not get relayed to those people?

This last question is important. Its answer provides part of the explanation for the fact that, after fifteen years of Web commerce, we’re still seeing designs like the one that appears at the top of this post.

Looking for information about composed refactorings

2012-02-05T03:56:29+00:00

Even after all these years, I take too-big steps while coding. In less finicky languages (or finicky languages with good tool support), I can get away with it, but I can’t do that in C++ (a language I’ve been working in the past two weeks). So I’ve had to think more explicitly about taking baby steps. And that’s made me realize that there must be some refactoring literature out there that I’ve overlooked or forgotten. That literature would talk about composing small-scale refactorings into medium-scale changes. The only example I can think of is Joshua Kerievsky’s Refactoring to Patterns. I like that book a lot, but it’s about certain types of medium-scale refactorings (those that result in classic design patterns), and there are other, more mundane sorts.

Let me give an example. This past Friday, I was pairing with Edward Monical-Vuylsteke on what amounts to a desktop GUI app. Call the class in question Controller, as it’s roughly a Controller in the Model-View-Controller pattern. Part of what Controller did was observe what I’ll call a ValueTweaker UI widget. When the user asked for a change to the value (that is, tweaked some bit of UI), the Controller would observe that, cogitate about it a bit, and then perhaps tell an object deeper in the system to make the corresponding change in some hardware somewhere.

That was the state of things at the end of Story A. Story B asked us to add a second ValueTweaker to let the user change a different hardware value (of the same type as the first, but reachable via a wildly different low-level interface). The change from “one” to “two” was a pretty strong hint that the Controller should be split into three objects of two classes:

A ValueController that would handle only the task of mediating between a ValueTweaker on the UI and the hardware value described in story A.
A second ValueController that would do the same job for story B’s new value. The difference between the two ValueControllers wouldn’t be in their code but in which lower-level objects they’d be connected to.
A simplified Controller that would continue to do whatever else the original Controller had done.

The question is: how to split one class into two? The way we did it was to follow these steps (if I remember right):

Create an empty ValueController class with an empty ValueController test suite. Have Controller be its subclass. All the Controller tests still pass.
One by one, move the Controller tests that are about controlling values up into the ValueController test suite. Move the code to make them pass.
When that’s done, we still have a Controller object that does everything it used to do. We also have an end-to-end test that checks that a tweak of the UI propagates down into the system to make a change and also propagates back up to the UI to show that the change happened.
Change the end-to-end test so that it no longer refers to Controller but only to ValueController.
Change the object that wires up this whole subsystem so that it (1) creates both a Controller and ValueController object, (2) connects the ValueController to the ValueTweaker and to the appropriate object down toward the hardware, and (3) continues to connect the Controller to the objects that don’t have anything to do with changing the tweakable value. Confirm (manually) that both tweaking and the remaining Controller end-to-end behaviors work.
Since it no longer uses its superclass’s behavior, change Controller so that it’s no longer a subclass of ValueController.
Change the wire-up-the-whole-subsystem object so that it also makes story B’s vertical slice (new ValueTweaker, new ValueController, new connection to the hardware). Confirm manually.

I consider that a medium-scale refactoring. It’s not something that even a smart IDE can do for you in one safe step, but it’s still something that (even in C++!) can easily be finished in less than an afternoon.

So: where are such medium-scale refactorings documented?

PHP Summit in München

2012-02-03T04:15:29+00:00

This blog posting is in German as the event it relates to is German-only.
Sorry for the inconvenience.

Bei allen PHP-Themen zählt nichts mehr als die Praxis. Deshalb bieten wir unsere Power-Workshops interaktiv und mit intensivem Praxisbezug an. Über die behandelten Themen entscheiden die Teilnehmer mit ihren konkreten Fragen. Anstelle von Frontalunterricht erleben sie die Entwicklung von neuem Code unmittelbar. Mit Augenzwinkern und Spaß erläutern Sebastian Bergmann, Arne Blankerts und Stefan Priebsch Entwicklungsmethoden und Tools und stellen Trends und Konzepte vor.

Der nächste PHP Summit findet im März in München statt. Und das sind die Workshops:

Update PHP: Neue Features und Technologien nutzen

Workshop von Sebastian Bergmann

PHP 5.3 und PHP 5.4 überzeugen im Programmieralltag durch eine Fülle relevanter Vereinfachungen. Lernen Sie die innovativen Features und geschickte Einsatzmöglichkeiten der neuen Versionen kennen. Entdecken Sie das Lösungspotenzial aktueller Technologien aus dem PHP-Umfeld (memcached, ZeroMQ …) für Ihre Fragestellungen.

Weg mit Strubbelcode: identifizieren – verbessern – vermeiden

Workshop von Sebastian Bergmann

Früher oder später wird unsauber geschriebener Code zum Ärgernis. Nicht nur für den, der ihn warten muss. Änderungen und Erweiterungen können im Extremfall den Code unwirtschaftlich machen. Lernen Sie schlechten Code durch statische Codeanalyse aufzufinden und in test- und wartbaren Code umzuschreiben. Lernen Sie mithilfe der SOLID-Prinzipien, nachhaltig wartbaren Code zu schreiben.

Best Practices – aus dem Alltag für den Alltag

Workshop von Arne Blankerts

Natürlich könnte man das Rad jeden Tag neu erfinden. Meist fehlt dafür die Zeit, Spaß macht es auch nicht und Fehler können sich so immer wieder an denselben Stellen einschleichen. Für die vielen alltäglichen Probleme, die nur geringfügig von schon vorhandenen Lösungen abweichen, gibt es clevere Ansätze, die das Leben leichter machen. In einer komplett vom Auditorium gesteuerten Live Session zeigt der Workshop dafür programmatische Konzepte und lädt zur Diskussion über Tools und klassische Fragestellungen ein.

Auf Fehler sicher vorbereitet sein

Workshop von Arne Blankerts

Programme und Webseiten enthalten Fehler. Immer. Sie werden sichtbar, wenn Benutzer falsche, ungültige oder unerwartete Eingaben machen, der Zugriff auf die Datenbank plötzlich unmöglich ist oder die Festplatte überläuft. Um solche und andere Probleme sicher abzufangen, gibt es verschiedene Ansätze, die hier vorgestellt und diskutiert werden. Wie man Exceptions richtig anwendet, warum ein eigener Error Handler hilfreich ist und dass Debugging viel mit Sicherheit zu tun hat, vermittelt der Workshop lebendig und anschaulich.

Objektorientierte Programmierung (OOP) in PHP I: Basiswissen

Workshop von Stefan Priebsch

Der Workshop klärt die Frage, was es mit der objektorientierten Programmierung eigentlich auf sich hat. Jenseits der bekannten Standardbeispiele führt er praktisch in die OOP mit PHP ein. Dabei werden neben den Grundlagen und zentralen Prinzipien der OOP interessante Features beispielsweise aus der Standard PHP Library (SPL) vorgestellt und ihr sinnvoller Praxiseinsatz vorgeführt. Neben Faktenwissen wird eine Denkweise vermittelt, die es erlaubt, unnötig komplizierte Ansätze von vornherein zu umgehen.

Objektorientierte Programmierung (OOP) in PHP II: Aufbauwissen

Workshop von Stefan Priebsch

Der Workshop zeigt Teilnehmern, die mit den Grundlagen der OOP vertraut sind, fortgeschrittene Techniken wie Dependency Injection, abstrakte Klassen, Interfaces und Best Practices für erfolgreiche OOP. Eine Live-Coding-Session macht den Einsatz der vorgestellten Techniken in der Praxis erfahrbar. Außer Faktenwissen zeigt der Workshop, dass gute Lösungen einfache Lösungen sind: Einfache Objekte sind leichter wieder zu verwenden und vermeiden Fehler.

PHP-Anwendungen testen: Basiswissen

Workshop von Sebastian Bergmann

Der Workshop vermittelt grundlegende Kenntnisse und Fähigkeiten im Einsatz von PHPUnit bei Unit Tests, Datenbank-Interaktionstests, Edge-to-Edge- Tests und End-to-End-Tests. Sie lernen alles, was Sie über das Schreiben, Ausführen und Organisieren von Unit Tests mit PHPUnit beherrschen müssen.

PHP-Anwendungen testen: Aufbauwissen

Workshop von Sebastian Bergmann

PHPUnit clever einsetzen: Die Teilnehmer lernen die besten Praktiken beim Einsatz von PHPUnit und erprobte Strategien bei der Einführung von Testmaßnahmen an vorhandener Software kennen. Anhand zahlreicher Beispiele entwickeln Sie einen Blick für schlechte Tests und lernen, sie zu verbessern, schwer testbaren Code zu identifizieren und Legacy-Code Schritt für Schritt testbar zu machen.

Geheimtipp XML: Die smarte Art der Anwendung

Workshop von Arne Blankerts

XML ist das Standardformat für den Datenaustausch im Web und verbirgt sich in vielen, für Endanwender oft nicht sichtbaren Dateiformaten (z. B. aktuelle MS-Office-Versionen) und Formen. Der Workshop zeigt, dass XML richtig Spaß machen kann, wenn die übermittelten Daten validiert werden und der Zugriff mit effiziente APIs erfolgt. In der Live Session erweitern wir das DOM-API von PHP um eigene Methoden und unterstützen die Lokalisierung durch PHP. Eigene XSD Schemas zur Validierung werden definiert und der Einsatz von XPath als Abfragesprache vorgeführt. Vermittelt wird auch der Einblick in eine wirklich effektive Fehlerbehandlung.

Neu: Tuning für Web-Anwendungen

Workshop von Arne Blankerts

Wer einen ungeschickten Weg zur Lösung beschreitet, handelt sich Probleme ein, die er gar nicht haben möchte. In der Regel geht das zulasten des Tempos und der Skalierbarkeit. Wie man solche Schwachpunkte in gängigen Aufbauten und Architekturen vorab erkennt und durch bessere Konzepte von vornherein vermeiden kann, zeigt dieser Workshop an zahlreichen Praxisbeispielen. Eigene Denk- und Herangehensweisen können überprüft und modifiziert werden.

Entwurfsmuster I: Die wichtigsten Standards

Workshop von Stefan Priebsch

Warum das Rad jedes Mal neu erfinden? Die Teilnehmer lernen in diesem Workshop wichtige Entwurfsmuster kennen, die sich in PHP-Webanwendungen besonders bewährt haben. Häufig in der Praxis auftretende Schwierigkeiten werden vorgeführt und Lösungen werden live programmiert. Vor- und Nachteile sowie Einsatzmöglichkeiten und typische Fehler bei der Anwendung von Standardmustern werden ausführlich erläutert und diskutiert.

Entwurfsmuster II: Integration anspruchsvoller Patterns

Workshop von Stefan Priebsch

Der Workshop behandelt Themenstellungen wie temporale Patterns, das Speichern von Objekten in relationale und NoSQL-Datenbanken oder Enterprise Integration Patterns mit Schwerpunkten, die von den Teilnehmern frei festgelegt werden. Eine Reihe typischer Probleme wird vorgeführt und analysiert. Die zu ihrer Lösung eingesetzten (wenig bekannten) Entwurfsmuster vermitteln den Teilnehmern fortgeschrittene Kenntnisse, die im Alltag wertvoll sind.

Moderne Versionskontrolle mit Git

Workshop von Sebastian Bergmann

Git ist ein mächtiges Versionsverwaltungssystem, mit dem Sie alle Bestandteile Ihrer Software durch alle Änderungen und Versionen zuverlässig verfolgen können. Damit schaffen Sie die Grundlage für die kontinuierliche Integration der Software. Der Workshop vermittelt Grundlagen für den Einsatz von Git, Best Practices und Prozesse für Entwicklung, Release-Management und Deployment, die Ihre Teamarbeit spürbar effektiver machen.

Kontinuierliche Integration mit Jenkins

Workshop von Sebastian Bergmann

Wer die Qualität seiner Software während der Entwicklung und im Betrieb messen und kontrollieren will, profitiert vom Einrichten einer Umgebung, in der PHP-Projekte kontinuierlich integriert werden können. Mit der Kombination dynamischer und statischer Testverfahren automatisieren Sie wiederkehrende Aufgaben, sind über die Softwarequalität auf dem Laufenden und minimieren Projektrisiken erfolgreich. Der Workshop vermittelt die Grundlagen der kontinuierlichen Integration und Inspektion von PHP-Software. Sie üben Installation, Konfiguration und Betrieb von Jenkins für PHP-Projekte.

Am besten mit dem Schlimmsten rechnen!

Workshop von Arne Blankerts

In Zeiten ständiger Einbruchsversuche durch gelangweilte Anwender ergibt es Sinn, wirklich immer mit Angriffen aus dem Internet zu rechnen. Auch vermeintlich sichere Strukturen bieten versierten Hackern oft keinen nennenswerten Widerstand. Verantwortungsbewusste Entwickler müssen wissen, wie man der eigenen Anwendung oder Infrastruktur schaden könnte. Der Workshop vermittelt grundlegende Zusammenhänge und zeigt in einer Live Session, was Begriffe wie XSS, CSRF oder SQL-Injection bedeuten, wie man Angreifern das Leben schwer macht und wirksame Lösungen implementiert.

Attacke! Was Angriffe im Web erfolgreich macht

Workshop von Arne Blankerts

Anwendungen sind im Internet permanent Angreifern ausgesetzt. Im Workshop wechseln die Teilnehmer die Seiten und greifen selbst eine Demo-Blackbox-Anwendung an. Sie lernen, wie Angreifer Informationen sammeln, Sicherheitslücken ausnutzen und Server übernehmen. Vermittelt werden Methoden, um Lücken zu finden, sie zu schließen und Angriffe abzufangen. Der abschließende Blick in den Quellcode sensibilisiert für Schwachstellen und zeigt, wie man sie behebt.

Framework: Basics in drei Stunden

Workshop von Stefan Priebsch

Grundlegende Fragen nach der Struktur einer Webanwendung und dem Schreiben von Code, der test-, wart- und erweiterbar ist, werden an einem praktischen Beispiel beantwortet. Wichtige Entwurfsmuster und Best Practices werden vorgeführt. Feedback und Fragen der Teilnehmenden steuern den Verlauf der Session. Ziel ist ein besseres Verständnis für die wesentlichen Konzepte von Frameworks.

Warum der Turm in Pisa schief steht

Workshop von Stefan Priebsch

Nichtfunktionale Aspekte werden in ihrer Bedeutung für technische Entscheidungen oft unterschätzt oder ganz übersehen. Je komplexer die Fragestellungen desto wahrscheinlicher ist es, dass aus diesem Wegschauen Probleme resultieren, die später nicht leicht einzufangen sind. Wer sich vor der Entwicklung Gedanken zur Architektur macht, löst heute Probleme von morgen. Der Workshop zeigt Architekturmuster für Webanwendungen und geht auf aktuelle Trends wie Distributed Caching, Message Queues und NoSQL ein. Es werden erprobte Bausteine vorgestellt, mit denen hochperformante und skalierbare Webanwendungen in PHP entwickelt werden können.

Der PHP Summit bietet effizientes Lernen in entspannter Atmosphäre. Profitieren Sie von dieser einzigartig intensiven Form der Informationsvermittlung mit einem unschlagbaren Preis-Leistungs-Verhältnis und reservieren Sie sich Ihren Platz noch heute!

Signing Off

2012-02-02T14:13:01+00:00

By James Whittaker

This will be my last post on this blog. Tomorrow is my last day at Google. It was a great ride and a great pleasure to work alongside such brilliant engineers. I will hand over this blog to another test director and then find a new place for whatever blogging I do in the future.

Follow me on Twitter (@docjamesw) if you are interested in where I land and to find my next blog outlet.

Peace.

A Tool's Tale

2012-02-01T06:30:00+00:00

When Noah Sussman asked me to give a Code as Craft Technology Talk last week when I was consulting for Etsy I immediately said yes. However, I was a bit surprised when the talk was announced under the title "An Evening with Sebastian Bergmann". When I read that title the first time, it was just minutes after Arne, Stefan and I had talked about one of our favourite scenes from "A Knight's Tale":

Chaucer: I'm a writer.

Wat: A what?

Chaucer: A wha- a what? A writer. You know, I write, with ink, and parchment. Geoffrey Chaucer's the name, writing's the game. You've probably read my book? The Book of the Duchess? No? Well, it was allegorical.

Roland: Well, we won't hold that against you, that's for every man to decide for himself.

Just like with a talk that I gave last year, I suddenly had a chain of associations in my head that I just had to follow. And down the rabbit hole I went once more ...

Hi! I have no idea why this talk is titled "An Evening with Sebastian Bergmann". I hope that this evening will turn into an interesting discussion about PHPUnit and all things testing. To break the ice, and to not appear completely unprepared, I came up with the following slides ...

So I recently talked with my friends Arne and Stefan about the movie "A Knight's Tale". Somehow the idea stuck in my head that a variation of that title might be a good idea for a talk of mine. After dismissing "A Fool's Tale" I arrived at "A Tool's Tale". So that's what we're stuck with and what I am going to try right now ...

What I just did was a variation on how Geoffrey Chaucer introduces himself in the movie.

Some time in 2001 I started to work on PHPUnit because I wanted to have something like JUnit for PHP. The initial "port" was completed within one weekend. The code was ugly because I had to emulate exceptions which PHP 4 did not have. On November 27th 2001 I checked the code into cvs.php.net.

A lot has changed since 2001. Hopefully not very many developers are still stuck with PHP 4. As of PHPUnit 2.0, which was released on July 14 2004, the day after PHP 5.0.0 was released, PHP 5 is required to run PHPUnit.

PHPUnit is neither the only testing framework nor the only quality assurance tool for PHP. Over the last years a nice ecosystem of static analysis tools started to grow.

I get this question a lot: why do only Germans work on tools that tell me that my code is bad? I do not have an answer for this, sorry. But the statement is also not true: PHP_CodeSniffer is not developed by a German.

PHP has changed and so the have the tools we use to write and maintain code. In 2006 the PHPUnit code was migrated from CVS to Subversion ...

... and in December 2009 from Subversion to Git and GitHub.

PHPUnit is no longer one big monolithic package. It has been refactored to components that are being reused by other testing frameworks, for instance. Unfortunately, this refactoring was not without problems.

PHPUnit gets more and more convenience functionality. Here are a couple of examples:

Tests can be written, both unintentionally and intentionally, in bad ways. Bad tests can lie. They can give you a false sense of security by reporting that something is tested when it really is not.

This is why I started iplementing coutermeasures against bad tests such as the (ultimately useless but still interesting) assertion counting ...

... as well as the strict execution mode.

There is code in PHPUnit that I am not proud of. I am trying to make the world a better place by eliminating one singleton at a time, for instance. Unfortunately, cleaning up code sometimes breaks things. Contrary to what a popular opinion on Twitter is, I do not like breaking backwards compatibility in PHPUnit and try really hard to avoid it.

This is a "good" example of such a backwards compatibility breakage. It happened in PHPUnit 3.6 because I eliminated a Singleton. Had I known that many developers used this API (instead of the XML configuration file) to set up a code coverage blacklist or whitelist I would probably not have done the change. If more people would have tested the release candidates leading up to PHPUnit 3.6 they could have told me ... oh well.

At this point the "Evening with Sebastian Bergmann" turned into the interesting discussion I had hoped for.

What Exploratory Testing Is Not (Part 5): Undocumented Testing

2012-01-30T14:25:42+00:00

This week I had the great misfortune of reading yet another article which makes the false and ridiculous claim that exploratory testing is “undocumented”. After years and years of plenty of people talking about and writing about and practicing excellent documentation as part of an exploratory testing approach, it’s depressing to see that there are still people shovelling fresh manure onto a pile that should have been carted off years ago.

Like the other approaches to test activities that have been discussed in this series (“touring“, “after-everything-else“, “tool-free“, and “quick testing“), “documented vs. undocumented” is in a category orthogonal to “exploratory vs. scripted”. True: usually scripted activities are performed by some agency following a set of instructions that has been written down somewhere. But we could choose to think of “scripted” in a slightly different and more expansive way, as “prescriptive”, or “mimeomorphic“. A scripted activity, in this sense, is one for which the actions to be performed have been established in advance, and the choices of the actions are not determined by the agency performing them. In that sense, a cook at McDonalds doesn’t read a script as he prepares your burger, but the preparation of a McDonald’s burger is a highly scripted activity.

Thus any kind of testing can be heavily documented or completely undocumented. A thoroughly documented test might be highly exploratory in nature, or it might be highly scripted.

In the Rapid Software Testing class, James Bach and I point out that when someone says “that should be documented”, what they’re really saying is “that should be documented if and how and when it serves our purposes.” So, let’s start by looking at the “when”.

When we question anything in order to evaluate it, there are moments in the process in which we might choose to record ideas or actions. I’ve broken these down into three basic categories that I hope you find helpful:

Before

During

After

There are “before”, “during”, and “after” moments with respect to any test activity, whether it’s a part of test design, test execution, result interpretation, or learning. Again, a hallmark of exploratory testing is the tester’s freedom and responsibility to optimize the value of the work as it’s happening. That means that when it’s important to record something, the tester is not only welcome but encouraged to

pick up a pen
take a screen shot
launch a session of Rapid Reporter
create or update a mind map
fire up a screen recorder
initiate logging (if it doesn’t start by default on the product you’re testing—and if logging isn’t available, you might consider identifying that as a testability problem and a related product and project risk)
sketch out a flowchart diagram
type notes into a private or shared repository
add to a table of data in Excel
fire off a note to a programmer or a product owner

and that’s an incomplete list. But they’re all forms of documentation.

Freedom to document at will should also mean that the tester is free to refrain from documenting something when the documentation doesn’t add value. At the same time, the tester is responsible and accountable for that decision. In Rapid Testing, we recommend writing down (or saving, or illustrating) only the things that are necessary or valuable to the project, and only when the value of doing so exceeds the cost. This doesn’t mean no documentation; it means the most informative yet fastest and least expensive documentation that completely fulfils the testing mission. Integrating that with testing work leads, we hold, to excellent testing—but it takes practice and skill.

For most test activities, it’s possible to relay information to other people orally, or even sometimes by allowing people to observe our behaviour. (At the beginning of the Rapid Testing class, I sometimes silently hold aloft a 5″ x 8″ index card in landscape orientation. I fold it in half along the horizontal axis, and write my first name on one side using a coloured marker. Everyone in the class mimics my actions. Without a single word of instruction being given or questions being asked, either verbally or in writing, the mission has been accomplished: each person now has a tent card in front of him.)

There’s a potential risk associated with an exploratory approach: that the tester might fail to document something important. In that case, we do what skilled people do with risk: we manage it. James Bach talks at length about managing exploratory testing sessions here. Producing appropriate documentation is partly a technical process, but the technical considerations are dominated by business imperatives: cost, value, and risk. There are also social considerations, too. The tester, the test lead, the test manager, the programmers, other managers, and the product owner determine collaboratively what’s important to document and what’s not so important with respect to the current testing mission. In an exploratory approach, we’re more likely to be emphasizing the discovery of new information. So we’re less likely to spend time on documenting what we will do, more likely to document what we are doing and what we have done. We could do a good deal of preparatory reading and writing, even in an exploratory approach—but we realize that there’s an ever-increasing risk that new discoveries will undermine the worth of what we write ahead of time.

That leads directly to “our purposes”, the task that we want to accomplish when documenting something. Just as testing itself has many possible missions, so too does test documentation. Here’s a decidedly non-exhaustive list, prepared over a couple of minutes:

to express testing strategy and tactics for an entire project, or for projects in general

to keep a set of personal notes to help structure a debriefing conversation

to outline testing activities for a test cycle

to report on activities during testing execution

to outline attributes of a particular quality criterion

to catalogue ideas about risk

to describe test coverage

to account for the work that we’ve done

to program a machine to perform a given set of actions

to alert people to potential problems in the product

to guide a tester’s actions over a test session

to identify structures in the application or service

to provide a description of how to use a particular test tool that we’ve crafted

to describe the tester’s role, skills, and qualifications

to explain business rules to someone else on the team

to outline scenarios in which the product might be used or tested

to identify, for a tester, a specific, explicit sequence of actions to perform, input to provide, and observations to make

That last item is the classic form of highly scripted testing, and that kind of documentation is usually absent from exploratory testing. Even so, a tester can take an exploratory approach using a script as a point of departure or as a reference, just as you might use a trail map to help guide an off-trail hike (among other things, you might want to discover shortcuts or avoid the usual pathways). So when someone says that “exploratory testing is undocumented”, I hear them saying something else. I hear them saying, “I only understand one form of test documentation, and I’ve successfully ignored every other approach to it or purpose for it.”

If you look in the appendices for the Rapid Software Testing class (you can find a .PDF at http://www.satisfice.com/rst-appendices.pdf), you’ll see a large number of examples of documentation that are entirely consistent with an exploratory approach. That’s just one source. For each item in my partial list above, here’s a partial list of approaches, examples, and tools.

Testing strategy and tactics for an entire project, or for projects in general.
Look at the Satisfice Heuristic Test Strategy Model and the Context Model for Heuristic Test Planning (these also appear in the RST Appendices).

An outline of testing activities for a test cycle.
Look at the General Functionality and Stability Test Procedure for Certifed for Microsoft Windows Logo. See also the OWL Quality Plan (and the Risk and Task Correlation) in the RST Appendices.

Keeping a set of personal notes to help structure a debriefing or other conversation.
See the “Beans ‘R Us Test Report” in the RST Appendices; or see my notes on testing an in-flight entertainment system which I did for fun on a flight from India to Amsterdam.

Recording activities and ideas during test execution
A video camera or a screen recording tool can capture the specific actions of a tester for later playback and review. Well-designed log files may also provide a kind of retrospective record about what was testing. Still neither of these provide insight into the tester’s mind. Recorded narration or conversation can do that; tools like BB Test Assistant, Camtasia, or Morae can help. The classic approach, of course, is to take notes. Have a look at my presentation, “An Exploratory Tester’s Notebook“, which has examples of freestyle notes taken during an impromptu testing session, and detailed, annotated examples of Session-Based Test Management sessions. Shmuel Gerson’s Rapid Reporter and Jonathan Kohl’s Session Tester are tools oriented towards taking notes (and, in the former case, including screen captures) of testing sessions on the fly.

Outlining many attributes of a particular quality criterion
See “Heuristics of Software Testability” in the RST Appendices for one example.

Cataloguing ideas about risk
Several examples of this in the RST Appendices, most extensively in the “Deployment Planning and Risk Analysis” example. You’ll also find an “Install Risk Catalog”; “The Risk of Incompatibility”; the Risk vs. Tasks section in the “OWL Quality Plan”; the “Y2K Compliance Report”; “Round Results Risk A”, which shows a mapping of Risk Areas vs. Test Strategy and Tasks.

Describing or outlining test coverage
A mapping establishes or illustrates relationships between things. We can use any of these to help us think about test coverage. In testing, a map might look like a road map, but it might also look like a list, a chart, a table, or a pile of stories. These can be constructed before, after, or during a given test activity, with the goal of covering the map with tests, or using testing to extend the map. I catalogued several ways of thinking about coverage and reporting on it, in three articles Got You Covered, Cover or Discover, and A Map By Any Other Name. Several examples of lightweight coverage outlines can be found in the RST Appendices (“Putt Putt Saves the Zoo”, “Table Formatting Test Notes”, There are also coverage ideas incorporated into the Apollo mission notes that we’ve titled “Guideword Heuristics for Astronauts”).

Accounting for testing work that we’ve done.
See Session-Based Test Management, and see “An Exploratory Tester’s Notebook“. Darren McMillan provides excellent examples of annotated mind maps; scroll down to the section headed “Session Reports”, and continue through “Simplifying feedback to management” and “Simplifying feedback to groups”. A forthcoming article, written by me, shows how a senior test manager tracks testing sessions at a half-day granularity level.

Programming a machine to help you to explore
See all manner of books on programming, both references and cookbooks, but for testers in particular, have a look at Brian Marick’s Everyday Scripting with Ruby. Check out Pete Houghton’s splendid examples of exploratory test automation that begin here. Cem Kaner (often in collaboration with Doug Hoffman) write extensively about automation-assisted exploratory testing; an example is here.

Alerting people to potential problems in the product
In general, bug reporting systems provide one way to handle the task of recording and reporting problems in the product. James Bach provides an example of a report that he provided to a client (along with a more informal account of the session).

Guiding a tester’s actions over a test session
Guiding a tester involves skills like chartering and checklisting. Start with the documentation on Session Based Test Management (http://www.satisfice.com/sbtm). Selena Delesie has produced an excellent blog post on chartering exploratory testing sessions. The title of Cem Kaner’s presentation at CAST 2008, The Value of Checklists and the Danger of Scripts: What legal training suggests for testers describes the content perfectly. Michael Hunter’s You Are Not Done Yet lists can be used and adapted to your context as a set of checklists.

To identify structures in the application or service
The “Product Elements” section in the Heuristic Test Strategy Model provides a kind of framework for documenting product structures. In the RST Appendices, the test notes for “Putt Putt Saves the Zoo” and “Diskmapper”, and the “OWL Quality Plan” provide examples of identifying several different structures in the programs under test. Mind mapping provides a means of describing and illustrating structures, too; see Darren McMillan’s examples here and here. Ruud Cox and Ru Cindrea used a mind map of product elements to help win the Best Bug Report award in the Test Lab at EuroSTAR 2011. I’ve created a list of structures that support exploratory testing, and many of these are related to structures in the product.

Providing a description of how to use a particular test tool that we’ve crafted
While working at a bank, I developed (in Excel and VBA) a tool that could be used as an oracle and as a way of recording test results. (Thanks to non-disclosure agreements, I can describe these, but cannot provide examples.) When I left the project, I was obliged to document my work. I didn’t work on the assumption that anyone off the street would be reading the document. Instead, I presumed that anyone assigned to that testing job and to using that tool, would have the rapid learning skill to explore the tool, the product, and the business domain in a mutually supportive way. So I crafted documentation that was intended to tell testers just enough to get them exploring.

Explaining business rules to someone else on the team
I did include documentation for novices of one kind: within the documentation for that testing tool, I included a general description of how foreign exchange transactions worked from the bank’s perspective, and how appropriate accounts got credited and debited. I had learned this by reverse-engineering use cases and consulting with the local business analyst. I summarized it with a two-page document written in simple, direct language, referring disrectly to the simpler use cases and explaining the more confusing bits in more detail. For those whose learning style was oriented toward code, I also described the tables and array formulas that applied the business rules.

Outlining scenarios in which the product might be used or tested
I discuss some issues about scenarios here—why they’re important, and why it’s important to keep them open-ended and open to interpretation. It’s more important to record than to prescribe, since in a good scenario, you’ll observe and discover much more than you’ve articulated in advance. Cem Kaner gives ideas on how to produce scenarios; Hans Buwalda presents examples of soap opera testing.

Identifying required tester skill
People with skill don’t need prescriptive documentation for every little thing. Responsible managers identify the skills needed to test, and who commit to employing people who either have those skills or can develop them quickly. James Bach eliminated 50 pages of otiose documentation with two paragraphs. (Otiose is a marvelous word; it’s fun to look it up in a thesaurus.)

Identifying, for a tester, a particular explicit sequence of actions to perform, input to provide, and observations to make.
Again, a document that attempts to specify exactly what a tester should do is the hallmark of scripted testing. James Bach articulates a paradox that has not yet been noted clearly in our craft: in order to perform a scripted test well, you need signficant amounts of skill and tacit knowledge (and you also need to ignore the script on occasion, and you need to know when those occasions are). There’s another interesting issue here: preparing such documents usually depends on exploratory activity. There’s no script to tell you how to write a script. (You might argue there’s one exception. You can follow this script to write a test script: take each line of a requirements document, and add the words “Verify that” to the beginning of each line.)

Now, just as you can perform testing badly using any approach, you can perform exploratory testing and document it inappropriately, either by under-documenting it OR over-documenting it using any of the kinds of documentation above. But, as this document shows, the notion that exploratory testing is by its nature undocumented is not only ignorant, but aggressively ignorant about both testing and documentation. Whenever you see someone claim that exploratory testing is undocumented, I’d ask you to help by setting the record straight. Feel free to refer to this blog post, if you find it helpful; also, please point me to other exemplars of excellent documentation that are consistent with exploratory approaches. If we all work together, we can bury this myth, while providing excellent records and reports for our clients.

We’ve Hired My Sister– Website Will Improve Soon!

2012-01-30T09:59:12+00:00

Satisfice, Inc. welcomes my sister Erika Good to our company. She’s our new Director of Marketing and Communication.

Erika is two years older than me. When I first got kicked out of my home, in 1981, I was a freshman in high school. Erika and I would meet each morning (on the mornings I didn’t oversleep or otherwise skip school) in the school library to check in. Later, it was Erika who alerted my father that I had started drinking, which is why I have only been drunk twice in my life– both when I was 15 years old– and why I barely ever drink now. Erika and I have always had a bond. Now she and my wife Lenore will share the burden of keeping me productive when I’m on the road and helping me organize and publish all my stuff.

Erika is quiet, polite, and unassuming (the opposite of me, in other words); a dedicated housewife and mother of two grown boys. She’s also an expert marksman (she competed as a member the Vermont National Guard pistol team— wait, she’s saying it was the rifle team. How did I not know that?), an artist, an author as well as a private pilot and webmaster, who also helps run her husband’s property business… Geez, I didn’t realize how much she’s done until I had to write that sentence!

One immediate effect she will have is on our long neglected website. She’s already put up the announcement of the Rapid Testing Intensive, and added a video page. She’s helping me create a brand new bibliography and resource list. She’s going to add tags to all the blog entries and get a lot more content published that has been sitting on my hard drive.

A Consulting Session With an Unfortunate Victim

2012-01-28T03:50:02+00:00

This poor girl from Ghana inherited many kilograms of gold and can’t get at it because she can’t pay the back due rent on the gold storage unit. She has reached out to me, a total stranger, for help. I use my testing skills to help her understand that she’s being scammed. But you know, sometimes it’s hard to help people see the truth when fantasy seems less painful…

A stranger approaches on Skype…

[12:29:22 AM] Irene Kusi: Hello

[12:29:33 AM] James Bach: hi

[12:29:44 AM] Irene Kusi: How are you doing ?

[12:29:54 AM] James Bach: Why are you contacting me?

[12:30:10 AM] Irene Kusi: I am very sorry about that and sorry again if i disturb you. I am Irene Kusi from the UK but i am now in Ghana and i would like to know more about you.

[12:30:51 AM] James Bach: Why are you contacting ME, though?

[12:31:39 AM] Irene Kusi: Like i just told you .sorry if i disturb you.

[12:32:06 AM] James Bach: You haven’t told me why you came to me instead of someone else. Is there any reason?

[12:32:10 AM] Irene Kusi: Can we get to know each other personally? What do you mean ?

[12:32:27 AM] James Bach: Look, strangers contact me often for business purposes. That’s normal. I’m trying to figure out if this is for business. Usually people tell me why they are contacting me.

[12:33:08 AM] Irene Kusi: Yes is not strictly business. I just would like to get to know you

[12:33:18 AM] James Bach: So, you don’t know who I am.

[12:33:54 AM] Irene Kusi: Yes i really dont know you but i found your profile on skype and decided to get to know you if i can work with you.

[12:34:23 AM] James Bach: You just chose me at random? Seems like a lot of people from Ghana are doing that.

[12:35:13 AM] Irene Kusi: Why have some people from ghana done something like that to you ?

[12:35:18 AM] James Bach: Yes you are not the first person from Ghana to randomly contact me

[12:35:40 AM] Irene Kusi: And what did they want ??

[12:35:51 AM] James Bach: They wouldn’t tell me… Just like you aren’t telling me.

(She cuts the small-talk and tells me her problem)

[12:36:30 AM] Irene Kusi: My main motive here is to find someone who will help me inherit my property thats why i am on skype .

[12:36:40 AM] James Bach: oh, a 419 thing. I understand

[12:38:27 AM] Irene Kusi: Oh who said is a 419 thing

[12:39:03 AM] James Bach: I did, “oh, a 419 thing” See, that’s where I said it.

[12:39:23 AM] Irene Kusi: what shows is a 419 thing

[12:39:41 AM] James Bach: You said “My main motive here is to find someone who will help me inherit my property thats why i am on skype.” If you haven’t heard of 419, that’s okay.

[12:40:54 AM] Irene Kusi: Ok

[12:41:12 AM] James Bach: Usually it’s Nigerians who need this kind of help. I didn’t know it was happening in Ghana, too.

[12:41:34 AM] Irene Kusi: Ok .so are you going to help me or not ?

[12:41:46 AM] James Bach: What do you need?

[12:42:06 AM] Irene Kusi: I will like you to help me pay with the bills so that i can get the gold

[12:43:07 AM] James Bach: Oh, you want money. Why didn’t you say so at the start?

[12:43:20 AM] Irene Kusi: I dont know why

(With my tester skills, I immediately comprehend what’s going on: she’s too innocent; she sees only the good in people)

[12:44:26 AM] James Bach: Okay, but, why would you trust me? You don’t know me. I might be a bad guy.

[12:45:11 AM] Irene Kusi: Nothing i have just gotten to know that you a very giid guy and i will trsut you .

[12:46:39 AM] James Bach: I think you have to be careful, though. There are many people in the world who may try to take advantage of you. Especially if you have inherited gold. Let me ask you this: If I help you, will you share part of the gold with me?

[12:47:51 AM] Irene Kusi: I will do exactly that not just half but i will let you get the whole gold that is i will bring it to the USA and then we will sell it there and share the money. But before we share the money i will give you 5kilograms of the gold for free just for helping me get it . [Editors note: that's about $300,000]

[12:48:30 AM] James Bach: See? You were just about to trust me and then I tried to take your gold! Irene, you are WAY too trusting. People will want to take from you and not give you anything.

[12:49:14 AM] Irene Kusi: Oh so will you do that to me ?

[12:49:28 AM] James Bach: I just proved that you would LET me if I TRIED to. That was a test. I do testing for a living. And you showed that you are too innocent and good. This is a hard world, full of cheaters.

[12:50:07 AM] Irene Kusi: Oh okay thank you so much for telling me this .

[12:50:13 AM] James Bach: They may try to contact you and fool you into giving up your money. So be more careful.

[12:50:26 AM] Irene Kusi: Thank you

(Now I go into consulting mode… First, learn about the problem. Often the real problem is not the one they come to you with.)

[12:50:31 AM] James Bach: Why don’t you keep your money? You will need it.

[12:51:16 AM] Irene Kusi: Ok so are you going to help me pay for the bills so that i get the gold ??

[12:51:46 AM] James Bach: I thought you said the gold belonged to you.

[12:51:52 AM] Irene Kusi: Yes

[12:52:04 AM] James Bach: Then you have much much more money than I have. You can pay them yourself, can’t you?

[12:52:21 AM] Irene Kusi: Oh no not right now. You know my dad left the Gold as a form of inheritance for me and since he could keep all that in his house he had to give it to his lawyer.

[12:53:14 AM] James Bach: Okay, so the lawyer has it.

[12:53:42 AM] Irene Kusi: Yes and what the lawyer is also saying is that he gave it to a security house here in ghana

[12:53:55 AM] James Bach: That’s good. Then you know where it is. What happened when you went to get it?

[12:54:42 AM] Irene Kusi: They told me that i have to pay for the number of years that it has been there so that they can give it to me . And the lawyer told me that i also need to get a man who will stand by me just to help me

[12:55:42 AM] James Bach: Is this place a bank?

[12:56:03 AM] Irene Kusi: No is not bank but a storage house .

[12:56:19 AM] James Bach: Is it licensed to store gold?

[12:56:29 AM] Irene Kusi: Yes. Is a licensed company. It has the sole mandate to store gold for people

(Use my knowledge of local laws and customs to reveal the true culprit. A consultant should have deep knowledge in useful domains.)

[12:56:57 AM] James Bach: Then they are violating Ghanan law by charging you to release it. Your lawyer should have told you that. Do you trust that lawyer? Do you know him well?

[12:57:27 AM] Irene Kusi: Yes he is the family lawyer .

[12:57:51 AM] James Bach: Then why is he telling you something that is against the laws of your own country? I think you need to talk to a different lawyer.

[12:58:59 AM] Irene Kusi: When we went there they told me that is not like they are charging me for the gold which is mine said by my late dad but for the number of years they have kept the gold for safe keeping thats what they mean. The lawyer has got nothing to do with this at all.

[1:01:34 AM] James Bach: That’s called an “advance fee scam” and they are trying to trick you. You need to go to the police. But I think your lawyer may be in on this. He may even have paid off the police.

(My kind of consulting involves tough, direct speech, on occasion. Sometimes that’s hard for clients to hear.)

[1:02:07 AM] Irene Kusi: You cant talk to my lawyer like that

[1:02:12 AM] James Bach: Your lawyer, if he knows Ghanan law, knows that they aren’t ALLOWED to charge you. They can’t charge you! You are being tricked!

[1:02:26 AM] Irene Kusi: Ok

[1:02:39 AM] James Bach: I told you that you have to be careful. Gold is VERY valuable. People want to steal it.

[1:03:03 AM] Irene Kusi: so now are you going to help me pay the bills or not ?

[1:03:46 AM] James Bach: I’m telling you that you don’t owe any bills. It’s a trap. They want you to pay them and then they won’t give you anything. You need to wake up Irene. You are a crime victim.

[1:04:05 AM] Irene Kusi: Stop what you are saying

[1:04:06 AM] James Bach: That gold is yours. You deserve it.

[1:04:08 AM] Irene Kusi: i hate it. what do you take me for ? you are not ready to pay fpor the bills thats why you are doing this

[1:04:40 AM] James Bach: I think you put too much belief in people who are now trying to hurt you. Now you are upset and you want to blame me. But I am just the messenger.

[1:05:08 AM] Irene Kusi: do you think the storage house is an orphanage home where they keep things for free?

[1:05:59 AM] James Bach: Stand up for your rights, Irene. Even in Ghana, there are laws and people have to follow them. They can’t just steal your gold and say you can’t have it because it’s been “sitting there for a long time”

[1:06:24 AM] Irene Kusi: Stop what you are saying

[1:06:26 AM] James Bach: Under GHANA LAW a licensed gold storage house CANNOT charge for storage AFTER the fact. They take payment UP FRONT. Your father already paid. Your lawyer knows that. You have to fight for what is right.

[1:06:49 AM] Irene Kusi: no he didnt pay

[1:07:06 AM] James Bach: THAT’S WHAT YOUR LAWYER TOLD YOU? I can’t believe what a liar this guy is. Do you think you can just bring gold to a storage house and not pay them to store it? He must have paid. So now you are asking strangers to help you pay a bill that YOU DON’T EVEN HAVE.

[1:08:23 AM] Irene Kusi: Bye

[1:08:29 AM] James Bach: I hope you get your gold. Think about what I said.

(If the client is not ready, it doesn’t matter what you tell them: they won’t benefit. I like to think Irene is taking my words to heart. But I have to accept that she probably won’t. Meanwhile I can only hope that her lawyer has invested his ill-gotten gains in an American solar company.)

Rapid Testing Intensive

2012-01-25T02:03:21+00:00

Announcing the Rapid Testing Intensive seminar.

This is something new. From July 24 to 28, my brother and I are going to run a short, extreme testing project over five days. We have a nice big space, good Internet, and we’ll be cut off from all distractions.

The most interesting element, from my point of view, is that students can attend onsite OR online. Onsite attendance is more trouble, more work, more expense, but also a completely vivid immersive testing experience where you will work shoulder to shoulder with two of the guys who created the Rapid Testing methodology that is now taught all over the world. Online participants will also test and attend optional webinars twice a day during the week. We will have a dedicated coordinator and an online forum so that the experience will be exactly like working as a remote tester on a busy project team.

The outcome of the week will be a test report that will become a part of each participant’s professional portfolio.

I’m so glad Jon is doing this with me. We complement each other well. Jon is a day-to-day test manager who has applied Rapid Testing in his work for nearly all his career since the mid-90’s. I’m a consultant who’s been teaching Rapid Testing since I invented it. Besides, it gives me an excuse to use this graphic…

This may look like George Lucas and Vladimir Lenin, but it’s me and Jon…

Avoid Trivial Bugs, Report What Works

2012-01-24T18:10:00+00:00

I’ve been testing this darn thing all morning and I haven't found a single bug, or even an issue. My manager probably thinks I’m not testing well enough. My other tester colleagues keep finding bugs in their projects. Maybe I’m not a very good tester. My next scrum report is going to be lame. This sucks, man.

Wrong! It probably doesn’t suck. Not finding bugs may be a good thing. Your team may be building stuff that works. And you get to be the lucky dude who delivers the good news.

If there is lots of stuff that works and no bugs, you have even more to report than testers who keep finding bugs. Testers who keep finding bugs are probably executing fewer tests than you so they know less about their products than you. Instead of figuring out what works, they are stuck investigating what doesn’t work. They’ll still need to figure out what works eventually, it’s just going to take them a while to get there. And that sucks.

My manager is probably looking at my low bug count metric, thinking I’m not doing anything. Logging bugs makes me feel like a bad ass. There must be something I can log…hmmm…I know, I’ll log a bug for this user message; it’s not really worded as well as it could be…it has been like that for the last four years.

No! No! No! That’s gaming the system. It’s not going to work. You’re going to get a reputation as a tester who logs trivial bugs. Your manager is only counting bugs because you’re not giving her anything else. She just wants to know what you’re doing. Help your manager. Show her where to find your test reports, session sheets, or test execution results. Invite her to your scrum meetings. Tell her how busy you’ve been knocking out tests and how bad ass your entire project team is.

Think about it.

Reporting what works may be better than reporting trivial bugs.

Software Testing Resources

2012-01-24T10:30:19+00:00

Software Testing resources that are helpful to development community

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

An ode to running a database on bare metal

2012-01-24T09:03:01+00:00

No, my muse is not quite as strong as to inspire me to write an ode, but I still want to emphasize a few points about the goodness of running a database on bare metal.

At Evite, we use sharded MySQL for our production database. We designed the current architecture in 2009, when NoSQL was still very much in its infancy, so MySQL seemed a solid choice, a technology that we could at least understand. As I explained elsewhere, we do use MySQL in an almost non-relational way, and we sharded from the get-go, with the idea that it's better to scale horizontally than vertically.

We initially launched with the database hosted at a data center on a few Dell PE2970 servers, each with 16 GB of RAM and 2 quad-core CPUs. Each server was running 2 MySQL instances. We didn't get a chance to dark launch, but the initial load testing we did showed that we should be OK. However, there is nothing like production traffic to really stress test your infrastructure, and we soon realized that we have an insufficient number of servers for the peak traffic we were expecting towards the end of the year.

We decided to scale horizontally in EC2, with one MySQL instance per m1.xlarge EC2 instance. At the time we also engaged Percona and they helped us fine-tune our Percona XtraDB MySQL configuration so we could get the most out of the m1.xlarge horsepower. We managed to scale sufficiently enough for our high season in 2010, although we had plenty of pain points. We chose to use EBS volumes for our database files, because at the time EBS still gave people the illusion of stability and durability. We were very soon confronted with severe performance issues, manifested as very high CPU I/O wait times, which were sometimes so high as to make the instance useless.

I described in a previous post how proficient we became at failing over from a master that went AWOL to a slave. Our issues with EBS volumes were compounded by the fact that our database access pattern is very write-intensive, and a shared medium such as EBS was far from ideal. Our devops team was constantly on the alert, and it seemed like we were always rebuilding instances and recovering from EC2 instance failures, although the end-user experience was not affected.

Long story short, we decided to bring the database back in-house, at the data center, on 'real' bare-metal servers. No virtualization, thanks. The whole process went relatively smoothly. One important point I want to make here is that we already had a year's worth of hard numbers at that point regarding the access patterns to our database, iops/sec, MySQL query types, etc, etc. So it made it easy to do proper capacity planning this time, in the presence of production traffic.

We started by buying 2 Dell C2100 servers, monster machines, with dual Intel Xeon X5650 processors (for a total of 24 cores), 144 GB RAM, and 12 x 1 TB hard disks out of which we prepared a 6 TB RAID 10 volume which we further divided in LVM logical volumes for specific types of MySQL files.

We put 2 MySQL instances on each server, and we engaged Percona again to help us fine-tune the configuration, this time including not only MySQL, but also the hardware and the OS. They were super helpful to us, as usual. Here are only some of the things they recommended, which we implemented:

set vm.swappiness kernel setting to 0 in /etc/sysctl.conf
set InnoDB flush method to O_DIRECT because we can rely on the RAID controller to do the caching (we also mounted XFS with the nobarrier option in conjunction with this change)
disable MySQL query cache, which uses a global mutex that can cause performance issues when used on a multi-core server
various other optimizations which were dependent on our setup, things like tweaking MySQL configuration options such as key_buffer_size and innodb_io_capacity

One important MySQL configuration option that we had to tweak was innodb_buffer_pool_size. If we set it too high, the server could start swapping. If we set it too low, the disk I/O on the server could become too problematic. Since we had 144 GB of RAM and we were running 2 MySQL instances per server, we decided to give each instance 60 GB of RAM. This proved to strike a good balance.

Once the fine-tuning was done, we directed production traffic away from 4 EC2 m1.xlarge instances to 2 x 2 MySQL instances, with each pair running on a C2100. We then sat back and wallowed for a while in the goodness of the I/O numbers we were observing. Basically, the servers were barely working. This is how life should be.

We soon migrated all of our MySQL masters back into the data center. We left the slaves running in EC2 (still one m1.xlarge slave per MySQL master instance), but we changed them from being EBS-backed to using the local ephemeral disk in RAID 0 with LVM. We look at EC2 in this case as a secondary data center, used only in emergency situations.

One thing that bit us in our bare-metal setup was....a bare-metal issue around the LSI MegaRAID controllers. I already blogged about the problems we had with the battery relearning cycle, and with decreased performance in the presence of bad drives. But these things were easy to fix (again thanks to our friends at Percona for diagnosing these issues correctly in the first place...)

I am happy to report that we went through our high season for 2011 without a glitch in this setup. Our devops team slept much better at night too! One nice thing about having EC2 as a 'secondary data center' is that if need be, we can scale out horizontally by launching more EC2 instances. In fact, we doubled the number of MySQL slave instances for the duration of our high season, with the thought that if we need to, we can double the number of shards at the application layer, and thus scale horizontally that way. We didn't have to do any tweaking fortunately, but we were able to -- a strategy which would otherwise be hard to pull off if we didn't have any cloud presence, unless we bought a lot of extra capacity at the data center.

This brings me to one of the points I want to make in this post: it is a very valuable strategy to be able to use the cloud to roll out a new architecture (which you designed from the get-go however to be horizontally scalable) and to gauge its performance in the presence of real production traffic. You will get less than optimal performance per instance (because of virtualization vs. real hardware) , but since you can scale horizontally, you should be able to sustain the desired level of traffic for your application. You will get hard numbers that will help you do capacity planning and you will be able to bring the database infrastructure back to real hardware if you so wish, like we did. Note that Zynga has a similar strategy -- they roll out new games in EC2 and once they get a handle on how much traffic a game has, they bring it back into the data center (although it looks like they still use a private cloud and not bare metal).

Another point I want to make is that the cloud is not ready yet for write-intensive transactional databases, mainly because of the very poor I/O performance that you get on virtual instances in the cloud (compounded by shared network storage such as EBS). Adrian Cockcroft will reply that Netflix is doing just fine and they're exclusively in EC2. I hope they are doing just fine, and I hope his devops team is getting some good sleep at night, but I'm not sure. I need to perhaps qualify my point and say that the cloud is not ready for traditional transactional databases such as MySQL and PostgreSQL, which require manual sharding to be horizontally scalable. If I had to look at redesigning our database architecture today, I'd definitely try out HBase, Riak and maybe Cassandra. The promise there at least is that adding a new node to the cluster in these technologies is much less painful than in the manual sharding and scaling scenario. This still doesn't guarantee that you won't end up paying for a lot of instances to compensate for poor individual I/O per instance. Maybe a cloud vendor like Joyent with their SmartMachines will make a difference in this area (in fact, it is on our TODO list to test out their Percona SmartMachine).

Note however that there's something to be said about using good ol' RDBMS technologies. Ryan Mack says this in a Facebook Engineering post:

"After a few discussions we decided to build on four of our core technologies: MySQL/InnoDB for storage and replication, Multifeed (the technology that powers News Feed) for ranking, Thrift for communications, and memcached for caching. We chose well-understood technologies so we could better predict capacity needs and rely on our existing monitoring and operational tool kits."

The emphasis on the last sentence is mine. It's the operational aspect of a new architecture that will kill you first. With a well understood architecture, at least you have a chance to tame it.

Yet another point I'd like to make is: do not base your disaster recovery strategy in EC2 around EBS volumes, especially if you have a write-intensive database. It's not worth the performance loss, and most of all it's not worth the severe and unpredictable fluctuation in performance. It works much better in our experience to turn the ephemeral disks of an m1.xlarge EC2 instance into a RAID 0 array and put LVM on top of that, and use it for storing the various MySQL file types. We are then able to do LVM snapshots of that volume, and upload the snapshots to S3. To build a new slave, we can restore the snapshot from S3, then catch up the replication with the master. Works fine.

There you have it. An ode in prose to running your database on bare metal. Try it, you may sleep better at night!

Scripts or No Scripts, Managers Might Have to Manage

2012-01-23T18:38:03+00:00

A fellow named Oren Reshef writes in response to my post on Worthwhile Documentation.

Let me be the devil’s advocate for a post.

Not having fully detailed test steps may lead to insufficient data in bug reports.

Yup, that could be a risk (although having fully detailed steps in a test script might also lead to insufficient data in bug reports; and insufficient to whom, exactly?).

So what do you do with a problem like that? You manage it. You train the tester, reminding her of the heuristic that each problem report needs a problem description; an example of something that shows the problem; and why she thinks it’s a problem (that is, the oracle; the principle or mechanism by which the tester recognizes the problem). Problem, example, and why; PEW. You praise and reward the tester for producing reports that follow the PEW heuristic; you critique reports that don’t have them. You show the tester lots of examples of bug reports, and ask her to differentiate between the good ones and the bad ones, why each one might be consider good or bad, and in what ways. If the tester isn’t getting it, you have the tester work with and be coached by someone who does get it. The coach talks the tester through the process of identifying a problem, deciding why it’s a problem, and outlining the necessary information. Sometimes it’s steps and specific data; sometimes the steps are obvious and it’s only the data you need to specify; sometimes the problem happens with any old data, and it’s the steps that are important. And sometimes the description of the problem contains enough information that you need supply neither steps nor data. As a tester under time pressure, she needs to develop the skill to do this rapidly and well—or, if nothing works, she might have to find a job for which she is better suited.

You can argue that a good tester should include the needed information and steps in her bug report, but this raise (at least) two problems:

- The same information may be duplicated across many bugs, and even worst it will not be consistent.

As a manager, I can not only argue that a tester should include the needed information; I can require that a tester include the needed information. Come on, Mr. Advocate… this is a problem that a capable tester and a capable test manager (and presumably your client) can solve. If “the same” information is duplicated across many bugs, might that be an interesting factor worth noting? A test result, if you will? Will this actually persist for long without the test manager (or test leads, or the test team) noticing or managing it?

And in any case, would a script solve the problem that you post above? If you can solve that problem in a script, can you solve it in a (set of) bug report(s)?

Writing test steps is not as trivial as it sounds (for example due to cognitive biases, or simply by overlooking steps that seems obvious to you), and to be efficient they also need to be peer reviewed and tested. You don’t want that to happen in a bug report.

“Writing test steps is not as trivial as it sounds.” I know. It’s non-trivial in terms of time, and it’s non-trivial in terms of skill, and it’s non-trivial in terms of cost. That’s why I write about those problems. That’s why James Bach writes about them.

Again: how do you solve problems like testers providing inefficient repro steps? You solve it with training, practice, coaching, review, supervision, observation, interaction… that is, if you don’t like the results you’re getting, you steer the testers in the direction you want them to go, with leadership and management.

The tester may choose the same steps over and over, or steps that are easier for her but does not represent real customers.

Yes, I often hear things like this to justify poor testing. “Real customers” according to whom? It seems as though many organizations have a problem recognizing that hackers are real; that people under pressure are real; that people who make mistakes are real; that people who can become distracted are real. That people who get up and go away from the keyboard, such that a transaction times out are real.

Is it the role of testers to behave always like idealized “real” customers? That’s like saying that it’s the role of airport security to assume that all of the business class customers are “real” business people. I’d argue that it’s nice for testers to be able to act like customers, but it’s far more important for testers to act like testers. It’s the tester’s role to identify important vulnerabilities in the product. Sometimes that involves behaving like a typical customer, and sometimes it involves behaving like an atypical customer, or and sometimes it involves behaving like someone who is not a customer at all. But again, mostly it involves behaving like a tester.

Again you may argue that a good tester should take all that into account, but it’s not that simple to verify it especially for tests involving many short trivial steps.

Maybe it isn’t that simple. If that’s a problem, what about logging? What about screen capture tools? Such tools will track activities far more accurately than a script the tester allegedly followed. After all, a test script is just a rumour of how something should be done, and the claim that the script was followed is also a rumour. What about direct supervision and scrutiny? What about occasional pairing? What about reviewing the testers’ work? What about providing feedback to testers, while affording them both freedom and responsibility?

And would scripts solve that problem when (for example) you’re a recording bug that you’ve just discovered (probably after deviating from a script)? How, exactly? What happens when a problem identified by a script is fixed? Does the value of the script stay constant over time?

Detailed test steps (at least to some extent) might important if your test activity might be transferred to another offshore team someday (happened to me a few weeks ago, I sent them a test document with only high level details and hoped for the best), or your customer requires in-depth understanding of your tests (a multi-billion Canadian telecommunication company insisted on getting those from us during the late 90’s, we chose the least readable TestDirector export format and shipped it to them…).

Ah, yes. “I sent them a test document with only high level details and hoped for the best.” What can I say about “hope” as a management approach? Does a pile of test scripts impart in-depth understanding? Or are they (as I suspect) a way of responding to a question that you didn’t know how to answer, which was in fact a question that the telco didn’t know how to ask?

Going through some set of actions by rote is not a test. A test script is not a test. A test is what you think and what you do. It is a complex, cognitive activity that requires the presence or the development of much tacit knowledge. Raw data or raw instructions at best provide you with a miniscule fraction of what you need to know. If someone wanted in-depth understanding of how a retail store works, would you send them a pile of uncontextualized cash register receipts?

The Devil’s Advocate never seems to have a thoughtful manager for a client. I would suggest that a tester neither hire nor work for the devil.

Thank you for playing the devil’s advocate, Oren.

That’s a Nice Theory

2012-01-21T15:43:17+00:00

Dale Emery has taught me an enormous amount about using resistance as a resource.

I’m grateful. I use his ideas every time I set foot in a classroom or start consulting with a new client.

In particular, I channel my inner Dale whenever discussing any of the various controversial things I advocate, such as:

The whole team is responsible for testing and quality, not just QA or the designated testers.

If the regression tests aren’t automated and the team is having a hard time finishing all the testing within a sprint, have programmers help execute them.

Do away with traditional (and often punitive) defect metrics like % missed by phase. Focus instead on metrics related to accomplishments: story points completed, cycle time, and test coverage.

In many organizations these suggestions fly in the face of their accepted “best practices.” Such ideas also tread on political toes. So one response I hear a lot is: “That’s a nice theory, but it won’t work here.”

Before learning techniques for reframing that resistance into a resource, I would end up in a position based argument that amounted to the professional equivalent of “Will too!” “Will not!” “Will too!” “Will not!” Not useful.

Dale’s nicer and wiser than I am, so even when I don’t handle the interaction as well as I would like, by leaning on Dale’s techniques, I certainly handle the conversation better than I otherwise would have. (Although I’m far from perfect at this. Sometimes people succeed in pushing my buttons in such a way that I forget everything I know about how to communicate effectively.)

The first thing I need to know is whether the results of doing whatever it is would be useful. So I ask something like:

Do you think this practice could help improve things?

If I hear “No,” as the reply, we have a fundamental and possibly insurmountable difference in perspective. Nothing I can say will make them try the practice if they do not believe there is any value in it.

I can explore their reasoning. I can say, “That’s interesting. Why not?” But if someone flat out does not believe that a practice I advocate will help, and I disagree even after listening to their reasoning, there is a good chance that I will not be able to help them. Further discussion will cause more harm than good. The best thing for me to do is to stop.

On the other hand if I hear “Yes…but…” then we have a different conversation. First I have to understand what follows the “but…” Often it’s:

It won’t work here.

At this point I am tempted to ask, “Why not?”

But I don’t.

“Why not?” won’t get us anywhere. We’ll end up running down a rathole of excuses starting with “our context is different.” (And of course, they’re right. Their context is different. Every context is different.)

So instead of asking “Why not?” I flip it around. I ask:

What would have to change for this practice to work here?

Now we get a list of objections, but each one is framed as a neat little impediment statement.

It would need to allow for this inherent complexity in our situation.

We’d need to allow time for it.

We’d need executive support.

We’d need money in the training budget.

We’d need to get the programmers to buy in.

We’d need the QA manager to agree.

And we can work on each of those impediments in much the same way, following the trail of reasons why this is a nice theory that can’t possibly work in their real world context all the way down to the bottom.

In what way would it have to accommodate the complexity?

What would have to happen in order to make time for it?

What would have to happen in order to get executive support?

What would have to happen in order to get budget money?

What would have to happen in order for the programmers to buy in?

What would have to happen in order for the QA manager to agree?

The answers usually reveal perfectly practical steps. We can talk to the people in a position of authority or influence who can get us resources, training, budget money. We can try a small pilot. We can experiment with variations.

The simple reframe from “Why not?” to “What would have to change?” opens up possibilities. What could have become an argument becomes instead a brainstorming session. The result is a chain of steps we can take to go from where we are now to where we want to be.

Eight Things You May Not Need To Test

2012-01-20T20:39:00+00:00

This article will be published in a future addition of the Software Test Professionals Insider – community news. I didn’t get a chance to write my blog post this week so I thought I would cheat and publish it on my own blog first.

I will also be interviewed about it on Rich Hand’s live Blog Talk Radio Show on Tuesday, January 31st at 1PM eastern time.

My article is below. If it makes sense to you or bothers you, make sure you tune in to the radio show to ask questions…and leave a comment here, of course.

Don’t Test It

As testers, we ask ourselves lots of questions:

What is the best test I can execute right now?
What is my test approach going to be?
Is that a bug?
Am I done yet?

But how many of us ask questions like the following?

Does this Feature need to ever be tested?
Does it need to be tested by me?
Who cares if it doesn’t work?

In my opinion, not enough of us ask questions like the three above. Maybe it’s because we’ve been taught to test everything. Some of us even have a process that requires every Feature to be stamped “Tested” by someone on the QA team. We treat testing like a routine factory procedure and sometimes we even take pride in saying...

“I am the tester. Therefore, everything must be tested...by me...even if a non-tester already tested it...even if I already know it will pass...even if a programmer needs to tell me how to test it...I must test it, no exceptions!”

This type of thinking may be giving testers a bad reputation. It emphasizes testing importance because of a thoughtless process rather than a service to provide the most valuable information to someone.

James Bach came up with the following test execution heuristic:

Basic Heuristic: “If it exists, I want to test it”

I disagree with that heuristic, as it is shown above and often published. However, I completely agree with the full version James published when he introduced it in his 7/8/2006 blog post:

“If it exists, I want to test it. (The only exception is if I have something more important to do.)”

The second sentence is huge! Why? Because often we do have something more important to do, and it’s usually another test! Unfortunately, importance is not always obvious. So rather than measuring importance, I like to ask the three questions above and look for things that may not be worth my time to test. Here are eight examples of what I’m talking about:

Features that don’t go to production - My team has these every iteration. These are things like enhancements to error logging tables or audit reports to track production activity. On Agile teams these fall under the umbrella of Developer User Stories. The bits literally do not go to production and by their nature cannot directly affect users.
Patches for critical production problems that can’t get worse - One afternoon our customers called tech support indicating they were on the verge of missing a critical deadline because our product had a blocking bug. We had one hour to deliver the fix to production. The programmer had the fix ready quickly and the risk of further breaking production was insignificant because production was currently useless. Want to be a hero? Don’t slow things down. Pass it through to production. Test it later if you need to.
Cosmetic bug fixes with timely test setup - We fixed a spelling mistake that had shown up on a screen shot of a user error message. The user was unaware of the spelling mistake but we fixed it anyway; quick and easy. Triggering said error message required about 30 minutes of setup. Is it worth it?
Straight forward configuration changes - Last year our product began encountering abnormally large production jobs it could not process. A programmer attempted to fix the problem with an obvious configuration change. There was no easy way to create a job large enough to cross the threshold in the QA environment. We made the configuration change in production and the users happily did the testing for us.
Too technical for a non-programmer to test - Testing some functionality requires performing actions while using breakpoints in the code to reproduce race conditions. Sometimes a tester is no match for the tools and skills of a programmer with intimate knowledge of the product code. Discuss the tests but step aside.
Non-tester on loan - If a non-tester on the team is willing to help test, or better yet, wants to help test a certain Feature, take advantage of it. Share test ideas and ask for test reports. If you’re satisfied, don’t test it.
No repro steps - Occasionally a programmer will take a stab at something. There are often errors reported for which nobody can determine the reproduction steps. We may want to regression test the updated area, but we won’t prevent the apparent fix from deploying just because we don’t know if it works or not.
Inadequate test data or hardware - Let’s face it. Most of us don’t have as many load balanced servers in our QA environment as we do in production. When a valid test requires production resources not available outside of production, we may not be able to test it.

Many of you are probably trying to imagine cases where the items above could result in problems if untested. I can do that too. Remember, these are items that may not be worth our time to test. Weigh them against what else you can do and ask your stakeholders when it’s not obvious.

If you do choose not to test something, it’s important not to mislead. Here is the approach we use on my team. During our Feature Reviews, we (testers) say, “we are not going to test this”. If someone disagrees, we change our mind and test it. If no one disagrees, we “rubber stamp” it. Which means we indicate nothing was tested (on the work item or story) and pass it through so it can proceed to production. The expression “rubber stamping” came from the familiar image of an administrative worker rubber stamping stacks of papers without really spending any time on each. The rubber stamp is valuable, however. It tells us something did not slip through the cracks. Instead, we used our brains and determined our energy was best used elsewhere.

So the next time you find yourself embarking on testing that feels much less important than other testing you could be doing, you may want to consider...not testing it. In time, your team will grow to respect your decision and benefit from fewer bottlenecks and increased test coverage where you can actually add value.

Blog Rolling with MongoDB, Node.js and Coffeescript

2012-01-16T18:54:26+00:00

This morning I woke up with a lingering thought on my mind that was left over from recent conversations. In the technical community we often get so invested in our work that rather than talk about the simple building blocks that build our success we talk about the huge breakthroughs we make. The problem however is that our breakthroughs most often aren’t accessible to someone who wants to just get started. So today I will give an intro tutorial to using node.js, coffeescript and mongodb to build a simple blog. It builds off the concept in a tutorial I first used to learn node.js more than a year ago, but with a completely from scratch approach. In this tutorial I will also cover practicing Behavior Driven Development using Mocha.

Getting Started

Quite obviously, you’re going to need node.js and mongodb installed. I recommend downloading and installing from the node.js website and following the instructions. I will note that this tutorial covers 0.6.7 so if you come across this post a year from now the API might have changed significantly since then. You will also need mongodb installed, you can download it here. If you use Ubuntu (or some other Debian derivative) you should consider installing from their apt repository. OSX? No problem, you can also install it via homebrew or macports.

Finally, since we’ll be using coffeescript for this tutorial, run npm -g install coffee-script (you might need to sudo) to install coffeescript. Run coffee from the commandline to access the coffeescript REPL. If all works well, install these additional packages listed below via npm that we’ll be using throughout the tutorial.

express
mocha

Now let’s bootstrap our project structure. Type express coffeepress to generate a skeleton express project structure. You should see output similar to the following:

Notice how at the end it says to cd to the directory and type npm install? Let’s follow those instructions. Let’s run what we have so far by typing node app.js and navigating to http://localhost:3000. This is the default structure that gives a good starting point. Feel free to investigate the files under the directory before moving on. I even suggest poking around by changing the view a bit and changing the title from express to “My Coffeepress Blog”.

Porting to Coffeescript

At this point, let’s port our backend to coffeescript. I used to copy and paste files into the js2coffee website but you can also install js2coffee via npm. So run the following:

Now you can run coffee app.coffee to run the same app, but now in coffeescript. Take a look at the resulting files to get a feel for what has changed. New to coffeescript? Then I recommend taking a gander at coffeescript.org before moving on. Here is the project structure so far.

Basic Navigation

I like to try and work my way from the outside in while developing a site or feature, materializing components into existence as I need them. So let’s start by working on the initial navigation of the site with some simple in-memory storage of blog posts. This is a good time to get our test framework setup and write a few simple tests against our routes. Normally I prefer to not write tests against my routes, shoving logic into heavy models or services. However I have come to learn that untested components of a system serve as a gravity well for untested code that eventually leads to clients calling you about broken applications. What follows serves as both an introduction to Mocha as well as express’ routing mechanism.

Let’s edit our package.json to include our test framework dependencies.

We include should so that we can use BDD style assertions (more on this in a bit). Write a simple test case located at test/routes-test.coffee with the following code to get us started with mocha

Now run this by typing mocha from the root of the project directory. It should pass. Let’s go ahead and make it fail by changing 4 to 5 and rerunning it. Hopefully this gives you a good feel for our test framework before we move on and change this test to reflect our existing index route. Swap the code in this test out with the following.

Here we fake our requests and response in order to capture what is passed into the response. We fake the render method and verify that our rendered view is “index” and that the variable title is equal to what we expect to be passed in. Run the tests and make changes to your route to make it pass.

Now let’s add a post variable that will be an array of posts we’ll display on the front page. Add the following assertion right after the title assertion:

Run the tests to see it fail and change the route to have a posts array variable available in the template.

Unfortunately you’ll notice that the test fails. This is due to a subtle difference between equal and eql. The former enforces strict equality while the latter is a bit looser, so we change our assertion to use eql. Take a look at the should documentation for more information.

Next let’s write tests for the “new post” route.

Run it, see the failure, and rework our routes.coffee file to include the route (with no implementation yet)

You’ll notice our test passes. That’s not good. Why? Because we put our assertion in our req.render callback, which never gets executed. Doh! How can we make absolutely sure it gets called during our test run? Old school thinking would have you assign a local variable outside the scope of the callback that gets assigned during execution and then can be verified against later on. However we have no guarantee that the routing logic will be synchronous!

Thankfully mocha has a feature that allows for easy testing in these situations. In the method declaration of our test specify a parameter named done. This is a callback that we can call anywhere to indicate the test is done. Basically the test will wait up to a default of 2000ms for it to be called. With this in mind, let’s modify our tests with the following:

If we run this via mocha now we’ll notice that we have one failure. Let’s go ahead and implement the route and connect it into our router.

And connecting it up in the app.coffee

Modifying the Views

This code is useless without views, so let’s modify our views a bit. Let’s modify our layout.jade to link to the new posts page. This layout also makes use of twitter bootstrap because I’m too lazy to design something for this tutorial.

And create our add_post view at views/add_post.jade. An interesting thing to note here that I’ll touch on in a bit is that I prefix the input names with post.

Now let’s add another route to handle the post. This time I’m going to kind of skip delving into the details of writing the test, but you can look at what I have so far here if you’d like to see it.

For now, we’re just going to store each post in an array. Nothing fancy yet. We also add a new route to app.coffee. We could refactor or use some express-mvc plugin to reduce adding each route by hand, but I think it’s good to do it like this to get a feel for express’ low level routing mechanisms.

Finally, we’ll add one last view for viewing a single post:

Go ahead and start the application up and navigate to http://localhost:3000. Post a few posts and play around a bit. You can see the finished application we have so far here.

Mongoose

Whew. I hope I haven’t lost you yet. Especially with the tests against the routes… I know those are always a bit painful! Now that we have functional blog let’s make it work by storing posts in mongodb using Mongoose.

Let’s add a dependency on mongoose to our project and freeze it at version 2.4.10. As always, run npm install to bring it in. Now we’ll create an initial test to just test mongoose out.

Here we import both mongoose and the model object that we’re going to create. Since we want our test to start with a clean slate, we use the before hook (which runs once before anything else in the test runs) to both connect to the database and then remove all of the Post objects from mongodb. We pass the done callback to the remove call so that tests don’t run until all Posts have been removed.

Now we create a new Post instance. You can pass an object literal in to set properties on the model, so we do that here. Finally, in our post.save callback we look the post back up and verify certain attributes have been set. It’s a dumb test (and in fact I rarely test mongoose’s behavior like this), but it does verify that we’ve configured our model correctly.

Now let’s implement our model to make the test pass.

Pretty simple. Now let’s refit our routes to use the Post model instead of an in memory array.

That’s all good and dandy, but one last hiccup is our tests for our routes now fail. Chalk this one up to not having any abstraction or dependency injection in place, but that is fine for now, we’ll live with it and change the tests.

Finally we need our app to actually connect to mongoose when we run it. I like to do this based on the express configuration. This is immensely important if you have mongodb running on servers separated from your application. For this example we’ll just use the databases coffeepress-dev and coffeepress-prod.

Run it and write a few posts. Restart the app and you’ll see the posts still there. Woot!

Conclusion

Well, that about wraps it up… you can see this tutorial in it’s finished glory on the finished branch of the repository. There’s a bit missing out here that we’d implement in the real world. Obviously some kind of authentication would be in order if we took this further, possibly using mongoose-auth. We’d also want to add some validation when posting. These are all excellent topics for future posts but for now I hope this was enough to help you get going!

On the Usability of Codecademy

2012-01-16T17:09:21+00:00

I asked a friend of mine (who isn’t a coder) if she could try out the amazing Codecademy for me.

She is a smart person, but again, she isn’t a programmer. I thought Codecademy would be a great way to introduce her to programming for the web. As a programmer, I thought it was extremely well made and assumed it would work for anyone.

She landed on the homepage and read the instructions.

She wrote her name (“Elizabeth”), then decided however to change it to the shortened form of her name, “Liz”. To do so she highlighted the whole string she had just typed and tried to press the backspace key to delete it. The whole page disappeared and the browser redirected her to the new tab.

Liz: “Why did it do that?”

Me: “You can’t highlight a word like you’d do in Microsoft Word. You’re in a browser, so the backspace button tells the browser to go back to the previous page you visited in that particular tab. You need to delete the string one letter at the time.”

She reopened the page and tried again, this time things were successful.

Next, she had to measure the length of her name. But she forgot the quotes, so she got this:

Liz: “What happened?”

Me: “What do you think happened?”

Liz: “Some kind of error, but what does ‘Liz is not defined’ mean?”

Me: “You accidentally forgot the quotes.”

Liz: “Oh, OK, let me try again.”

She wrote the first quote but pressed enter before immediately realizing she had forgotten the closing one. As such she encountered the following scenario:

Liz: “I messed it, let me try again.”

Liz: “I typed it right this time, why didn’t it work?”

Me: “Since you didn’t close the first string, it thinks that the second line is still part of the first command. Do you know how to get out of this?”

Liz: “No, but let me try.”

No, she didn’t close the string and pressed enter twice or deleted a bunch of characters (this web REPL allows it, kudos to Codecademy). She simply reloaded the page. Thankfully the same exercise (#1) was reloaded. After completing it successfully though, instead of exercise #2, exercise #0 reappeared, asking her to enter her first name again.

After raising an eyebrow at that point, she went along with it. Next she was asked to try some math.

She can’t be faulted for thinking that “for instance” meant she could try something else. Yet, she got an “Oops, try again.” message.

At that point I suspended this tiny hallway test and I asked her what she thought of this system. She found the site to be a bit unintuitive, but above all felt like she didn’t know how to get unstuck when she made an error.

Now, before you all jump in and tell me that my friend is dumb, that she’ll never be a programmer, or that you can’t idiot-proof a design, keep in mind the very point of this post.

Usability is hard because real users use our UIs and software in different and unexpected ways. Codecamedy did a very good job, but even they left plenty of room for improvement. As web developers it’s our duty to give usability and user experience a significant amount of attention. The UI is the software to the end user.

No related posts.

Who Tests Your Automated Tests?

2012-01-11T21:32:00+00:00

Who tests the automated tests? Well, we know the test automation engineer does. But per our usual rationale, shouldn’t someone other than the test automation engineer test the automated tests? After all, they are coded…in some cases by people with less programming experience than the product programmers.

We’re experimenting with manual testers testing automated tests on one of my project teams. The test automation engineer hands each completed automated test to a manual tester. The manual tester then executes the automated test. At this point the manual tester is testing two things at the same time. They are:

Testing the automated test and
testing whatever the automated test tests

Or, to be more precise, we can use Michael Bolton speak and say the tester is:

Testing the automated check and
checking whatever the automated check checks

Whatever you call it, during this exercise, it’s important to distinguish the above two activities. If the automated test's execution results in a “Fail”, it doesn’t mean the test of the automated test fails…but it may. Are you with me still? An automated test’s execution result of “Fail” may, in fact, mean the automated test is a damn good test. It may have found a bug in the product under test. But that is completely up to the manual tester who is testing the automated test. One cannot trust the expected result of an automated test until one has finished testing the automated test.

Thus, the tester of the automated test will need to evaluate the test somehow and declare it to be a good test. They may be able to do this several ways:

Manipulate the product to see if the test both passes and fails under the right conditions.
Execute the automated test by itself then as part of the test suite to determine if the setup/teardown routines adapt sufficiently.
Read the automated test’s code (e.g., does the Assert check the intended observation correctly?).
Manually test the same thing the automated test checks.
Manipulate the product such that the test cannot evaluate its check. Does the test resolve as “Inconclusive”?

It would be nice to have the luxury of time/resources to test the automated checks thoroughly but in the end, I suspect we will have to draw the line somewhere and trust the test automation engineer’s test of their check. In the meantime, we’ll see where this gets us.

Public Class in England March 7-9

2012-01-09T10:42:10+00:00

Hey, I’ll be doing a public class in England, once again. This time in Cambridge. See the details, here.

This year I also have public classes in Estonia, Romania, Australia, and New Zealand.

Agile Adjustments: a WordCount Story

2012-01-05T21:03:58+00:00

I originally wrote this for the AYE website in 2007. It’s no longer published there so I’m posting it here. Despite itching to tweak some words and add a better conclusion, I resisted the temptation to edit it other than formatting it for this blog. It’s as I wrote it in 2007. (Despite being 4 years old, I think this post is still relevant…perhaps even more so today with Agile having crossed the chasm.)

We were in the middle of my Agile Testing class, and the simulation had run for two rounds so far. Some of the participants created “software” on index cards. Others tested it. Still others deployed it. The participants were wholly engaged in their work for the fictitious “Word Count, Inc.” As the facilitator, I was running the simulation in 15 minute rounds followed by 15 minute reflect-and-adjust mini-retrospectives.

After the second round, during the mini-retrospective, I asked, “What do you see happening?”

“The deployment team looked like they were twiddling their thumbs for most of the round,” one participant observed.

Another participant added, “I think that’s because most of the cards are still on the QA table,” she said. “QA is a bottleneck.”

“No, the problem is that development didn’t deliver anything until the very last minute.” objected one of the QA team members.

“Well that’s because it took us most of the last round to coordinate with the deployment team,” one of the Developers countered.

“Your cards were all mixed up when you delivered them. We sent them back so you could sort them out. That’s hardly a ‘coordination’ problem.” scowled a Deployment team member.

Mixed up source code, software stuck in QA, late deliverables. Sounded like a real world project to me.

I shifted the conversation: “What would you like to change to improve the outcome in the next iteration?”

The answers varied: “Hold more project meetings to coordinate efforts!” “Appoint a project manager to keep everything on track!” “More people in QA!” “Define a source code control process!” The suggestions may all have been different, but there was a general trend: the participants wanted to add control points, process steps, and personnel in an attempt to reduce the chaos.

For the next round, the team adopted new practices: adding a new role of project manager; adding more meetings; and adding a strict change control process. During the next round I observed the team use half their available time standing in a big group discussing how to proceed. It seemed to me that in their attempt to control the chaos, they created a process in which it was almost impossible to get anything done. Once again, they weren’t able to deploy an updated version. And at the end of the round, the project manager quit the role in disgust and went back to “coding” on cards.

The team meant well when they added the role of project manager, and added more meetings, but their strategy backfired.

Most groups that go through the WordCount, Inc. simulation encounter problems similar to the ones that this team encountered. Some react by attempting to introduce the same kinds of controls as this group, with similar results. But some respond differently.

One group responded to the mixed-up-source-code problem by creating a centralized code repository that was visible and shared by all. Instead of creating a change control process to manage the multiple copies of the source code floating around, they posted one copy to be shared by all in a central location: the paper equivalent of source control.

Another group responded to coordination and bottleneck problems by co-locating teams. Instead of holding meetings, they coordinated efforts by working together.

Yet another group established an “automated” regression test suite that the deployment team always ran prior to each deployment. They then posted the test results on a Big Visible Chart so everyone knew the current state of the deployed system.

These steps all had the effect of making the team more Agile by increasing visibility, increasing feedback, improving collaboration, and increasing communication. And the end result for each group was success.

When reflecting-and-adjusting, it’s easy to reach for command-and-control solutions, to add quality gates and checkpoints and formal processes. But the irony is that such process changes often increase the level of chaos rather than reducing it. They introduce delays and bloat the process without solving the core problem.

It happens in the real world too.

One organization struggling with buggy code decided to create a role of Code Czar. Before any code could be checked into the source control system, it had to go through the Code Czar who would walk through the proposed changes with the programmer. The Code Czar role required someone very senior. Someone with tremendous experience with the large, complex code base under development. Someone who was also very, very busy. The result: code checkins were delayed whenever the Code Czar was unavailable. Worse, despite having more experience than anyone else on the team, the Code Czar couldn’t always tell what effect a given set of changes might have. The delays in checkins weren’t worth it; they did not result in an overall improvement in code quality.

By contrast, many teams find that automated unit tests work far better as a code quality feedback mechanism than a designated human code reviewer. Instead of waiting for a very busy person to become available, programmers can find out for themselves in minutes if their latest changes will have undesired side effects.

Even Agile teams that regularly reflect-and-adapt in iteration retrospectives are not immune to the temptation to revert to command-and-control practices. For example, Agile teams struggling to test everything during an iteration sometimes create a formal testing phase outside the iteration. I even heard of one organization that was struggling with completing all the tasks in an iteration attempt to solve the problem by having their Scrum Master do a Work Breakdown Structure (WBS) and delegate tasks to specific team members. Not surprisingly, both solutions caused more problems than they solved.

So how can you tell if a given process change will actually be an improvement and make a team more Agile? Before implementing a process change, consider how (or if) the proposed change supports Agile values like visibility, feedback, communication, collaboration, efficiency, and rapid and frequent deliveries. Also ask yourself these questions:

Does the process change rely on humans achieving perfection? To succeed in the role, the Code Czar would have had to have perfect knowledge of all the interdependencies in the code. Similarly, some processes rely on having perfect requirements up front. Successful practices don’t rely on perfect knowledge or perfect work products. Instead, they rely on fast feedback and visibility to enable the team to detect problems early, correct them while they’re small, and enable the team to improve iteratively.

Does it result in more time talking than working? Beware any process improvement that involves more meetings. More meetings rarely solve either communication or coordination problems. As the project manager in the simulation discovered, talking about work doesn’t increase the amount of work actually accomplished. As an alternative to meetings, consider collaborative working sessions where team members do the work rather than talking about it.

Does it introduce unnecessary delays or false dependencies? Whenever a process change increases the number of formal hand-offs, it slows things down but may not improve the overall outcome. The Code Czar learned this the hard way.

Change Artist Challenge #11: Putting Theory Into Practice

2012-01-05T12:14:21+00:00

There's nothing more practical than a good theory. - Kenneth Boulding

Reading a book is one thing. Applying what you learn is quite another. If you don't apply it soon, it simply fades away. The same is true of any educational experience. If you come back from a class and don't start using some of the material, you may as well not have gone in the first place.

The Challenge
Your challenge is to review the chapters in any of the four Quality Software Management volumes concerning specifics of the Anticipating organization and consider each idea in terms of the artistry that you can use to introduce it to your organization. Try to create at least one specific action item that will advance the transformation to that way of doing things.

Experiences
1. I started a brown bag special-interest group on our new CASE tool as a place for people who were using it to share learnings, and as a low-risk place for those who weren't using it to find out about it. The hardest part for me—and the real challenge—was to be the first speaker. I haven't been a person who enjoys speaking in front of groups, but I got some support and made myself do it. The group now runs on its own—with little nudges from me once in a while—and there's no trouble getting speakers. It has tripled in size as our use of the tool has grown, and people think that without the group the tool would have died in the original group, or at least not spread.

2. I set out to measure something that would be useful to upper management and to the people whose work was being measured. After a few false starts, I hit upon measuring resolution time for failures found in test. I set up a system to capture this data from our bug database and to plot it automatically week by week. One of the surprising things it showed was the way the new configuration management system actually slowed down resolution time. Since I was advocating the new system, I was rather disappointed, but I resisted the temptation to fudge the figures. Management wanted to throw the system out, but I invoked the Satir Change Model to get a few weeks grace period. With the help of some investigation into the causes of Chaos, the graph improved. In about three weeks, the resolution time was back to what it was before the tool, and after six weeks, the time was cut by 32%. This was the first time anyone had ever demonstrated the value of a new tool in our organization.

3. My challenge to myself was to open up information in my organization. To do this, I decided to be the model by using Public Project Progress Posters for the three projects I'm managing. I was surprised by the emotional reactions—mine and others'. I was apprehensive and defensive, yet proud of my courage. One of the other managers came into my office, shut the door, and started screaming obscenities at me for embarrassing him (because he wasn't going to post his progress). The people in the projects were generally accepting, though I spent a lot of time in the next two weeks explaining how to read the posters, what certain slippages meant, and what I was going to do about them. It was a lot more trouble than I anticipated, but now that things have settled down, it seems to be worth it.

Reference

This post is part of the series, adapted from the book, Becoming a Change Artist.

PHP Training in Montréal

2012-01-04T08:00:00+00:00

Together with my friends and partners from thePHP.cc, Arne Blankerts and Stefan Priebsch, I will be giving an "Advanced PHP Development" training in Montréal in March.

With all PHP topics, nothing counts more than their practical application. This is why thePHP.cc once again offers a highly interactive and practical training right before the ConFoo conference. Based on their own specific needs and questions, the attendees decide on the topics that are covered. They experience the development of new code at first hand, following their own agendas rather than those of the three trainers.

With a twinkle in their eyes and a dash of fun, Sebastian Bergmann, Arne Blankerts, and Stefan Priebsch explain development methods and tools, present trends and concepts, and perform joint code review sessions. This is efficient learning in a relaxed environment.

Topics covered include development best practices, object-oriented programming, design patterns, web application architecture, security, software quality and testing. All three trainers will be present at all times, so attendees have an unparalleled level of access to top-notch PHP expertise.

This training gives answers to every question you ever had about PHP and software development with PHP. Attendees should bring their own code and as many questions as they have.

As the training is scheduled right before the ConFoo conference and the topics covered complement its presentations, the Advanced PHP Development" training is the perfect preparation for attendees to get the most out of the conference.

Register now to secure your seat as the number of attendees is limited to ensure a beneficial learning environment.

Tester New Year’s Resolutions

2012-01-03T18:17:00+00:00

Dear project team,

This year I will…

target my testing to find you the right information sooner and trust your decision to ship early.
not test everything just because it is possible to test. Instead, I’ll spend my energy where I think my services are most valuable to you. I’ll tell you what I decide not to test and why. If you disagree, I will change my plan and test it.
consider the idea to stop executing tests that I’m 99.99% sure will pass.
not stress out about test deadlines. When I run out of time I will share that with you; in the past you have either jumped in to help or given me more time. It typically is not as terrible as I anticipate. Nevertheless, I promise not to be a slacker because it doesn’t feel nearly as good as being an over-achiever.
swallow my pride and ask you questions sooner, rather than hoping I understand later (even though I enjoy self-education via independent exploring and experimenting).
pay more attention to the business needs behind the things I test, as they will help me focus my test coverage.
look for ways to increase my value to you, like giving impromptu test reports, offering to log your bugs, and testing breadth before depth to at least catch the obvious ones early.
learn something about Selenium because everyone keeps talking about it and I feel dumb not knowing much about it. And who knows, someday I may have an opportunity to test a website for a change.
congratulate you on good work and take more interest in your achievements as BAs, Programmers, and CMs.
read that Data Warehouse Toolkit book by Kimball that you keep referring to. I’m sure much of it will be boring but I think it will help me respect your development efforts and determine new test ideas. It should also increase my Data Warehouse vocabulary.
squeeze time out of each day for learning something new about testing because fresh ideas make my job more interesting and make me a better tester. I will share these test ideas with you for the fun of it. Who knows, it may lead to something we can use here on a project.
stay late to meet deadlines or accommodate production releases…sometimes. Not often, hopefully. But I will do my time like others on our team and I will thank you when you work late. My personal time is important to me, therefore, it must also be important to you.
pay attention to my little tester light bulb that occasionally goes off with new thoughts. I will attempt to blog about these thoughts on my personal blog during non-work time.

How about you? Any Tester New Year’s Resolutions?

Why Checking Is Not Enough

2011-12-28T05:33:03+00:00

Here is a specific, real-world example of testing where the focus doesn’t include explicit checking, and does not result in yes-or-no answers to predetermined questions.

This morning, I acted on a piece of email I received several days ago, offering a free upgrade to a PDF conversion package which I’ll call “PDFThing”. I’ll walk you through what happened, and parts of my thought process as it happened.

Since the email is addressed to me, and since it notes that I had purchased upgrade insurance, I presume that the company has all of the data needed to know which product was associated with that email address.

The mail message includes this text: “It’ll only take a few minutes, but we’ll need your serial number (also known as a license key) to deliver your upgade. [How do I find my serial number?]” (The text in square brackets was a link.)

I note that “upgrade” is misspelled. Spelling can be checked, and a spelling checker would have found that problem, but checks can’t guarantee correct spelling. Do you doubt this? What if I had said that Czechs can’t guarantee correct spelling? Or that cheques can’t guarantee correct spelling? You would have noticed right away, but a spelling checker would not have.

I see a more serious potential problem, though. If the company has data about me, why not provide helpful serial number information directly and immediately? Options include “Your serial number is…” or if they’re worried about someone intercepting the mail, “The serial number can be found in your version of PDFThing by…”, in a way specific to the version that is associated with that email address.

I click on the link. It takes me to an FAQ page that has a list of questions. Conveniently, the question titled “Where do I find my serial number for PDFThing?” already shows an answer:

“It depends on what version of PDFThing you have and where you bought it. If you bought PDFThing 7 from our website, your serial number (in alpha-numeric characters) can be found under the Help tab in the About section.”

I have PDFThing 6, though. And I purchased it at a store. So I apply an oracle: consistency with an implicit purpose. An implicit purpose of this answer is to convey information to users of *any* version of PDFThing, purchased *anywhere*. The answer doesn’t do that. Is this a problem? I don’t know if the product owner will consider this a problem at all, or a problem worth fixing, so I can’t provide a yes-or-no answer. What I can do as a tester is to note a possible problem, and move on.

I decide to open my existing version PDFThing, and I apply another set of consistency heuristics: consistency with history; consistency within the product; consistency with an implicit claim; consistency within the product. Maybe the serial number for version 6 is located in the same place as the serial number for version 7. I click on Help and then About. I find that the serial number is not in the place referred to by the FAQ text, so with respect to the product I own, that text is misleading and incorrect. Plus I apply the “consistency with comparable product” heuristic; many products put the serial number in the Help/About box. All in all, this looks more like a problem. Will the product owner see it that way?

I dimly remember that I received a copy of the serial number in the e-mail that I got when I first registered the product. I go on a hunt for that e-mail. It takes me a few minutes to find it, but eventually I do. I copy the serial number to the clipboard and I returned to the original e-mail and click on upgrade to download the product. My own impatience and exasperation suggests to me that there’s a problem here. Note that although you can test for an emotional reaction, you can’t check for it. At best, you can anticipate things like delays of a certain duration as the program is executing. Measurement theorists call that a surrogate measure—using one kind of measureument as a substitute or a stand-in for the thing that we’d actually like to measure.

When the product finishes downloading, I begin the installation process. I’m prompted for the serial number for my older version, which I provide. The installer accepts the serial number and prompts me for a directory into which the new version of PDFThing should be installed. I notice that the product is being installed into a folder that is specific to version 7. I suspect that the prior version of PDFThing is not being uninstalled, so rather than accepting the default directory, I browse upward. I find that indeed the new product is not replacing the old product. Problem or no problem? For a check to determine this, the decision rule would have to have been decided and programmed in advance.

I go to Add/Remove programs, and begin the uninstallation process for PDFThing 6. The uninstaller posts a dialog saying, “The following applications should be closed before continuing the install”, and in the window beneath, I see a reference to the title of an email message I’m drafting in Outlook. That makes sense; PDFThing 6 installs a toolbar in Outlook so that I can print PDFs directly from the program. Still, there’s no reference to Outlook itself, though. So is that the message that the designer of the program wants me to see?

I close the offending message window and save the message as a draft. I return to the uninstallation dialog, press Retry, and the uninstallation proceeds. It appears to make some progress. I switch to another window and continue working while uninstallation continues in background. After a brief interval, it posts the same dialog as before, but this time tells me that Microsoft Word should be closed before continuing the install. Is this the behaviour that the designer wanted?

I now wonder what would have happened had I not chosen to uninstall PDFThing 6. Would Outlook and Word have acquired a second set of toolbars for PDFThing 7? Would they be separate? Would the new one have replaced the old one? I could have perhaps have programmed checks for that, but would it have been worthwhile to do that? Wouldn’t eyeballing it be cheaper and faster? Maybe not; maybe there are bunches of registry entries and files and configuration settings and stuff connected to Outlook (and Word, and Explorer, and PowerPoint, and Excel) such that we’d really need a program to help us probe that. Would a check have wondered and raised that issue to programmers or designers?

The installation process continues. In the middle, a browser window appears, asking me why I’m uninstalling PDFThing 6. The options are “I don’t want to purchase or continue with the trial”; “I purchased the product and am uninstalling the trial”; “I’m upgrading to the latest version”; “I’m about to move my PDFThing 6 license to another computer.” It seems to me that the third option would be unnecessary if PDFThing 7′s installation program automatically removed PDFThing 6. So is this the uninstallation process that the product owner wants?

I answer the question (I’m upgrading), and the Web page offers a thank you for answering the question. In the interim, the uninstallation process seems to have terminated. Was it successful? I don’t know. Did the designers intend that uninstallation should end immediately? And what if I hadn’t had an active Internet connection; what would have happened then? Would checks raise these questions? Perhaps the development of checks might have, but the checks themselves would not have.

I return to the installer for PDFThing 7, and start it up again. Oddly, I’m not asked for my serial number this time. Has the product retained it from the last attempt? I don’t know. How would I find out?

The installation process carries on for a while, and at the end, I’m presented with a dialog that asks whether I want to buy the product or activate it. I choose the latter; I’ve already bought it. The activation window asks for the serial number. I provide it, and immediately I’m presented with this error dialog (which I haven’t altered):

Note that the dialog is titled “Information”; the name of the product isn’t provided. Look as well at the formatting of the message; it looks sloppy and unprofessional to me. Oh, and dammit, it IS the right serial number (it was accepted last time). Is this what the product owner wants?

I dismiss the dialog, and the activation dialog has a moving graphic indicating that the product is waiting from something. Otherwise, the product seems hung. Just in case, though, I click on the Activate button again. The “Information” dialog above appears again. There’s no choice apparent except to dismiss the two dialogs and get on the line to technical support, whereupon the costs will really begin to rack up.

Now you could say that, if I were a tester working for the PDFThing people, I should have received all of this information before beginning test execution, whereupon I should have prepared checks to be applied against the product. It’s a fine idea. But even when we’re working on the best imaginable teams in the best-managed projects, as soon as we begin to test test, we begin immediately to discover things that no one—neither testers, designers, programmers, nor product owner—had anticipated or considered before testing revealed them. It’s simply fatuous to suggest that everyone involved in the development of the product knows exactly what they will want or need from the outset. It’s even more fatuous to suggest that they should know such a thing. Software development is not simply construction according to prescribed plans. It is development. Like testing itself, it is a process of exploration, discovery, investigation, and learning.

It’s important not to confuse checks with oracles. An oracle is a principle or mechanism by which we recognize a problem. A check is a mechanism, an observation linked to a decision rule. That rule is based on a single application of a single principle. A check provides a signal, a bit, when the product’s behaviour or state is inconsistent with that principle. A check follows a rule; it does not apply a heuristic. Testing, which may include many checks, is not so restricted. Testing may produce a yes-or-no answer, but it may also produce an observation, a question, a concern, a dilemma, a new test idea, or a new check idea. Testing is not governed by rules; it is governed by heuristics that, to be applied appropriately, require sapient awareness and judgement.

Checking is an approach to making sure that we get the right answers, for questions and desired answers that we’ve already determined in advance. A passing check doesn’t tell us that the product is acceptable. At best, a check that doesn’t pass suggests that there is a problem in the product that might make it unacceptable.

Testing incorporates checking, but is a far richer set of activities: exposing ourselves to the unexpected, making new observations, spotting unanticipated problems, and raising new questions. Yet not even testing is about telling people that the product is acceptable. On the one hand, testing may have a different purpose. Cem Kaner, in the BBST course, lists

Finding defects
Maximizing bug count
Blocking premature product releases
Helping managers make ship / no-ship decisions
Minimizing technical support costs
Assessing conformance to specification
Assessing conformance to regulations
Minimizing safety-related lawsuit risk
Finding safe scenarios for use of the product (workarounds that make the product potentially tolerable, in spite of the bugs)
Assessing quality
Verifying the correctness of the product

To which I would add

assessing compatibility with other products or systems
assessing readiness for internal deployment
ensuring that that which used to work still works
design-oriented testing, such as review or test-driven development
understanding the workings of a poorly-documented product or library
evaluating the usefulness of a new tool or service
refining notions of risk

On the other hand, much of the time, we’re testing to help determine whether a product is acceptable for release. But decisions about acceptability are in the hands of managers, programmers, designers; those who build the product (and ultimately, acceptability is the decision of the product owner). Testing is about investigating the product to reveal knowledge that informs the acceptability decision. Sometimes that information comes in the form of binary answers to known questions; checks. Sometimes that information comes in the form of discoveries that pose new ideas, new risks, and new questions for those who are responsible for building and releasing the product.

2nd Annual QA/Test Job Posting Study

2011-12-23T20:38:34+00:00

This is a guest blog post by Daniel Frank, my assistant. Daniel took on the challenge of updating the QA/Test job study for 2011, just in time for making New Year’s resolutions. Enjoy! Elisabeth

It’s been a little over a year since Elisabeth published “Do Testers Have to Write Code,” the results of an in-depth survey of job ads that she and Melinda conducted to see if employers expect testers to program. The resounding conclusion, with 80% of tester job ads requesting some kind of programming skill, was “Yes.”

This year we wanted to see if things have changed, so I conducted the same study again. I also wanted to add a bit more granularity to the study, to see if there were any trends that were missed last time.

I screened the lists with the same basic guidelines as our previous study. That means I restricted my search to the US only. I only counted a job if it was described as a testing/QA position in the job title. I did not include recruiter listings in order to avoid the risk of including duplicate jobs or even fake jobs used to gather pools of applicants.

Our final sample size this year is 164 jobs. That’s a little less than last year. Why?

The lists were sparse. There just aren’t that many job ads out there. Many of the job ads I found were from recruiters or were repeats, with the same company listing the same position several weeks in a row.

The simple fact that I had a hard time finding the same number of ads as last year is interesting information all on its own. From an overall economic standpoint, the country is in no more of a slump than we were in 2010. So why are there fewer listings for testers? Could it be that Alberto Savoia, who recently declared testing dead, is correct? We’ll come back to that question later.

Back to the study…

Like last year, the majority of our jobs came from Craigslist (90) and LinkedIn (64). The rest of them came from a smattering of other sites.

The data includes an even higher proportion of jobs in California than last year: 102 of the listings were in CA, with the remainder divided in small chunks between 28 other states. Unsurprisingly,Texas, Massachusetts, and Washington are the three runners up.

Last year there was some question of whether or not the sample was biased simply because we’re located in California. However, I took extra steps to try and get equal representation. The simple fact is that a search that might find 70 jobs when I filter the location for CA will result in 30 jobs or fewer if I filter for another area. If anything, I’d estimate that California is actually under represented.

I kept track of the job titles. By far the most popular title is “QA Engineer” (99 of the listings). 136 of the titles contained “QA” compared with only 32 containing the word “Test.”

An interesting side note: when I searched for the word “test” in the body of job ads, I found far more developer positions than similar searches for “qa” did. It would seem that at the same time QA/Test positions are requiring more coding skills, developer positions are requiring more testing skills. That might be another interesting job ad survey project.

So how much coding are testers expected to do?

Of the 164 listings, 102 jobs say they require knowledge of at least one programming language, and 38 jobs indicate coding is a nice to have. That’s 140 out of 164, or 85.37% of the sample. That’s an even higher percentage than last year. It’s difficult to say if the 5% uptick represents a real increase in demand, but at the very least it’s fair to say that demand for testers who code remains high.

I used the same criteria that Elisabeth and Melinda used last year. That means that I counted a job as requiring programming if the job required experience in or knowledge of a specific language, or if the job duties mentioned a language. There were 7 jobs which listed broad experience requirements like “must be able to script in multiple languages,” which also counted as requiring programming.

There were some judgment calls to be made about what may or may not count as a programming language. For the purpose of the results here, I counted SQL or other relational database knowledge as a programming language in order to be consistent with last year. However, unlike last year, I tracked proficiency in relational databases separately. This will let me track specific trends more easily in future studies.

One of the questions Elisabeth wanted to answer last year was whether jobs with self-identified Agile organizations required testers to code more than other jobs. This year 46 of of the 58 Agile job ads list programming skills as required or nice to have. That’s 79.31%, which is actually a lot less than last year’s 90%. However, this is one of those places where the small sample size has to be taken into consideration. In 2010, 49 out 55 agile jobs mentioned programming. Today, 46 out of 58 jobs mention it. Just a few jobs result in a 10% variation.

An enduring question about any kind of job is how much it pays. I saw even less mentions of pay this time around. Only 7 jobs even listed it, and 5 of those were button-pushing game testing positions in the $10-$20/hour range. The other two ran around $85,000-$105,000. Most positions simply don’t provide up front salary estimations, so we cannot draw any real conclusions from these data points.

Just for fun, I also noted whenever a job requested a certification. In 164 jobs I found exactly 4 mentions of certification, and not a single one was required. 3 of them were vendor or technology certifications that had nothing to do with testing. And even in the single instance where a testing certification was nice to have, it was the CSTE offered by QAI, rather than the much more hyped ISTQB. So it would seem that testing certifications are not much in demand. The bottom line is that someone looking to improve their marketability would be much better served by upskilling to a new proficiency rather than picking up an irrelevant certification.

And that’s about it for our study. If you’d like to dig through the raw data to look for any trends I may have missed, I’ll be happy to send it to you. Drop me a line.

Now back to the question about the number of QA/Test jobs out there. Could it be that there are fewer QA/Test positions? Was this just a matter of luck and timing, or is there a trend?

Alberto Savoia gave a talk titled “Test is Dead” at GTAC (dressed as the Grim Reaper). He may have used intentionally inflammatory hyperbole to make his point, but that doesn’t change the fact that he had interesting points to make.

Alberto points out that especially in web development, speed is paramount. Further, the biggest challenge isn’t in building “it” right, but in building the right “it.” So the goal is to get a minimum viable product out as quickly as possible, and get fast feedback from real users and customers. Traditional black box testing ends up taking a back seat in this type of development, and these projects often rely heavily on user feedback instead.

At STARWest 2011, James Whittaker of Google gave a talk titled “All That Testing is Getting in the Way of Quality” where he talked about the closest thing to a traditional testing role they have at Google. It’s called the “Test Engineer,” and they spend anywhere from 20%-80% of their time writing code. He also explains how Google utilizes their user bases to do almost all of their exploratory tests. As he puts it, “Users are better at being users than testers are, by definition.”

With James and Alberto’s talks firmly in mind, I can’t help but wonder if the difficulty I experienced in finding job ads that met my criteria is indicative of a sea-change in the industry rather than an anomaly. Could it be that we’re seeing a reduction in the number of QA/Test positions?

What do you think? Are you seeing fewer QA/Test positions in your organization or (if you’re looking) in your job search?

What Exploratory Testing is Not

2011-12-22T04:43:06+00:00

Michael Bolton has gone off like a volcano in Iceland, writing a series about what exploratory testing isn’t:

http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-1-touring

http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-2-after-everything-else-testing

http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-3-tool-free-testing

http://www.developsense.com/blog/2011/12/what-exploratory-testing-is-not-part-4-quick-tests

http://www.developsense.com/blog/2011/12/what-exploratory-testing-isnt-part-5-undocumented-testing

Another thing I would add to this:

Exploratory testing is not defined by any specific example of exploratory testing.

Just as tap dancing does not characterize ballroom dancing, you can’t take any one example of exploratory testing and treat that as representative of the entire concept of ET.

If you were to hear me singing an aria by Mozart, that would be an example of opera singing. It would be an example of BAD opera singing, but it would truly be an example of the style. Similarly, I regularly talk to testers who go “oh yeah I’ve seen that exploratory testing stuff but it’s not structured… not documented… not x… not y… not whatever.” And my reply is “you probably haven’t seen skilled exploratory testing. Would you like to hear me sing an opera now? OR, I could show you a good example of ET in practice.”

Exploratory testing can be done in an unskilled, slapdash, silly way. Just as a unskilled driver behind the wheel of a car is still a driver who is driving a car, a poor tester can still be doing ET– albeit probably not very well.

The cool thing about ET is that, even done badly, it’s still a great way to find some bugs. Michael and I try to help you do much, much better than that.

The core idea of ET remains as it always has been. It’s been expressed in many different ways, but boils down to this: test design and test execution and learning mixed together in a mutually supportive way. Whenever you see that, and to the degree that you see that, you are seeing exploratory testing.

Test Idea Wall

2011-12-20T18:43:00+00:00

After seeing Mark Vasko’s CAST 2011 lightning talk, I was inspired to create a Test Idea Wall with one of my project teams. Much to my surprise, the damn thing actually works.

When I’m taking a break from testing something, I pause as I walk past the Test Idea Wall. My brain jumps around between the pictures and discovers gaps in my test coverage.

Our wall is incredibly simple, but so far it contains the main test idea triggers we forget. For example, the picture of the pad lock reminds us to consider locking scenarios, something that is often just an afterthought, but always gets us fruitful information:

What if we run the same tests as a read-only user?
What if we run the same tests while another user has our lock?
What if we run the same tests while the system has our lock?
What if certain users should not have this permission?

Thanks, Mark!

What Exploratory Testing Is Not (Part 2): After-Everything-Else Testing

2011-12-20T09:26:37+00:00

Exploratory testing is not “after-everything-else-is-done” testing. Exploratory testing can (and does) take place at any stage of testing or development.

Indeed, TDD (test-driven development) is a form of exploratory development. TDD happens in loops, in which the programmer develops a check, then develops the code to make the check pass (along with all of the previous checks), then fixes any problems that she has discovered, and then loops back to implementing a new bit of behaviour and inventing a new check. The information obtained from each loop feeds into the next; and the activity is guided and structured by the person or people involved in the moment, rather than in advance. The checks themselves are scripted, but the activity required to produce them and to analyze the results is not. Compared to the complex cognitive activity—exploratory, iterative—that’s going on as code is being developed, the checks themselves—scripted, linear—are trivial.

Requirement review is an exploratory activity too. Review of requirements (or specifications, or user stories, or examples) tends happens early on in a development cycle, whether it’s a long or a short cycle. While review might be guided by checklists, the people involved in the activity are making decisions on the fly as they go through loops of design, investigation, discovery, and learning. The outcome of each loop feeds back into the next activity, often immediately.

Code review can also be done in a scripted way or an exploratory way. When humans analyze the code, it’s an unscripted, self-directed activity that happens in loops; so it is exploratory. We call it review, but it’s gathering information with the intention of informing a decision; so it is testing. There is a way to review code that involves the application of scripted processes, via a tools that people generally call “static testing tools. When a machine parses code and produces a report, by definition it’s a form of checking, and it’s scripted. Yet using those tools productively requires a great deal of exploratory activity. Parsing and interpreting the report and responding to it is polimorphic, human action—unscripted, open-ended, iterative, and therefore exploratory.

Learning about a new product or a new feature is an exploratory activity if you want to do it or foster it well. Some suggest that test scripts provide a useful means of training testers. Research into learning shows that people tend to learn more quickly and more deeply when their learning is based on interaction and feedback; guided, perhaps, but not controlled. If you really want to learn about a product, try creating a mind map, documenting some aspect of the program’s behaviour, or creating plausible scenarios in which people might use—or misuse—the product. All of these activities promote learning, and they’re all exploratory activities. There’s far more information that you can use, apply, and discover than a script can tell you about. Come to think of it… where does the script come from?

Developing a test procedure—even developing a test script, whether for a machine or a human to follow, or developing the kind of “test” that skilled testers would call a demonstration—is an exploratory activity. There is no script that specifies how to write a new script for a particular purpose. Heard about a new feature and pondering how you might test it? You’ve already begun testing; you’re doing test design and you’re probably learning as you go. To the extent that you use the product or interact with it, bounce ideas off other people, or think critically about your design, you’re testing, and you’re doing it in an unscripted way. Some might suggest that certain tools create scripts that can perform automatic checks. Yet reviewing those checks for appropriateness, interpreting the results, and troubleshooting unexpected outcomes are all exploratory activities.

Supposing that a programmer, midway through a sprint, decides that she’d like some feedback on the work that she’s done so far on a new module. She hands you a bit of code to look at. You might interact with the code directly through a test tool that she provided, or (say) via the Ruby interpreter, or you might write some script code to exercise some of the functions in the module. In any event, you find some problems in it. In order to investigate a problem that you’ve discovered, you must explore. You must explore whether your recognition of the problem was triggered by your own interaction with the program or by a mechanically executed script. You’re in control of the activity; each new test around the problem feeds back into your choice of the next activity, and into the story that you’re going to tell about the product.

All of the larger activities that I’ve described above are exploratory, and they all happen before you have a completed function or story or sprint. Exploratory testing is not a stage or phase of testing to be performed after you’ve performed your other test techniques. Exploratory testing is not an “other” test technique, because it’s not a technique at all. Exploratory testing is not a thing that you do, but rather a way that you work (and think, and act), the hallmarks being who (or what) is in control, and the extent to which your activity is part of a loop, rather than a straight line. Any test technique can be applied in a scripted way or in an exploratory way. To those who say “we do exploratory testing after our acceptance tests are all running green”, I would suggest looking carefully and observing the extent to which you’re doing exploratory testing all the way along.

Worthwhile Documentation

2011-12-20T03:10:05+00:00

In the Rapid Software Testing class, we focus on ways of doing the fastest, least expensive testing that still completely fulfills the mission. That involves doing some things more quickly, and it also involves doing other things less, or less wastefully. One of the prime candidates for radical waste reduction is documentation that’s incongruent with the testing mission.

Medical device projects typically present a high degree of risk. Excellent testing helps teams and product owners to identify risks and problems in the product. The quality of testing is a function of the skill of the tester; one would not set loose an incapable tester on high-risk project. Yet some managers have told me that they commission people to write test documentation in a particular style. That style is, to me, overly elaborate and specific with respect to actions to perform and observations to make. Yet at the same time, that style is remarkably devoid of ideas about motivation or risk.

I sometimes ask managers why they use this style of instruction. They usually answer, “because we want anyone to be able to walk up to this system and test it.”

“Anyone?” I ask. “Why anyone?”

“You know how it is. If we have to test a new revision of this program a year from now, there’s a good chance that we won’t have the same testers.” (Dude. If you’re inflicting on your staff the idea of testing as writing or following instructions for an automaton, I might have an explanation for you.)

“Anyone?” I ask. “How about a cat?”

“Well, Michael, that’s silly. Cats can’t think. Cats can’t read.”

“How about my daughter? She’s seven, and she can read well enough to read that. And she could follow the steps pretty well, too.”

“We don’t hire children here!”

“Okay,” I offer. “Would you hire a completely incompetent tester who needed to be told absolutely everything, in painful detail?”

“We wouldn’t hire anyone like that.”

“Fair enough, and I’d hope not. So, why do you insist that people write instructions for them that way?”

Let me be clear: when the situation calls for skilled testers, you don’t need overly specific instructions for them. On the other hand, if you don’t have skilled testers, you’ve got a problem that scripted testing won’t be able to solve.

Here’s a splendid example of a machete that we believe that managers could use to cut through jungles of waste. In a recent project that involved work with FDA-regulated medical devices, James Bach found a huge number of excruciatingly overspecified, low-value test cases aimed at “anyone”. The following two paragraphs replaced 50 pages of waste.

3.0 Test Procedures

3.1 General Testing Protocol

In the test descriptions that follow, the word “verify” is used to highlight specific items that must be checked. In addition to those items, the tester shall at all times be alert for any unexplained or erroneous behaviour of the product. The tester shall bear in mind that, regardless of any specific requirement for any specific test, there is the overarching general requirement that the product shall not pose an unacceptable risk of harm to the patient, including an unacceptable risk due to reasonably foreseeable misuse.

Test personnel requirements: The tester shall be thoroughly familiar with the Generator and Workstation Function Requirement Specifications, as well as the working principles of the devices themselves. The tester shall also know the workings of the power test jig and associated software, including how to configure and calibrate it and how to recognize it is not working correctly. The tester shall have sufficient skill in data analysis and measurement theory to make sense of statistical test results. The tester shall be sufficiently familiar with test design to complement this protocol with exploratory testing in the event that anomalies appear that require investigation. The tester shall know how to keep test records to a credible professional standard.

To me, that’s something worth writing down. Follow those instructions, and your team will save time, save work, and put the emphasis in the right places: on risk, and on meeting and mitigating that risk with skills.

What Exploratory Testing Is Not (Part 4): Quick Tests

2011-12-19T15:46:44+00:00

Quick testing is another approach to testing that can be done in a scripted way or an exploratory way. A tester using a highly exploratory approach is likely to perform many quick tests, and quick tests are often key elements in an exploratory approach. Nonetheless, quick testing and exploratory testing aren’t the same.

Quick tests are inexpensive tests that require little time or effort to prepare or perform. They may not even require a great deal of knowledge about the application being tested or its business domain, but they can help to develop new knowledge rapidly. Rather than emphasizing comprehensiveness or integrity, quick tests are designed to reveal information in a hurry at a minimal cost.

A quick test can be a great way to learn about the product, or to identify areas of risk, fragility, or confusion. A tester can almost always sneak a quick test or two into other testing activity. A burst of quick tests can help as the first activities during a smoke or sanity test. Cycles of relatively unplanned, informal quick tests may help to you discover or refine a more comprehensive or formal plan.

James Bach and I provide examples of many kinds of quick tests in the Rapid Software Testing class. You’ll notice that some of them are called tours. Note that not all tours are quick, and not all quick tests are tours. Here’s a catalog.

Happy Path
Perform a task, from start to finish, that an end-user might be expected to do. Use the product in the most simple, expected, straightforward way, just as the most optimistic programmer or designer might imagine users to behave. Look for anything that might confuse, delay, or irritate a reasonable person. Cem Kaner sometimes calls this “sympathetic testing”. Lean towards learning about the product, rather than finding bugs. If you do see obvious problems, it may be bad news for the product.

Variable Tour
Tour a product looking for anything that is variable and vary it. Vary it as far as possible, in every dimension possible. If you’re using quick tests for learning, seek and catalog the important variables. Look for potential relationships between them. Identifying and exploring variations is part of the basic structure of our testing when we first encounter a product.

Sample Data Tour
Employ any sample data you can, and all that you can. For one kind of quick tests, prefer simple values whose effects are easy to see or calculate. For a different kind of quick test, choose complex or extreme data sets. Observe the units or formats in which data can be entered, and try changing them. Challenge the assumption that the programmers have thought to reject or massage inappropriate data. Once you’ve got a handle on your ideas about reasonable or mildly challenging data, you might choose to try…

Input Constraint Attack
Discover sources of input and attempt to violate constraints on that input. Try some samples of pathological data: use zeroes where large numbers are expected; use negative numbers where positive numbers are expected; use huge numbers where modestly-sized ones are expected; use letters in every place that’s supposed to handle only numbers, and vice versa. Use a geometrically expanding string in a field. Keep doubling its length until the product crashes. Use characters that are in some way distinct from your idea of “normal” or “expected”. Inject noise of any kind into a system and see what happens. Use Satisfice’s PerlClip utility to create strings of arbitrary length and content; use PerlClip’s counterstring feature to create a string that tells you its own length so that you can see where an application cuts off input.

People tend to talk a lot about input constraint attacks. Perhaps that’s because input constraint attacks are used by hackers to compromise systems; perhaps it’s because input constraint attacks can be performed relatively straightforwardly; perhaps it’s because they can be described relatively easily; perhaps it’s because input constraint attacks can produce dramatic and highly unexpected results. Yet they’re by no means the only kind of quick test, and they’re certainly not the only way to test using an exploratory approach.

Documentation Tour
Look in the online help or user manual and find some instructions about how to perform some interesting activity. Do those actions explicitly. Then improvise from them and experiment. If your product has a tutorial, follow it. You may expose a problem in the product or in the documentation; either way, you’ve found an inconsistency that is potentially important. Even if you don’t expose a problem, you’ll still be learning about the product.

File Tour
Have a look at the folder where the program’s .EXE file is found. Check out the directory structure, including subs. Look for READMEs, help files, log files, installation scripts, .cfg, .ini, .rc files.
Look at the names of .DLLs, and extrapolate on the functions that they might contain or the ways in which their absence might undermine the application. Use whatever supplemental material you’ve got to guide or focus your actions. Another way to gather information for this kind of test: use tools to monitor the installation, and take the output from the tool as a point of departure.

Complexity Tour
Tour a product looking for the most complex features, the most challenging data sets, and the biggest interdepencies. Look for hidden nooks and crannies, but also look for the program’s high-traffic areas, busy markets, big office buildings, and train stations—places where there’s lots of interactions, and where bugs might be blending in with the crowd.

Menu, Window, and Dialog Tour
Tour a product looking for all the menus (main and context menus), menu items, windows, toolbars, icons, and other controls. Walk through them. Try them. Catalog them, or construct a mind map.

Keyboard and Mouse Tour
Tour a product looking for all the things you can do with a keyboard and mouse. Hit all of the keys on the keyboard. Hit all the F-keys; hit Enter, Tab, Escape, Backspace; run through the alphabet in order, and combine each key with Shift, Ctrl, Alt, the Windows key, CMD or Option, on other platforms, the AltGr key in Europe. Click (right, left, both, double, triple) on everything. Combine clicks with shifted keys.

Interruptions
Start activities and stop them in the middle. Stop them at awkward times. Perform stoppages using cancel buttons, O/S level interrupts (ctrl-alt-delete or task manager). Arrange for other programs to interrupt (such as screensavers or virus checkers). Also, try suspending an activity and returning later. Put your laptop into sleep or hibernation mode.

Undermining
Start using a function when the system is in an appropriate state, then change the state part way through.
Delete a file while it is being edited; eject a disk; pull net cables or power cords) to get the machine an inappropriate state. This is similar to interruption, except you are expecting the function to interrupt itself by detecting that it no longer can proceed safely.

Adjustments
Set some parameter to a certain value, then, at any later time, reset that value to something else without resetting or recreating the containing document or data structure. Programmers often expect settings or variables to be adjusted through the GUI. Hackers and tinkerers expect to find other ways.

Dog Piling
Whatever you’re doing, do more of it and do other stuff as well while you’re at it. Get more processes going at once; try to establish more states existing concurrently. Invoke nested dialog boxes and non-modal dialogs. On multi-user systems, get more people using the system or simulate that with tools. If your test seems to trigger odd behaviour, pile on in the same place until the odd becomes crazy.

Continuous Use
While testing, do not reset the system. Leave windows and files open; let disk and memory usage mount.
You’re hoping to show that the system loses track of things or ties itself in knots over time.

Feature Interactions
Discover where individual functions interact or share data. Look for any interdependencies. Explore them, exploit them, and stress them out. Look for places where the program repeats itself or allows you to do the same thing in different places. For example, for data to be displayed in different ways and in different places, and seek inconsistencies. For example, load up all the fields in a form to their maximums and then traverse to the report generator.

Summon Help
Bring up the context-sensitive help feature during some operation or activity. Does the product’s help file explain things in a useful way, or does it offend the user’s intelligence by simply restating what’s already on the screen? Is help even available at all?

Click Frenzy
Ever notice how a cat or a kid can crash a system with ease? Testing is more than “banging on the keyboard”, but that phrase wasn’t coined for nothing. Try banging on the keyboard. Try clicking everywhere. Poke every square centimeter of every screen until you find a secret button.

Shoe Test
Use auto-repeat on the keyboard for a very cheap stress test. Look for dialog boxes constructed such that pressing a key leads to, say, another dialog box (perhaps an error message) that also has a button connected to the same key that returns to the first dialog box. Place a shoe on the keyboard and walk away. Let the test run for an hour. If there’s a resource or memory leak, this kind of test could expose it. Note that some lightweight automation can provide you with a virtual shoe.

Blink Test
Find an aspect of the product that produces huge amounts of data or does some operation very quickly. Look through a long log file or browse database records, deliberately scrolling too quickly to see in detail. Notice trends in line lengths, or the look or shape of the data. Use Excel’s conditional formatting feature to highlight interesting distinctions between cells of data. Soften your focus. If you have a test lab with banks of monitors, scan them or stroll by them; patterns of misbehaviour can be surprisingly prominent and easy to spot.

Error Message Hangover
Programmers are rewarded for implementing features. There’s a psychological problem with errors or exceptions: the label itself suggests that something has gone wrong. People often actively avoid thinking about problems or mistakes, and as a consequence, programmers sometimes handle errors poorly. Make error messages happen and test hard after they are dismissed. Watch for subtle changes in behaviour between error and normal paths. With automation, make the same error conditions appear thousands of times.

Resource Starvation
Progressively lower memory, disk space, display resolution, and other resources. Keep starving the product until it collapses, or gracefully (we hope) degrades.

Multiple Instances
Run a lot of instances of the app at the same time. Open, use, update, and save the same files. Manipulate them from different windows. Watch for competition for resources and settings.

Crazy Configs
Modify the operating system’s configuration in non-standard or non-default ways, either before or after installing the product. Turn on “high contrast” accessibility mode, or change the localization defaults. Change the letter of the system hard drive. Put things in non-default directories. Use RegEdit (for registry entries) or a text editor (for initialization files) to corrupt your program’s settings in a way that should trigger an error message, recovery, or an appropriate default behavior.

Again: quick tests tend to be highly exploratory, but they represent only one kind of exploratory testing. Don’t be fooled into believing that quick testing—or certain kinds of quick testing—is all there is to exploratory testing.

TDD Workflow (Sinatra / Haml / jQuery) Part 1

2011-12-17T23:08:25+00:00

Introduction

This is a draft. Worth continuing the series? Let me know.

Critter4Us is a webapp used to schedule teaching animals at the University of Illinois Vet School. Its original version was a Ruby/Sinatra application with a Cappuccino front end. Cappuccino lets you write desktop-like webapps using a framework modeled after Apple’s Cocoa. I chose it for two reasons: it made it easy to test front-end code headlessly (which was harder back then than it is now), and it let me reuse my RubyCocoa experience.

Earlier this year, I decided it was time for another bout of experimentation. I decided to switch from Cappuccino to jQuery, Coffeescript, Haml because I thought they were technologies I should know and because they’d force me to develop a new TDD workflow. I’d never gotten comfortable with the testing—or for that matter, any of the design—of “traditional” front-end webapp code and the parts of the backend from the controller layer up.

I now think I’ve reached a plateau at which I’m temporarily comfortable, so this is a good time to report. Other people might find the approach and the tooling useful. And other people might explain places where my inexperience has led me astray.

The example

Suppose an instructor wants to reserve the horses Genesis and Guicho each Tuesday morning from October 8 through the 29th. For historical reasons, that’s done by making a reservation for the 8th and then making three copies. Here’s what that page looks like today:

(As you can see, I haven’t gotten to making the UI attractive yet.)

After the stop date is chosen, a click of the “Duplicate” button causes three successive Ajax calls to be sent to the server. (That’s a bit gratuitous - I just like the idea of the UI showing each reservation “coming to life” in turn.)

Ideally, Genesis and Guicho will be reserved for each Tuesday. However, it’s possible that one of them has already been reserved for the 22nd. In that case, the reservation is still made, but the animal is dropped from it. The user has to edit that reservation to pick a replacement animal.

Testing the Haml

I usually test in an outside-in, mock-heavy style derived from Growing Object-Oriented Software (GOOS). “Derived”, naturally, means I depart from GOOS in my very first step: I avoid automating end-to-end tests. For my small-scale purposes, I don’t find GUI tests (Watir, Selenium, etc.) worth the price. To be clear: I think about GOOS-style tests, and I execute them—I just don’t automate them.

Having thought of an end-to-end test for a particular user task, the next step is to sketch out a page that supports it. I sketch it out in Haml to be the beginnings of the page above.

My page testing is driven by this observation: the user will use the same page for many different reservations. It has a constant part that’s independent of the reservation, and a part that depends on the particular reservation. I only write automated tests for the variable part. I test the constant part by looking at it, by looking at the generated HTML, and by trying the finished story by hand.

There are three parts of this particular page that vary:

There’s a snippet of text that reminds the user what reservation she’s working with.
There has to be a starting date used to set up the calendar.
There has to be some URI that’s used to PUT a new repetition into the system. That URI will look something like /reservations/441/repetitions.

Here’s the setup for tests to check those three claims:

Whenever I create a model object, I usually give it a random constructor that builds it with innocuous values unless I give it explicit ones. I’ll explain why I’m giving it only one value below.
I hate the spooky action at a distance of setting instance variables in a controller in order to have some effect way off in another file written in a different language. I’d much rather have a nice explicit function-call interface and a builder or transformation approach. Most of the world thinks differently, so I concede. However, I do try to come closer to a function-call interface by passing in locals instead of setting instance variables.
Passing in a whole Reservation model object makes me queasy for reasons Bob Martin has explained. I’m intruding the model into the view rather than passing in just what the view needs. This is a consequence of some amount of laziness after bowing to the peer pressure to use partials. I do make some effort to separate presentation-relevant data from model-relevant objects, as you’ll see later.
In order not to repeat myself, I make one Ruby file the authority over all URIs and links. Therefore, any URI gets passed into the HAML. In pages like these, which exist to aid some specific user task and have a single URI that they use when the appropriate data has been supplied by the user, I call that single URI the “fulfillment” link.

That given, here’s the test in which I claim that a reservation summary appears on the page:

I use shoulda because I like nested contexts and strings to name tests, but I don’t need the rest of Rspec. I use Assert{2.0} because I find Hamcrest-style matchers too wordy and Assert{2.0} gives me usefully detailed failure messages. (It prints every subexpression of a false expression. Wrong is a more recent implementation of this idea.)

The actual reservation summary is created with a partial. By checking for ‘Dr. Dawn’, this test claims the partial has been “called”. (It’s thus a clumsier and less explicit example of a mockish test.) I’m testing only that here because I want the freedom to what the summary looks like without having to change this test.

The next test claims that a correct Javascript date exists in the HTML (presumably in some Javascript code):

Notice that the check is, again, minimalist. The test isn’t about how the date is used, just that it exists. Peeking ahead, it’s actually used like this:

There’s a gap in the test coverage, in that no automated test checks that this initialization code is correct. Pretty low risk, given that it’s straight-line code that cannot avoid execution whenever the page is loaded. So long as I try this page out once, and remember to try the page again if I change the initialization code, I should be safe.

The final test is this:

assert_text_has_attributes is part of a thin wrapper on top of the HTML parsing part of Capybara (not the integration testing part). I used to use assert_xhtml, a sweet package, but it’s dead in Ruby 1.9 and I haven’t found another one like it. Let me know if you have one.

The Haml code that passes these tests looks like this:

The <head> is, annoyingly, generated by another action at a distance, with a layout.haml file that gets wrapped around the above code. That file uses the third of the “parameters”:

What now?

The static page and the acceptance test I have in mind call for some Javascript. That needs to be tested into existence.

What Exploratory Testing Is Not (Part 3): Tool-Free Testing

2011-12-17T20:29:08+00:00

People often make a distinction between “automated” and “exploratory” testing. This is like the distinction between “red” cars and “family” cars. That is, “red” (colour) and “family” (some notion of purpose) are in orthogonal categories. A car can be one colour or another irrespective of its purpose, and a car can be used for a particular purpose irrespective of its colour. Testing, whether exploratory or not, can make heavy or light use of tools. Testing, whether it entails the use of tools or not, can be highly scripted or highly exploratory.

“Exploratory” testing is not “manual” testing. “Manual” isn’t a useful word for describing software testing in any case. When you’re testing, it’s not the hands that do the testing, any more than when you’re riding a bike it’s the feet that do the bike-riding. The brain does the testing; the hands, at best, provide one means of input and interaction with the thing we’re testing. And not even “manual” testing is manual in the sense of being tool- or machinery-free. You do you use a computer when you’re testing, don’t you?

(Well, mostly, but not always. If you’re reviewing requirements, specifications, code, or documentation, you might be looking at paper, but you’re still testing. A thought experiment or a conversation about a product is a kind of a test; you’re questioning something in order to evaluate it, pitting ideas against other ideas in an unscripted way. While you’re reviewing, are you using a pen to annotate the paper you’re reading? A notepad to record your observations? Sticky tabs to mark important places in the text? Then you’re using tools, low-tech as they might be.)

Some people think of test automation in terms of a robot that pounds on virtual keys more quickly, more reliably, and more deterministically than a human could. That’s certainly one potential notion of test automation, but it’s very limiting. That traditional view of test automation focuses on performing checks, but that’s not the only way in which automation can help testing.

In the Rapid Software Testing class, James Bach and I suggest a more expansive view of test automation: any use of tools to support testing. This helps keeps us open to the idea that machines can help us with almost any of the mimeomorphic, non-sapient aspects of testing, so that we can focus on and add power to the polimorphic, sapient aspects. Exploration is polimorphic activity, but it can include and be supported by mimeomorphic actions. Cem Kaner and Doug Hoffman take a similar tack: exploratory test automation is “computer-assisted testing that supports learning of new information about the quality of the software under test.” Learning new information is one of the hallmarks of exploratory testing, which usually points towards emphasizing variation rather than repetition.

That said, there can be a role for mechanized repetition, even when you’re using a highly exploratory approach: when repeating aspects of the test are intended to support discovery of something new or surprising. The key is not whether you’re mechanizing the activity. The key is what happens at the end of the activity. The less the results of one activity are permitted to inform the next, the more scripted the approach. If the repetition is part of a learning loop—a cycle of probing, discovering, investigating, and interpreting—that feeds back on itself immediately, then the approach is exploratory. James has also posted a number of motivations for repeating tests. Each one can (with the possible exception of “avoidance or indifference”) be entirely consistent with and supportive of exploration.

There are some actions that tools can perform better than humans, as long as the action doesn’t require human judgment or wisdom. Humanity can even get in the way of some desirable outcome. For example, when your exploration of some aspect of a product is based on statistical analysis, and randomization is part of the test design, it’s important to remember that people are downright lousy at generating randomized data. Even when people believe that they’re choosing numbers at random, there are underlying (and usually quite unconscious) patterns and biases that inform their choices. If you want random numbers, tools can help.

Tools can support exploration in plenty of other ways: data generation, system configuration; simulation; logging and video capture; probes that examine the internal state of the system; oracles that detect certain kinds of error conditions in a product or generate plausible results for comparison; visualization of data sets, key elements to observe, relationships, or timing; recording and reporting of test activity.

A few years back, I was doing testing of a teller workstation application at a bank (I’ve written about this in How to Reduce the Cost of Software Testing). The other testers, working on domestic transactions, were working from scripts that contained painfully detailed and explicit steps and observations. (Part of the pain came from the fact that the scripts were supplemented with screen shots, and the text and the images didn’t always agree.) My testing assignment involved foreign exchange, and the testing tasks I had been given were unscripted and, to a large degree, self-determined. In order to learn the application quickly, I had to explore, but this in no way meant that I didn’t use tools. On the contrary, in fact. In that context, Excel was the most readily available and powerful tool on hand. I used it (and its embedded Visual Basic for Applications) to:

maintain and update (at a key stroke) enormous tables of currencies, rates, and transaction types
access appropriate entries from the table via regular expression parsing
model the business rules of the application under test
display the intended flow of money through a transaction
add visual emphasis to the salient outcomes of tests and test scenarios
provide, using a comparable algorithm, clear results to which the product’s results could be compared
help in performing extremely rapid evaluation of a test idea
create tables of customer data so that I could perform a test using a variety of personas
accelerate my understanding of the product and the test space
enhance my learning about Boolean algebra and how it could be used in algorithms
record my work and illustrate outcomes for my clients
perform quick calculations when necessary
help me find more actual problems than the other four testers combined

All of this activity happened in a highly exploratory way; each of the activities interacted with the others. I used very rapid cycles of looking at what I needed to learn next about the application, experimenting with and performing tests, programming, asking questions of subject matter experts and programmers and managers, reporting, reading reference documentation, debugging, and learning. Tight loops of activities happening in parallel are what characterize exploratory processes. Yet this was not tool-free work; tools were absolutely central to my exploration of the product, to my learning about it, and to the mission of finding bugs. Indeed, without the tools, I would have had much more limited ideas about what could be tested, and how it could be tested.

The explorers of old used tools: compasses and astrolabes, maps and charts, ropes and pulleys, ships and wagons. These days, software testers explore applications by using mind-mapping software and text editors; spreadsheets and calculators; data generation tools and search engines; scripting tools and automation frameworks. The concept that characterizes exploratory testing is not the input mechanism, which can be fingers on a keyboard, tables of data pumped into the program via API calls, bits delivered through the network, signals from a variable voltage controller. Exploratory testing is about the way you work, and the extent to which test design, test execution, and learning support and reinforce each other. Tools are often a critical part of that process.

What Exploratory Testing Is Not (Part 1): Touring

2011-12-16T19:23:05+00:00

Touring is one way of structuring exploratory testing, but exploratory testing is not necessarily touring, and touring is not necessarily exploratory.

At one extreme, a tourist might parachute into a territory for which there is no detailed knowledge of the landscape, flora and fauna, or human culture, with the goal of identifying what’s there to be learned. Except in such cases, we wouldn’t call her a tourist; we’d call her an anthropologist, or a field botanist, or a field geologist, or an archaeologist. The activity is in this case is interactive with the territory. At the other extreme, a tourist might visit a travel agent, get on a plane to Orlando, meet a chartered bus at the airport, and sit through the rides at Disney World. The activity there is largely passive.

Touring a program can be done in a more scripted or more exploratory way, just as touring a city can be done in a more scripted or more exploratory way. A tourist has many options. Before going on a trip, a tourist might study what is already known about a particular destination. To prepare, she might supply herself with maps and travel guides, and some ideas about destinations of interest. Upon arrival, she might choose a set of walking tours from a guidebook and follow the routes closely, eating only at the restaurants identified in the guidebook, noting buildings and artifacts and other objects of interest by matching them with the descriptions and illustrations. At a given site, she might listen to a prepared audio guide that directs her observations very specifically. She might spend all of her time in the presence of a tour guide who tells her what to observe and how to interpret it. She might choose to accept everything the tour guide told her as the complete story, and refrain from asking questions. Even though the experience would be new to her, and she might learn something from it, she would not likely add much to what is already known. We call that activity touring, but it isn’t very exploratory, and a report on such a tour would largely recapitulate the guidebook. Is your testing like that?

On the other hand, rather than touring like a tourist, she might cover a territory as a historian, or a social scientist, or a travel writer. In that kind of role, she would have a research goal based on the idea of obtaining new knowledge. Learning something new and imparting it to other people requires a more open agenda than sitting on the bus while someone or something else directs your attention. Our researcher might make her way directly to particular destinations or landmarks and begin her research there, or she might amble through neighbourhoods or historical sites to discover new things about them. She could choose to focus on specific aspects of what’s there to observe, or she could choose to let the observations come to her—and, of course, she might do both. She might work with descriptions that she had been given with the intention of adding to them, or she might work from a set of questions that haven’t been asked before. Depending on her mission, she might choose to look for specific patterns or problems, or she might seek deeper understanding that would help her to identify or refine what kind of patterns or problems to look for. Even though the mission to discover new information might come from someone else, she remains in control of the specifics of the itinerary and of each activity from one moment to the next. Is your testing more like that?

One of the hallmarks of exploratory activity is the extent to which it is guided and structured by the person performing that activity. Another hallmark is the extent to which new knowledge feeds into choice of which action to perform next. Touring is not equivalent to exploration; touring can be done is a scripted way or an exploratory way.

A Big List of Interesting Programming Books Released in 2011

2011-12-16T19:16:08+00:00

The following is a list of 48 interesting programming books that were released in 2011. While technically some of them are only programming-related and not about coding per se, each one is sure to be of interest to some programmers.

As some of you know, I run a service called Any New Books?, which emails you a list of new books that are related to the categories of your choice each week. For the most part I pulled this list from the weekly staff picks there throughout this past year, just in time for your Christmas shopping.

The books are ordered by their current sale rank on Amazon (from the most popular to the least popular at the moment, with hardcovers first). I hope this page will help you discover a few titles you may have not have noticed yet.

[Mathematics book list]

The Information: A History, a Theory, a Flood

Stores: USA | Canada | Italy | Kindle | UK Kindle

By James Gleick

ISBN: 0375423729

Publisher: Pantheon

Publication date: March 1, 2011

Binding: Hardcover

Estimated price: $16.82

Share this book on Twitter |Facebook

The Art of Computer Programming, Volumes 1-4A Boxed Set

Stores: USA | Canada

By Donald E. Knuth

ISBN: 0321751043

Publisher: Addison-Wesley Professional

Publication date: March 13, 2011

Binding: Hardcover

Estimated price: $190.47

Share this book on Twitter |Facebook

By Robert Sedgewick, Kevin Wayne

ISBN: 032157351X

Publisher: Addison-Wesley Professional

Publication date: March 19, 2011

Binding: Hardcover

Estimated price: $59.99

Share this book on Twitter |Facebook

The Art of Computer Programming, Volume 4A: Combinatorial Algorithms, Part 1

Stores: USA | UK | Canada | Italy

By Donald E. Knuth

ISBN: 0201038048

Publisher: Addison-Wesley Professional

Publication date: January 22, 2011

Binding: Hardcover

Estimated price: $53.15

Share this book on Twitter |Facebook

By Editors at Morgan Kaufmann

ISBN: 0123849888

Publisher: Morgan Kaufmann

Publication date: February 7, 2011

Binding: Hardcover

Estimated price: $29.06

Share this book on Twitter |Facebook

By Nathan Yau

ISBN: 0470944889

Publisher: Wiley

Publication date: July 20, 2011

Binding: Paperback

Estimated price: $20.48

Share this book on Twitter |Facebook

By Zigurd Mednieks, Laird Dornin, G. Blake Meike, Masumi Nakamura

ISBN: 1449389694

Publisher: O’Reilly Media

Publication date: August 5, 2011

Binding: Paperback

Estimated price: $22.98

Share this book on Twitter |Facebook

By David Flanagan

ISBN: 0596805527

Publisher: O’Reilly Media

Publication date: May 10, 2011

Binding: Paperback

Estimated price: $27.48

Share this book on Twitter |Facebook

By Joe Conway, Aaron Hillegass

ISBN: 0321773772

Publisher: Addison-Wesley Professional

Publication date: July 2, 2011

Binding: Paperback

Estimated price: $26.95

Share this book on Twitter |Facebook

By Stephen G. Kochan

ISBN: 0321711394

Publisher: Addison-Wesley Professional

Publication date: June 20, 2011

Binding: Paperback

Estimated price: $25.99

Share this book on Twitter |Facebook

Cocoa Programming for Mac OS X (4th Edition)

Stores: USA | Canada | Kindle | UK Kindle

By Aaron Hillegass, Adam Preble

ISBN: 0321774086

Publisher: Addison-Wesley Professional

Publication date: November 19, 2011

Binding: Paperback

Estimated price: $27.99

Share this book on Twitter |Facebook

Introducing HTML5 (2nd Edition)

Stores: USA | UK | Canada | Kindle | UK Kindle

By Bruce Lawson, Remy Sharp

ISBN: 0321784421

Publisher: New Riders Press

Publication date: October 28, 2011

Binding: Paperback

Estimated price: $14.49

Share this book on Twitter |Facebook

By Mark Lutz

ISBN: 0596158106

Publisher: O’Reilly Media

Publication date: January 7, 2011

Binding: Paperback

Estimated price: $36.42

Share this book on Twitter |Facebook

Beginning Android Application Development

Stores: USA | UK | Canada | Italy

By Wei-Meng Lee

ISBN: 1118017110

Publisher: Wrox

Publication date: April 19, 2011

Binding: Paperback

Estimated price: $15.99

Share this book on Twitter |Facebook

Agile Web Development with Rails

Stores: USA | UK | Canada | Italy

By Sam Ruby, Dave Thomas, David Heinemeier Hansson

ISBN: 1934356549

Publisher: Pragmatic Bookshelf

Publication date: March 31, 2011

Binding: Paperback

Estimated price: $22.36

Share this book on Twitter |Facebook

By Russ Olsen

ISBN: 0321584104

Publisher: Addison-Wesley Professional

Publication date: February 21, 2011

Binding: Paperback

Estimated price: $24.99

Share this book on Twitter |Facebook

Matlab, Second Edition: A Practical Introduction to Programming and Problem Solving

Stores: USA | UK | Canada | Kindle | UK Kindle

By Stormy Attaway

ISBN: 0123850819

Publisher: Butterworth-Heinemann

Publication date: August 11, 2011

Binding: Paperback

Estimated price: $35.44

Share this book on Twitter |Facebook

By Charlie Hunt, Binu John

ISBN: 0137142528

Publisher: Prentice Hall

Publication date: October 10, 2011

Binding: Paperback

Estimated price: $40.73

Share this book on Twitter |Facebook

By Gabe Zichermann, Christopher Cunningham

ISBN: 1449397670

Publisher: O’Reilly Media

Publication date: August 19, 2011

Binding: Paperback

Estimated price: $14.34

Share this book on Twitter |Facebook

By Miran Lipovaca

ISBN: 1593272839

Publisher: No Starch Press

Publication date: April 18, 2011

Binding: Paperback

Estimated price: $21.55

Share this book on Twitter |Facebook

By Raffaele Cecco

ISBN: 1449393632

Publisher: O’Reilly Media

Publication date: August 2, 2011

Binding: Paperback

Estimated price: $21.31

Share this book on Twitter |Facebook

By John-David Warren, Josh Adams, Harald Molle

ISBN: 1430231831

Publisher: Apress

Publication date: July 14, 2011

Binding: Paperback

Estimated price: $26.38

Share this book on Twitter |Facebook

By Lars George

ISBN: 1449396100

Publisher: O’Reilly Media

Publication date: September 20, 2011

Binding: Paperback

Estimated price: $24.99

Share this book on Twitter |Facebook

The Tangled Web: A Guide to Securing Modern Web Applications

Stores: USA | Canada | Kindle | UK Kindle

By Michal Zalewski

ISBN: 1593273886

Publisher: No Starch Press

Publication date: November 26, 2011

Binding: Paperback

Estimated price: $26.50

Share this book on Twitter |Facebook

MongoDB in Action

Stores: USA

By Kyle Banker

ISBN: 1935182870

Publisher: Manning Publications

Publication date: December 16, 2011

Binding: Paperback

Estimated price: $27.46

Share this book on Twitter |Facebook

By Alan Gates

ISBN: 1449302645

Publisher: O’Reilly Media

Publication date: October 20, 2011

Binding: Paperback

Estimated price: $24.99

Share this book on Twitter |Facebook

By Doug Hellmann

ISBN: 0321767349

Publisher: Addison-Wesley Professional

Publication date: June 11, 2011

Binding: Paperback

Estimated price: $31.49

Share this book on Twitter |Facebook

The Joy of Clojure: Thinking the Clojure Way

Stores: USA | UK | Canada | Italy

By Michael Fogus, Chris Houser

ISBN: 1935182641

Publisher: Manning Publications

Publication date: April 4, 2011

Binding: Paperback

Estimated price: $21.99

Share this book on Twitter |Facebook

By Richard Wentk

ISBN: 111800759X

Publisher: Wiley

Publication date: June 28, 2011

Binding: Paperback

Estimated price: $19.99

Share this book on Twitter |Facebook

By Jurg van Vliet, Flavia Paganelli

ISBN: 1449393683

Publisher: O’Reilly Media

Publication date: March 14, 2011

Binding: Paperback

Estimated price: $24.22

Share this book on Twitter |Facebook

By Mark Dalrymple

ISBN: 0321706250

Publisher: Addison-Wesley Professional

Publication date: August 28, 2011

Binding: Paperback

Estimated price: $32.95

Share this book on Twitter |Facebook

CUDA Application Design and Development

Stores: USA | UK | Canada | Kindle | UK Kindle

By Rob Farber

ISBN: 0123884268

Publisher: Morgan Kaufmann

Publication date: November 14, 2011

Binding: Paperback

Estimated price: $43.98

Share this book on Twitter |Facebook

Sinatra: Up and Running

Stores: USA | Canada | Kindle | UK Kindle

By Alan Harris, Konstantin Haase

ISBN: 1449304230

Publisher: O’Reilly Media

Publication date: December 1, 2011

Binding: Paperback

Estimated price: $15.29

Share this book on Twitter |Facebook

By Jurgen Appelo

ISBN: 0321712471

Publisher: Addison-Wesley Professional

Publication date: January 7, 2011

Binding: Paperback

Estimated price: $29.50

Share this book on Twitter |Facebook

By Shashank Tiwari

ISBN: 047094224X

Publisher: Wrox

Publication date: September 13, 2011

Binding: Paperback

Estimated price: $16.99

Share this book on Twitter |Facebook

By Martin L. Abbott, Michael T. Fisher

ISBN: 0321753887

Publisher: Addison-Wesley Professional

Publication date: May 15, 2011

Binding: Paperback

Estimated price: $16.96

Share this book on Twitter |Facebook

Programming Concurrency on the JVM: Mastering Synchronization, STM, and Actors

Stores: USA | UK | Canada | Italy

By Venkat Subramaniam

ISBN: 193435676X

Publisher: Pragmatic Bookshelf

Publication date: September 2, 2011

Binding: Paperback

Estimated price: $19.29

Share this book on Twitter |Facebook

By James L. Williams

ISBN: 0321767365

Publisher: Addison-Wesley Professional

Publication date: October 5, 2011

Binding: Paperback

Estimated price: $16.54

Share this book on Twitter |Facebook

Semantic Web for the Working Ontologist, Second Edition: Effective Modeling in RDFS and OWL

Stores: USA | UK | Canada | Italy

By Dean Allemang, James Hendler

ISBN: 0123859654

Publisher: Morgan Kaufmann

Publication date: June 3, 2011

Binding: Paperback

Estimated price: $38.43

Share this book on Twitter |Facebook

Brave NUI World: Designing Natural User Interfaces for Touch and Gesture

Stores: USA | UK | Canada | Italy

By Daniel Wigdor, Dennis Wixon

ISBN: 0123822319

Publisher: Morgan Kaufmann

Publication date: April 27, 2011

Binding: Paperback

Estimated price: $18.49

Share this book on Twitter |Facebook

Haskell: The Craft of Functional Programming (3rd Edition)

Stores: USA | UK | Canada | Italy

By Simon Thompson

ISBN: 0201882957

Publisher: Addison-Wesley Professional

Publication date: October 2, 2011

Binding: Paperback

Estimated price: $44.96

Share this book on Twitter |Facebook

By Allen B. Downey

ISBN: 1449307116

Publisher: O’Reilly Media

Publication date: July 22, 2011

Binding: Paperback

Estimated price: $21.98

Share this book on Twitter |Facebook

Rails 3 in Action

Stores: USA | UK | Canada | Italy

By Ryan Bigg, Yehuda Katz

ISBN: 1935182277

Publisher: Manning Publications

Publication date: September 28, 2011

Binding: Paperback

Estimated price: $27.48

Share this book on Twitter |Facebook

Test Driven Development for Embedded C

Stores: USA | UK | Canada | Italy

By James W. Grenning

ISBN: 193435662X

Publisher: Pragmatic Bookshelf

Publication date: May 2, 2011

Binding: Paperback

Estimated price: $19.37

Share this book on Twitter |Facebook

MacRuby: The Definitive Guide: Ruby and Cocoa on OS X

Stores: USA | UK | Canada | Kindle | UK Kindle

By Matt Aimonetti

ISBN: 1449380379

Publisher: O’Reilly Media

Publication date: November 4, 2011

Binding: Paperback

Estimated price: $16.70

Share this book on Twitter |Facebook

Modern Information Retrieval: The Concepts and Technology behind Search (2nd Edition)

Stores: USA | UK | Canada | Italy

By Ricardo Baeza-Yates, Berthier Ribeiro-Neto

ISBN: 0321416910

Publisher: Addison-Wesley Professional

Publication date: February 10, 2011

Binding: Paperback

Estimated price: $56.24

Share this book on Twitter |Facebook

By Greg Nudelman

ISBN: 0470942231

Publisher: Wiley

Publication date: May 24, 2011

Binding: Paperback

Estimated price: $14.99

Share this book on Twitter |Facebook

By Fabio Claudio Ferracchiati, Emanuele Garofalo

ISBN: 1430233710

Publisher: Apress

Publication date: May 26, 2011

Binding: Paperback

Estimated price: $22.99

Share this book on Twitter |Facebook

No related posts.

Crowdsource Testing – Acting vs. Using

2011-12-15T22:36:00+00:00

My tech department had a little internal one day idea conference this week. I wanted to keep it from being a programmer fest so I pitched Crowdsource Testing as a topic. Although I knew nothing about it, I had just seen James Whittaker’s STARwest Keynote, and the notion of having non-testers test my product had been bouncing around my brain.

With a buzzword like “crowdsource” I guess I shouldn’t be surprised they picked it. That and maybe I was one of their only submissions.

Due to several personal conflicts I had little time to research, but I did spend about 12 hours alone with my brain in a long car ride. During that time I came up with a few ideas, or at least built on other people’s.

Would you hang this on your wall?

…only a handful of people raised their hands. But this is a Jackson Pollock! Worth millions! It’s quality art! Maybe software has something in common with art.
Two testing challenges that can probably not be solved by skilled in-house testers are:

Quality is subjective. (I tip my hat to Michael Bolton and Jerry Weinberg)
There are an infinite amount of tests to execute. (I tip my hat to James Bach and Robert Sabourin)

There is a difference between acting like a user and using software. (I tip my hat to James Whittaker). The only way to truly measure software quality is to use the software, as a user. How else can you hit just the right tests?
Walking through the various test levels: Skilled testers act. Team testers act (this is when non-testers on your product team help you test). User acceptance testing is acting (they are still pretending to do some work). Dogfooders…ah yes! Dogfooders are finally not acting. This is the first level to cross the boundary between pretending and acting. Why not stop here? It’s still weak in the infinite tests area.
The next level is crowdsource testing. Or is it crowdsourced testing? I see both used frequently. I can think of three ways to implement crowdsource testing:

Use a crowdsource testing company.
Do a limited beta release.
Do a public beta release.

Is crowdsource testing acting or using? If you outsource your crowdsource testing to a company (e.g., uTest, Mob4hire, TopCoder), now you’re hiring actors again. However, if you do a limited or public beta release, you’re asking people to use your product. See the difference?
Beta…is a magic word. It’s like stamping “draft” on the report you’re about to give your boss. It also says, this product is alive! New! Exciting! Maybe cutting edge. It’s fixable still. We won’t hang you out to dry. It can only get better!
Who should facilitate crowdsource testing? The skilled in-house tester. Maybe crowdsource testing is just another tool in their toolbox. If they spent this much time executing their own tests…

…maybe instead, they could spend this much time (blue) executing their own tests, and this much time (yellow) collecting test information from their crowd…
How do we get feedback from the crowd? (from least cool to coolest) The crowd calls our tech support team, virtual feedback meetings, we go find the feedback on twitter etc., the crowd provides feedback via tightly integrated feedback options on the product’s UI, we programmatically collect it in a stealthy way.
How do we enlist the crowd? Via creative spin and marketing (e.g., follow Hipster’s example).
Which products do we crowdsource test? NOT the mission critical ones. We crowdsource the trivial products like entertainment or social products.
How far can we take this? Far. We can ask the crowd to build the product too.
What can we do now?

BA’s: determine beta candidate features.
Testers: spend less time testing trivial features. Outsource those to the crowd.
Programmers: Decouple trivial features from critical features.
UX Designers: Help users find beta features and provide feedback.
Managers: Encourage more features to get released earlier.
Executives: Hire a crowdsource test czar.

There it is in a nutshell. Of course, the live presentation includes a bunch of fun extra stuff like the 1978 Alpo commercial that may have inspired the term “dogfooding”, as well as other theories and silliness.

Why Scripted Testing is Not for Novices

2011-12-15T21:31:32+00:00

…Unless you want bad testing.

Claire Moss writes:

I am surprised that you say that scripted testing is harder for novice
testers. I would have expected that having so much structure around
the tests would make getting into testing easier for someone with less
experience and that the scripted instructions would make up for a lack
of discipline on the part of the tester.

Structure != “being told what to do”
First, you are misusing the word “structure.” All testing is structured. If what you mean by structure is “externally imposed structure” then say that. But even if you are not aware of a structure in your testing, it is there. When I tell a novice tester to test, and don’t tell him how to test, he will be dominated by certain structures he is largely unaware of– or if aware he cannot verbalize or control them much. For instance: the user interface look and feel is a guiding structure for novice testers. They test what they see.

Cognitive science offer plenty of ideas and insights about the structures that guide our thinking and behavior. See the book Predictably Irrational by Dan Ariely for more on this.

Scripted testing always has at least two distinct parts: test design and test execution. They must be considered independently.

Scripted test execution is quite a bit more difficult than exploratory testing, unless you are assuming that the tester following the script has exactly the same knowledge and skill as the test designer (even then it is a qualitatively different sort of cognitive process than designing). An exploratory tester is following (indeed forming as he goes) his own intentions and ideas. But, a scripted tester, to do well, must apprehend the intent of the one who wrote the script. Moreover, the scripted tester must go beyond the stated intent and honor the tacit intent, as well– otherwise it’s just shallow, bad testing.

Try using a script to guide a 10 year-old to drive a car safely on a busy city street. I don’t believe it can be done. You can’t overcome lack of basic skills with written instructions.

And sure, yeah, there is also the discipline issue, but that’s a minor thing, compared to the other things.

As for scripted test design, that also is a special skill. I can ask my son to put together a computer. He knows how to do that. But if I were to ask him for a comprehensive step-by-step set of instructions to allow me to do it, I doubt the result would help me much. Writing a script requires patience, judgment, and lots of empathy for the person who will execute it. He doesn’t yet have those qualities.

Most people don’t like to write. They aren’t motivated. Now give them a task that requires excellent writing. Bad work generally results.

Both on the design side and the execution side, scripted testing done adequately is harder than exploratory testing done adequately. It’s hard to separate an integrated cognitive activity into two pieces and still make it work.

The reason managers assume it’s simpler and easier is that they have low standards for the quality of testing and yet a strong desire for the appearances of order and productivity.

When I am training a new tester, I begin with highly exploratory testing. Eventually, I will introduce elements of scripting. All skilled testers must feel comfortable with scripted testing, for those rare times when it’s quite important.

Examples

1. Start browser

2. Go to CNN.com

3. Test CNN.com and report any problems you find.

This looks like a script, and it is sort of a script, but the interesting details of the testing are left unspecified. One of the elements of good test scripting is to match the instructions to the level of the tester as well as to the design goal of the test. In this case, no design goal is apparent.

This script does not necessarily represent bad testing– because it doesn’t represent any testing whatsoever.

1. Open Notepad

2. Type “hello”

3. Verify that “hello” appears on the screen.

This script has the opposite problem. It specifies what is completely unnecessary to specify. If the tester follows this script, he is probably dumbing himself down. There may be some real good reason for these steps, but again, the design goal is not apparent. The tester’s mind is therefore not being effectively engaged. Congratulations, designer, you’ve managed to treat a sophisticated miracle of human procreation, gestation, mothering, socializing, educating, etc. as if he were the equivalent of an animated poking stick. That’s like buying an iPad, then using it as a serving tray for a platter of cheese.

Exploratory Testing is not “Experienced-Based” Testing

2011-12-10T00:32:35+00:00

Prabhat Nayak is yet another sapient tester recently hired by the rising Indian testing powerhouse, Moolya. Speaking of the ISTQB syllabus, he writes:

One such disagreement of mine is they have put “Exploratory Testing” on purely experienced based testing. James, correct me if I have got ET wrong (and I am always ready to be corrected if I have misunderstood something), a novice tester who has got great cognizance and sapience and can explore things better, can think of different ways the product may fail to perform per requirement can always do a great job in ET than a 5 years experienced tester who has only learned to execute a set of test cases. That is probably one of the beauties of ET. There is of course, always an advantage of having some experience but that alone doesn’t suffice ET to be put under experienced based testing.

You are quite correct Prabhat. Thank you for pointing this out.

The shadowy cabal known as the ISTQB insulates itself from debate and criticism. They make their decisions in secret (under NDA, if you can believe it!) and they don’t invite my opinion, nor anyone’s opinion who has made a dedicated study of exploratory testing. That alone would be a good reason to dismiss whatever they do or claim.

But this case is an especially sad example of incompetent analysis. Let me break it down:

What does “experience-based” mean?

Usually when people in the technical world speak of something as “x-based” they generally mean that it is “organized according to a model of x” or perhaps “dominated by a consideration of x.” The “x”, whatever it is, plays a special role in the method compared to its role in some other “normal” or “typical” method.

What is a normal or typical method of software testing? I’m not aware the the ISTQB explicitly takes a position on that. But by calling ET an experience-based technique, they imply that no other test technique involves the application of experience to a comparable degree. If they have intended that implication– that would be a claim both remarkable and absurd. Why should any test technique not benefit from experience? Do they think that a novice tester and an experienced tester would choose the exact same tests when practicing other test techniques? Do they think there is no value to experience except when using ET? What research have they done to substantiate this opinion? I bet none.

If they have not intended this implication, then by calling ET experience-based it seems to me they are merely making impressive sounds for the sake of it. They might as well have called ET “breathing-based” on the theory that testers will have to breathe while testing, too.

Ah, but maybe there is another interpretation. They may have called ET “experienced-based” not to imply that ET is any more experience-based than other techniques, but rather as a warning that expresses their belief that the ONLY way ET can be valuable is through the personal heroism and mastery of the individual tester. In other words, what they meant to say was that ET is “personal excellence-based” testing, rather than testing whose value derives from an explicit algorithm that is objective and independent of the tester himself.

I suspect that what’s really going on, here: They think the other techniques are concrete and scientific, whereas ET is somehow mystical and perhaps based on the same sort of dodgy magic that you find in Narnia or MiddleEarth. They say “experience-based” to refer to a dark and scary forest that some enter but none ever return therefrom… They say “experienced-based” because they have no understanding of any other basis that ET can possibly have!

Why would it be difficult for Factory School testing thinkers (of which ISTQB is a product) to understand the basis of ET?

It’s difficult for them because Factory School people, by the force of their creed, seek to minimize the role of humanness in any technical activity. They are radical mechanizers. They are looking for algorithms instead of heuristics. They want to focus on artifacts, not thoughts or feelings or activities. They need to deny the role and value of tacit knowledge and skill. Their theory of learning was state of the art in the 18th century: memorization and mimicry. Then, when they encounter ET, they look for something to memorize or mimic, and find nothing.

Those of us who study ET, when we try to share it, talk a lot about cognitive science, epistemology, and modern learning theory. We talk about the importance of practice. This sounds to the Factory Schoolers like incomprehensible new agey incantations in High Elvish. They suspect we are being deliberately obscure just to keep our clients confused and intimidated.

This is also what makes them want to call ET a technique, rather than an approach. I have, since the late nineties, characterized exploratory testing as an approach that applies to any technique. It is a mindset and set of behaviors that occur, to some degree, in ALL testing. To say “Let’s use ET, now” is technically as incoherent as saying “Let’s use knowledge, now.” You are always using knowledge, to some degree, in any work that you do. “Knowledge” is not a technique that you sometimes deploy. However, knowledge plays more a role in some situations and less a role in others. Knowledge is not always and equally applicable, nor is it uniformly applied even when applicable.

For the Factory Schoolers to admit that ET is endemic to all testing, to some degree, would force them to admit that their ignorance of ET is largely ignorance of testing itself! They cannot allow themselves to do that. They have invested everything in the claim that they understand testing. No, we will have to wait until those very proud and desperately self-inflated personalities retire, dry up, and blow away. The salvation of our craft will come from recruiting smart young testers into a better way of thinking about things like ET. The brain drain will eventually cause the Factory School culture to sink into the sea like a very boring version of Altantis.

Bottom Line: Most testing benefits from experience, but no special experience is necessary to do ET

Exploratory testing is not a technique, so it doesn’t need to be categorized alongside techniques. However, a more appropriate way to characterize ET, if you want to charactize it in some way, is to call it self-managed and self-structured (as opposed to externally managed and externally structured). It is testing wherein the design part of the process and the execution part of the process are parallel and interactive.

You know what else is self-managed and self-structured? Learning how to walk and talk. Does anyone suggest that only “experienced people” should be allowed to do that?

Gradle Tip: Start/Stop Embedded Jetty for System Tests

2011-12-06T23:18:31+00:00

I thought I’d share another feature of gradle that i have found extremely useful, starting and stopping an embedded jetty server when my tests run. This is really useful for projects that host web services as it allows me to hit them and very the correct results plus it verifies the full stack is configured correctly. One could quite possibly also use this setup on web projects and have Geb based tests run against their project.

Given a project setup using the jetty plugin as I described in my previous post, all you need to do is hook jetty into run before and after the test task:

And that’s it. Now whenever you run gradle test the embedded jetty server will run along with your tests.

Shapes of Actions

2011-12-05T20:33:06+00:00

In the spring of 2010, I was privileged to have a conversation with Simon Schaffer, who pointed me to the work of a sociologist and philosopher of science named Harry Collins. This year, I discovered and read Collins’ new book, Tacit and Explicit Knowledge, and a somewhat older book, The Shape of Actions (co-authored with Martin Kusch). My colleague James Bach and I believe that these books have great significance in terms of the way we understand, practice, learn, and teach the craft of testing. Three ideas in particular stand out: a distinction between two kinds of actions; a distinction between three types of tacit knowledge; and the notion of repair, whereby people fix up interactions with each other and with our media—and particularly with our machines. Today, I’ll talk about shapes of actions.

In The Shape of Actions, Collins and Kusch describe key differences between two kinds of intentional human actions that they call mimeomorphic and polimorphic. In both words, “morph” refers to shape, or form. “Mimeo-” refers to copying. (The grey-haired among us may remember that stencil printing machines used to be called mimeographs.) Mimeomorphic actions are actions that we want to do the same way every time, almost as though we were machines. Collins and Kusch use the example of a golf swing, a kind of action in which we want to eliminate variation and emphasize precision, regularity, and smoothness. “Poli-” is a pun, referring to two similar-sounding Greek roots. The Greek word polys refers to many, much, or several. The Greek polis—a different word entirely— literally means the city, so Collins and Kusch use “poli-” to emphasize the collective and diversified nature of human actions. Polimorphic actions are naturally and appropriately variable, and are rooted in social and human interactions and goals. Conversation is a canonical example of polimorphic action.

Most human life and human value is centred around polimorphic actions. Still, in many actions, there’s an interplay between the mimeomorphic and the polimorphic. Shifting gears, when performed by a human driver, is something that we almost always want to do smoothly, regularly, mechanically, and (most of the time) below the level of consciousness. Indeed, the majority of North American drivers delegate the mimeomorphic action of gear shifting to mechanisms in the car itself.

Making shifting into a mimeomophic action provides support for the parts of driving that are decidedly not mimeomorphic: merging into traffic, negotiating a left turn, and knowing when to break the letter of traffic laws while maintaining their spirit. Polimorphic actions are handled differently in different places, based on different social paradigms and performed for different purposes. Collins and Kusch note that in some parts of the world (Britain and North America, for example), responsibility for safety is governed by the idea of people following the rules, “violations from the rules of orderly flow being met with expressions of rage”. In Tacit and Explicit Knowledge, Collins point out that in other parts of the world (China, India), responsibility for safety is rooted in the collective, and is governed by the idea of drivers expecting the unexpected. To drivers from the West, drivers from these parts of the world drive in ways that we would consider suicidal or sociopathic. Equally surprisingly to us, people in China and India deal with this style of driving without getting upset or even remarking on it.

All this reminds me of the passage, written by Cem Kaner, in the preface of Testing Computer Software:

Some books say that if our projects are not “properly” controlled, if our written specifications are not always complete and up to date, if our code is not properly organized according to whatever methodology is fashionable, then, well, they should be. These books talk about testing when everyone else plays “by the rules.”

This book is about doing testing when your coworkers don’t, won’t and don’t have to follow the rules.

Consumer software projects are often characterized by a budget that is too small, a staff that is too small, a deadline that is too soon and which can’t be postponed for as long as it should be, and by a shared vision and a shared commitment among the developers.

The quality of a great product lies in the hands of the individuals designing, programming, testing, and documenting it, each of whom counts. Standards, specifications, committees, and change controls will not assure quality, nor do software houses rely on them to play that role. It is the commitment of the individuals to excellence, their mastery of the tools of their crafts, and their ability to work together that makes the product, not the rules.

That is: software development is a polimorphic activity, and if that’s true, testing needs to respond accordingly.

Software development involves mostly polimorphic actions, but some mimeomorphic actions help it along. Compiling a program is so much of a mimeomorphic action that these days we delegate it entirely to machines. Typing is mimeomorphic; we learn to touch-type mimeomorphically so that we can develop programs without the mechanics of typing getting in the way. Programming coaches and programming groups often mandate programmers to develop a specific style of indentation and punctuation to reduce overhead in reading and parsing code, and they mandate exercises or policies to make the regularity automatic. Even though code is designed to be run mimeomorphically, developing it, maintaining it, and interpreting it when things go wrong are all polimorphic actions.

Mimeomorphic activities tend to be easy to observe, so they tend to be easy to identify and to explicate. As a result, conversation, writing, and training in testing has tended to focus on artifacts, on documents, on procedures, and on things that can be automated—the mimeomorphic actions. Those conversations, writings, and training programs almost entirely ignore aspects of testing that are much less visible yet are far more important. This, I believe, is why so many people in our craft talk about writing test cases that are easily described as mimeomorphic actions. Those same people seem to spend little time in discussing how to test, which is composed mostly of polimorphic actions. The challenges of understanding polimorphic actions—combined with the ease of observing and describing mimeomorphic actions—explains why so many people confuse testing with checking. Those challenges explain why people credit Cucumber and its given/when/then formulas much more quickly than they credit the conversations that surround it. Those challenges explain why lowering cost by outsourcing checking work dominates the idea of increasing value by developing local testing skill. And those challenges explain why automation is often seen as some kind of silver bullet for testing problems.

Polimorphic actions are often based on tacit knowledge, different ways of valuing things, and social contexts. Collins notes that polimorphic actions

“can only be executed successfully by a person who understands the social context. Copying the visible behaviour that is the counterpart of an observed action is unlikely to reproduce the action unless it is a mimeomorphic action, because in the case of polimorphic actions, the right behavioural instantiations will change with context. Here (that is, in the book Tacit and Explicit Knowledge –MB) it will be concluded that, for now and the foreseeable future, polimorphic actions—and only polimorphic actions—remain outside the domain of the explicable, whichever of the four possible ways ‘explicable’ is defined. This has significance for the success of different kinds of machines and for the way we teach.”

Watch for a lot more discussion of polimorphic and mimeomorphic actions in the next few blog posts. Watch also for such discussion to work its way into the ways that James and I teach rapid testing.

From the mailbox: selecting test automation tools

2011-12-01T03:47:45+00:00

A long time ago, all the way back in 1999, I wrote an article on selecting GUI test automation tools. Someone recently found it and wrote me an email to ask about getting help with evaluating tools. I decided my response might be useful for other people trying to choose tools, so I turned it into a blog post.

By the way, so much has changed since my article on GUI testing tools was published back in 1999 that my approach is a little different these days. There are so many options available now that weren’t 12 years ago, and there are new options appearing nearly every day it seems.

Back in 1999 I advocated a heavy-weight evaluation process. I helped companies evaluate commercial tools, and at the time it made sense to spend lots of time and money on the evaluation process. The cost of making a mistake in tool selection was too high.

After all, once we chose a tool we would have to pay for it, and that licensing fee became a sunk cost. Further, the cost of switching between tools was exorbitant. Tests were tool-specific and could not move from one tool to another. Thus we’d have to throw away anything we created in Tool A if we later decided to adopt Tool B. Further, any new tool would cost even more money in licensing fees. So spending a month evaluating tools before making a 6-figure investment made sense.

But now the market has changed. Open source tools are surpassing commercial tools, so the license fee is less of an issue. There are still commercial tools, but I always recommend looking at the open source tools first to see if there’s anything that fits before diving into commercial tool evaluations.

So here’s my quick and dirty guide to test tool selection.

If you want a tool to do functional test automation (as opposed to unit testing), you will probably need both a framework and a driver.

The framework is responsible for defining the format of the tests, making the connection between the tests and test automation code, executing the tests, and reporting results.
The driver is responsible for manipulating the interface.

So, for example, on my side project entaggle.com, I use Cucumber (framework) with Capybara (driver).

To decide what combination of framework(s) and driver(s) are right for your context…

Step 1. Identify possible frameworks…

Consideration #1: Test Format

The first thing to consider is if you need a framework that supports expressing tests in a natural language (e.g. English), or in code.

This is a question for the whole team, not just the testers or programmers. Everyone on the project must be able to at least read the functional tests. Done well, the tests can become executable requirements. So the functional testing framework needs to support test formats that work for collaboration across the whole team.

Instead of assuming what the various stakeholders want to see, ask them.

In particular, if you are contemplating expressing tests in code, make very sure to ask the business stakeholders how they feel about that. And I don’t mean ask them like, “Hey, you don’t mind the occasional semi-colon, right? It’s no big deal, right? I mean, you’re SMART ENOUGH to read CODE, right?” That kind of questioning backs the business stakeholders into a corner. They might say, “OK,” but it’s only because they’ve been bullied.

I mean mock up some samples and ask like this: “Hey, here’s an example of some tests for our system written in a framework we’re considering using. Can you read this? What do you think it’s testing?” If they are comfortable with the tests, the format is probably going to work. If not, consider other frameworks.

Note that the reason that it’s useful to express expectations in English isn’t to dumb down the tests. This isn’t about making it possible for non-technical people to do all the automation.

Even with frameworks that express tests in natural language, There is still programming involved. Test automation is still inherently about programming.

But by separating the essence of the tests from the test support code, we’re able to separate the concerns in a way that makes it easier to collaborate on the tests, and further the tests become more maintainable and reusable.

When I explain all that, people sometimes ask me, “OK, that’s fine, but what’s the EASIEST test automation tool to learn?” Usually they’re thinking that “easy” is synonymous with “record and playback.”

Such kinds of easy paths may look inviting, but it’s a trap leads into a deep dark swamp from which there may be no escape. None of the tools I’ve talked about do record and playback. Yes, there is a Selenium recorder. I do not recommend using it except as a way to learn.

So natural language tests facilitate collaboration. But I’ve seen organizations write acceptance tests in Java with JUnit using Selenium as the driver and still get a high degree of collaboration. The important thing is the collaboration, not the test format.

In fact, there are advantages to expressing tests in code.

Using the same unit testing framework for the functional tests and the code-facing tests removes one layer of abstraction. That can reduce the complexity of the tests and make it easier for the technical folks to create and update the tests.

But the times I have seen this work well for the organization is when the business people were all technology savvy so they were able to read the tests just fine even when expressed in Java rather than English.

Consideration #2: Programming Language

The next consideration is the production code language.

If your production code is written in…	And you want to express expectations in natural language, consider…	Or you want to express expectations in code, consider…
Java	Robot Framework, JBehave, Fitnesse, Concordion	JUnit, TestNG
Ruby	Cucumber	Test::Unit, RSpec
.NET	Specflow	NUnit

By the way, the tools I’ve mentioned so far are not even remotely close to a comprehensive list. There are lots more tools listed on the AA-FTT spreadsheet. (The AA-FTT is the Agile Alliance Functional Testing Tools group. It’s a program of the Agile Alliance. The spreadsheet came out of work that the AA-FTT community did. If you need help interpreting the spreadsheet, you can ask questions about it on the AA-FTT mail list.)

So, why consider the language that the production code is written in? I advocate choosing a tool that will allow you to write the test automation code in the same language (or at least one of the same languages if there are several) as the production code for a number of reasons:

The programmers will already know the language. This is a huge boon for getting the programmers to collaborate on functional test automation.
It’s probably a real programming language with a real IDE that supports automated refactoring and other kinds of good programming groovy-ness. It’s critical to treat test automation code with the same level of care as production code. Test automation code should be well factored to increase maintainability, remove duplication, and exhibit SOLID principles.
It increases the probability that you’ll be able to bypass the GUI for setting up conditions and data. You may even be able to leverage test helper code from the unit tests. For example, on entaggle.com, I have some data generation code that is shared between the unit tests and the acceptance tests. Such reuse drastically cuts down on the cost of creating and maintaining automated tests.

Consideration #3: The Ecosystem

Finally, as you are considering frameworks, consider also the ecosystem in which that framework will live. I personally dismiss any test framework that does not play nicely with both the source control system and the automated build process or continuous integration server. That means at a bare minimum:

All assets must be flat files, no binaries. So no assets stored in databases, and no XLS spreadsheets (though comma separated values or .CSV files can be OK). In short, if you can’t read all the assets in a plain old text editor like Notepad, you’re going to run into problems with versioning.
It can execute from a command line and return an exit code of 0 if everything passes or some other number if there’s a failure. (You may need more than this to kick off the tests from the automated build and report results, but the exit code criteria is absolutely critical.)

Step 2. Choose your driver(s)…

A driver is just a library that knows how to manipulate the interface you’re testing against. You may actually need more than one driver depending on the interfaces in the system you’re testing. You might need one driver to handle web stuff while another driver can manipulate Windows apps.

Note that the awesome thing about the way test tools work these days is that you can use multiple drivers with any given functional testing framework. In fact, you can use multiple drivers all in a single test. Or you can have a test that executes against multiple interfaces. Not a copy of the test, but actually the same test. By separating concerns, separating the framework from the driver, we make it possible for tests to be completely driver agnostic.

Choosing drivers is often a matter of just finding the most popular driver for your particular technical context. It’s hard for me to offer advice on which drivers are good because there are so many more drivers available than I know about. Most of the work I do these days is web-based. So I use Selenium / WebDriver.

To find a specific driver for a specific kind of interface, look at the tools spreadsheet or ask on the AA-FTT mail list.

Step 3. Experiment

Don’t worry about choosing The One Right tool. Choose something that fits your basic criteria and see how it works in practice. These days it’s so much less costly to experiment and see how things go working with the tool on real stuff than to do an extensive tool evaluation.

How can this possibly be? First, lots of organizations are figuring out that the licensing costs are no longer an issue. Open source tools rule. Better yet, if you go with a tool that lets you express tests in natural language it’s really not that hard to convert tests from one framework to another. I converted a small set of Robot Framework tests to Cucumber and it took me almost no time to convert the tests themselves. The formats were remarkably similar. The test automation code took a little longer, but there was less of it.

Given that the cost of making a mistake on tool choice is so low, I recommend experimenting freely. Try a tool for a couple weeks on real tests for your real project. If it works well for the team, awesome. If not, try a different one.

But whatever you do, don’t spend a month (or more) in meetings speculating about what tools will work. Just pick something to start with so you can try and see right away. (As you all know, empirical evidence trumps speculation. )

Eventually, if you are in a larger organization, you might find that a proliferation of testing frameworks becomes a problem. It may be necessary to reduce the number of technologies that have to be supported and make reporting consistent across teams.

But beware premature standardization. Back in 1999, choosing a single tool gave large organizations an economy of scale. They could negotiate better deals on licenses and run everyone through the same training classes. Such economies of scale are evaporating in the open source world where license deals are irrelevant and training is much more likely to be informal and community-based.

So even in a large organization I advocate experimenting extensively before standardizing.

Also, it’s worth noting that while I can see a need to standardize on a testing framework, I see much less need to standardize on drivers. So be careful about what aspects of the test automation ecosystem you standardize on.

Good luck and happy automating…

Disposable Programs

2011-11-30T14:42:13+00:00

In many IT installations today, the number one problem is program maintenance.
Although the total problem is far from simple, there are a number of relatively simple ideas that can be applied immediately to furnish "prompt relief." One such idea is the disposable program.

The idea of disposable programs is not new. Every programmer has written code that was to be used once and then thrown away—codes such as:

1. First-cut subroutines, as for simple, quick formatting of output.

2. One-time reports.

3. Test drivers.

4. Research programs to probe some peculiar feature of the programming language, operating system, database, or other "black box."

5. Engineering assist programs, to help diagnose a particular hardware malfunction.

If you consider these ﬁve examples relative to your own experience, you will notice two categories:

KEPT: ﬁrst-cut routines, and one-time reports, and

DISPOSED: test drivers, research programs, and hardware testers. That is, though all are thought of as single use programs, the KEPT routines tend to be held, somewhere, "just in case." Only the DISPOSED programs are actually discarded, whether they should be or not.

Can you recall an instance when you wished you had actually retained a discarded program?

And can you recall cases of KEPT programs you devoutly wish you had destroyed when you had the chance? These are the programs you see and curse almost every day, as their user phones, pleading for "just one little change."

Perhaps we would immediately begin improving the maintenance situation by applying two simple rules about "one- time" programs:

1. If you are about to throw it away, keep it.

2. If you are about to keep it, throw it away.

Unfortunately, applying these two rules together creates an inﬁnite recursion. All programmers would be instantly paralyzed. There must be a better way. (Or do you believe that instant paralysis of all programmers would be of great benefit to the human species?)

Consider the examples once again; you'll notice that the underlying principle seems to be:

If the programmer is responsible for the decision, the program is discarded; but if the user is responsible the program is kept.

But why not just keep all programs, for all time? There are many reasons why a program brought out of hibernation could incur costs:

1. The hardware environment has changed.

2. The system software environment has changed.

3. The size or format of the data has changed.

4. The human environment has changed.

5. Some part of the program or its supporting material has been lost or damaged.

So it does cost to rerun an "unchanged" program, and the longer the period of hibernation, the greater the cost. But you already knew this—we all know this. Then why, oh why, do we keep tumbling into the same trap? And how do we get out, or stay out, of the trap.

Well, we'll watch for readers' ideas on these questions, and next blog entry, I'll give a few ideas of my own.

STARwest 2011 Lightning Talk Keynotes

2011-11-28T17:28:00+00:00

If there were a testing conference that consisted of only lightning talks, I would be the first to sign up. Maybe I have a short attention span or something. STARwest’s spin on lightning talks is “Lightning Talk Keynotes” in which (I assume) Lee Copeland handpicks lightning talk presenters. He did not disappoint. Here is my summary:

Michael Bolton

Michael started with RST’s formal definition of a Bug: “anything that threatens the value of the product”. Then he shared his definition of an Issue: “anything that threatens the value of the testing” (e.g., a tool I need, a skill I need). Finally, Bolton suggested, maybe issues are more important than bugs, because issues give bugs a place to hide.

Hans Buwalda

His main point was, “It’s the test, stupid”. Hans suggested, when test automation takes place on teams, it’s important to separate the testers from the test automation engineers. Don’t let the engineers dominate the process because no matter how fancy the programming is, what it tests is still more important.

Lee Copeland

Lee asked his wife why she cuts the ends off her roasts, and lays them against the long side of the roast, before cooking them. She wasn’t sure because she learned it from her mother. So they asked her mother why she cuts the ends off her roasts. Her mother had the same answer so they asked her grandmother. Her grandmother said, “Oh, that’s because my oven is too narrow to fit the whole roast in it”.

Lee suggested most processes begin with “if…then” statements (e.g., if the software is difficult to update, then we need specs). But over time, the “if” part fades away. Finally, Lee half-seriously suggested all processes should have a sunset clause.

Dale Emory

If an expert witness makes a single error, out of an otherwise perfect testimony, it raises doubts in the juror's minds. If 1 out of 800 automated tests throws a false positive, people accept that. If it keeps happening, people loose faith in the tests and stop using them. Dales suggests the following prioritized way of preventing the above:

Remove the test.
Try to fix the test.
If you are sure it works properly, add it back to the suite.

In summary, Dale suggests, reliability of the test suite is more important then coverage.

Julie Gardiner

Julie showed a picture of one of those sliding piece puzzles; the kind with one empty slot so adjacent pieces can slide into it. She pointed out that this puzzle could not be solved if it weren’t for the empty slot.

Julie suggested slack is essential for improvement, innovation, and morale and that teams may want to stop striving for 100% efficiency.

Julie calls this “the myth of 100% efficiency”.

Note: as a fun gimmicky add-on, she offered to give said puzzle to anyone that went up to her after her lightning talk to discuss it with her. I got one!

Bob Galen

Sorry, I didn’t take any notes other than “You know you’ve arrived when people are pulling you”. Either it was so compelling I didn’t have time to take notes, or I missed the take-away.

Dorothy Graham

Per Dorothy, coverage is a measure of some aspect of thoroughness. 100% coverage does not mean running 100% of all the tests we’ve thought of. Coverage is not a relationship between the tests and themselves. Instead, it is a relationship between the tests and the product under test.

Dorothy suggests, whenever you hear “coverage”, ask “Of what?”.

Jeff Payne

Jeff began by suggesting, “If you’re a tester and don’t know how to code in 5 years, you’ll be out of a job”. He said 80% of all tester job posts require coding and this is because we need more automated tests.

Martin Pol

My notes are brief here but I believe Martin was suggesting, in the near future, testers will need to focus more on non-functional tests. The example Martin gave is the cloud; if the cloud goes down, your services (dependent on the cloud) will be unavailable. This is an example of an extra dependency that comes with using future technology (i.e., the cloud).

Gradle: Using JNDI with the Jetty Plugin

2011-11-27T17:06:16+00:00

I use gradle a lot at work and I believe one discovery that was a true win was discovering how to fake JNDI when using the jettyRun task for local development. Originally googling and searching the documentation didn’t yield anything so I thought I’d write a quick post detailing how to do it in case you’re like me and googling for the same thing.

First off you need to create a jetty-env configuration file and put it somewhere in your project (I prefer src/test/resources). Here’s a sample of one I use that uses H2 for the dataSource:

This uses an already running H2 instance and runs an init script located in src/test/resources for populating the database with some tables. From our gradle script we need to reference the file from the jettyRun task (I also add it to the jettyRunWar task as well).

Finally, to complete this example, we want H2 running before jetty kicks off. So we add the h2 dependency to our build script and run the main method of org.h2.tools.Server.

I’ve created a sample spring MVC 3 project that makes use of all the above for local development. Just clone it and run gradle jettyRun to see it in action!

Download DB2 Express-C 9.7.5

2011-11-21T20:56:56+00:00

IBM recently released a new edition of DB2 Express-C, the free edition of their powerful commercial database. You can grab DB2 Express-C 9.7.5 for Linux, Unix, and Windows here.

This new version includes support for Oracle compatibility (which was previously only available for commercial editions), and a few other features, including an enhanced db shell known as CLP+.

If you downloaded DB2 in the past, take this new edition for a spin and let us know what you think. If you haven’t, now would be a great time to try out DB2 Express-C in its enhanced incarnation.

It’s entirely free of charge and there are no limits on the amount of data you can store in development, testing, or production.

In a future post I will also explain why, contrary to what some may believe, it’s an excellent choice for startups that are looking for a solution that goes beyond the relational paradigm.

No related posts.

STARwest’s “Test Estimation & the Art of Negotiation”

2011-11-21T18:11:00+00:00

I’m going back through my STARwest notes and I want to blog about a few other sessions I enjoyed.

One of those was Nancy Kelln and Lynn McKee’s, “Test Estimation and the Art of Negotiation”, in which they suggested a new way to answer the popular question…

How long will testing take?

I figured Nancy and Lynn would have some fresh and interesting things to say about test estimation since they hosted the Calgary Perspectives on Software Testing workshop on test estimation this year. I was right.

In this session, they tried to help us get beyond using what one guy in the audience referred to as a SWAG (Silly Wild-Ass Guess).

Nancy and Lynn pointed to the challenge of dependencies. Testers sometimes attempt to deal with dependencies by padding their estimates. This won’t work for Black Swans, which are the unknown unknowns; those events cannot be planned for or referenced.
The second challenge is optimism. We think we can do more than we can. Nancy and Lynn demonstrated an example of the impact of bugs on testing time. As more bugs are discovered, and their fixes need to be verified, more time is taken from new testing, time that is often under estimated.
The third challenge is identifying what testing means to each person. Does it include planning? Reporting? Lynn suggested to try to estimate how much fun one would have at Disneyland. Does the trip start when I leave my house, get to California, enter the park, or get on a ride? When does it end?

Eventually, Lynn and Nancy suggested the best estimate is no estimate at all.

Instead, it is a negotiation with the business (or your manager). When someone asks, “how long will testing take?”, maybe you should explain that testing is not a phase. Testing is exploration, discovery, learning, and reporting. Testing could end when there are no more interesting questions to answer but stopping testing is a business decision.

They further suggested that testers have a responsibility to help the business understand the trade-offs; if quality expectations are high, it may require more testing than if they are lower. Change your test approach to fit the needs. If you only have one day to test, you can still do your best to find the most mission critical information that you can in one day.

Apart from the session content, Lynn and Nancy were awesome presenters. They used volunteers from the audience for role plays, cartoons, brainstorms, and other interactive techniques to keep the session engaging. I heard they proposed a full day session for STPCon Spring. That would be a fun day.

Thanks Nancy and Lynn!

Seven years of blogging in less than 500 words

2011-11-17T14:43:45+00:00

In a couple of days, this blog will be7 years old. Hard to believe so much time has passed since my first Hello World post.

I keep reading that blogging is on thewane, and it's true, mainly because of the popularity of Twitter. ButI strongly believe that blogging is still important, and that morepeople should do it. For me, it's a way to give back to thecommunity. I can't even remember how many times I found solutions tomy technical problems by reading a blog post. I personally try topost something on my blog every single time I solve an issue thatI've struggled with. If you post documentation to a company wiki(assuming it's not confidential), I urge you to try to also blogpublicly about it – think of it as a public documentation that canhelp both you and others.

Blogging is also a great way to furtheryour career. Back in September 2008 I blogged about my experiences with EC2. I had launched an m1.small instance and I had started toplay with it. Lo and behold, I got a job offer based on that post. Iaccepted the offer, and that job allowed me to greatly enhance myexperience with cloud computing. This brings me to a more generalpoint: if you want to work on something you know nothing about, startsmall on your own, persevere, and yes, blog about it! In myexperience, you'll invariably find yourself in a position to be paidto work on it.

It's also pretty interesting for me tolook at my blog posts and to recognize in them the evolution of myown career. I started to blog when I was working as a tester. I waspassionate about automated testing, Python and agile processes and tools. Thisled to digging more into automation tools, then into automated deployments and configuration management in the context of system architecture. This led into cloud computing, and these days this isleading into Big Data analytics. Who knows what's next? Whatever itis, I'm sure it will be exciting, because I make an effort to make itso!

I think it's also important to recognize that learning a tool or a technique is necessary but not sufficient: at the end of the day it's what you do with it that matters. For me the progression has been testing->automation->cloud computing->data analytics->??? (I actually started my career as a programmer, but here I include only the period coinciding with my blogging years.)

This may be self-help-kind-of-corny,but a quote which is attributed (maybe wrongly) to Goethe reallyapplies to everybody:

“Whatever you can do, or dream you can do, begin it. Boldness has genius, power, and magic in it. Begin it now."

RPF: Google's Record Playback Framework

2011-11-17T05:34:16+00:00

By Jason Arbon

At GTAC, folks asked how well the Record/Playback (RPF) works in the Browser Integrated Test Environment (BITE). We were originally skeptical ourselves, but figured somebody should try. Here is some anecdotal data and some background on how we started measuring the quality of RPF.
The idea is to just let users use the application in the browser, record their actions, and save them as a javascript to play back as a regression test or repro later. Like most test tools, especially code generating ones, it works most of the time but its not perfect. Po Hu had an early version working, and decided to test this out on a real world product. Po, the developer of RPF, worked with the chrome web store team to see how an early version would work for them. Why chrome web store? It is a website with lots of data-driven UX, authentication, file upload, and it was changing all the time and breaking existing Selenium scripts: a pretty hard web testing problem, only targeted the chrome browser, and most importantly they were sitting 20 feet from us.

Before sharing with the chrome web store test developer Wensi Liu, we invested a bit of time in doing something we thought was clever: fuzzy matching and inline updating of the test scripts. Selenium rocks, but after an initial regression suite is created, many teams end up spending a lot of time simply maintaining their Selenium tests as the products constantly change. Rather than simply fail like the existing Selenium automation would do when a certain element isn’t found, and require some manual DOM inspection, updating the Java code and re-deploying, re-running, re-reviewing the test code what if the test script just kept running and updates to the code could be as simple as point and click? We would keep track of all the attributes in the element recorded, and when executing we would calculate the percent match between the recorded attributes and values and those found while running. If the match isn’t exact, but within tolerances (say only its parent node or class attribute had changed), we would log a warning and keep executing the test case. If the next test steps appeared to be working as well, the tests would keep executing during test passes only log warnings, or if in debug mode, they would pause and allow for a quick update of the matching rule with point and click via the BITE UI. We figured this might reduce the number of false-positive test failures and make updating them much quicker.

We were wrong, but in a good way!

We talked to the tester after a few days of leaving him alone with RPF. He’d already re-created most of his Selenium suite of tests in RPF, and the tests were already breaking because of product changes (its a tough life for a tester at google to keep up with the developers rate of change). He seemed happy, so we asked him how this new fuzzy matching fanciness was working, or not. Wensi was like “oh yeah, that? Don’t know. Didn’t really use it...”. We started to think how our update UX could have been confusing or not discoverable, or broken. Instead, Wensi said that when a test broke, it was just far easier to re-record the script. He had to re-test the product anyway, so why not turn recording on when he manually verified things were still working, remove the old test and save this newly recorded script for replay later?

During that first week of trying out RPF, Wensi found:

77% of the features in Webstore were testable by RPF
Generating regression test scripts via this early version of RPF was about 8X faster than building them via Selenium/WebDriver
The RPF scripts caught 6 functional regressions and many more intermittent server failures.
Common setup routines like login should be saved as modules for reuse (a crude version of this was working soon after)
RPF worked on Chrome OS, where Selenium by definition could never run as it required client-side binaries. RPF worked because it was a pure cloud solution, running entirely within the browser, communicating with a backend on the web.
Bugs filed via bite, provided a simple link, which would install BITE on the developers machine and re-execute the repros on their side. No need for manually crafted repro steps. This was cool.
Wensi wished RPF was cross browser. It only worked in Chrome, but people did occasionally visit the site with a non-Chrome browser.

So, we knew we were onto something interesting and continued development. In the near term though, chrome web store testing went back to using Selenium because that final 23% of features required some local Java code to handle file upload and secure checkout scenarios. In hindsight, a little testability work on the server could have solved this with some AJAX calls from the client.

We performed a check of how RPF faired on some of the top sites of the web. This is shared on the BITE project wiki. This is now a little bit out of date, with lots more fixes, but it gives you a feel for what doesn’t work. Consider it Alpha quality at this point. It works for most scenarios, but there are still some serious corner cases.

Joe Muharsky drove a lot of the UX (user experience) design for BITE to turn our original and clunky developer and functional-centric UX into something intuitive. Joe’s key focus was to keep the UX out of the way until it is needed, and make things as self-discoverable and findable as possible. We’ve haven't done formal usability studies yet, but have done several experiments with external crowd testers using these tools, with minimal instructions, as well as internal dogfooders filing bugs against Google Maps with little confusion. Some of the fancier parts of RPF have some hidden easter eggs of awkwardness, but the basic record and playback scenarios seem to be obvious to folks.

RPF has graduated from the experimental centralized test team to be a formal part of the Chrome team, and used regularly for regression test passes. The team also has an eye on enabling non-coding crowd sourced testers generate regression scripts via BITE/RPF.

Please join us in maintaining BITE/RPF, and be nice to Po Hu and Joel Hynoski who are driving this work forward within Google.

A Tester’s Commitments

2011-11-17T02:10:17+00:00

This is the latest version of the commitments I make when I work with a programmer.

Dear Programmer,

My job is to help you look good. My job is to support you as you create quality; to ease that burden instead of adding to it. In that spirit, I make the following commitments to you.

Sincerely,

Tester

I provide a service. You are an important client of that service. I am not satisfied unless you are satisfied.
I am not the gatekeeper of quality. I don’t “own” quality. Shipping a good product is a goal shared by all of us.
I will test your code as soon as I can after you deliver it to me. I know that you need my test results quickly (especially for fixes and new features).
I will strive to test in a way that allows you to be fully productive. I will not be a bottleneck.
I’ll make every reasonable effort to test, even if I have only partial information about the product.
I will learn the product quickly, and make use of that knowledge to test more cleverly.
I will test important things first, and try to find important problems. (I will also report things you might consider unimportant, just in case they turn out to be important after all, but I will spend less time on those.)
I will strive to test in the interests of everyone whose opinions matter, including you, so that you can make better decisions about the product.
I will write clear, concise, thoughtful, and respectful problem reports. (I may make suggestions about design, but I will never presume to be the designer.)
I will let you know how I’m testing, and invite your comments. And I will confer with you about little things you can do to make the product much easier to test.
I invite your special requests, such as if you need me to spot check something for you, help you document something, or run a special kind of test.
I will not carelessly waste your time. Or if I do, I will learn from that mistake.

(This is cool! Yong Goo Yeo has created a Prezi of this.)

An Important Comment

This comment came in from Stuart Taylor. I think it’s important enough that I should add this to the post:

Hi James,

I’m not sure if this is meant to be a little tongue in cheek, but that’s how I read it. That said, I fell at point 1. “I provide a service” really?

Yes, really. This is not a joke. And point #1 is the most important point.

This implies there may be alternative service providers, and if my customer relationship management isn’t up to snuff, I could lose this “client”. Already I feel subservient, I’m an option.

Of course there are alternative service providers. (There are alternative programmers as well, but I’m not concerned with that, here. I’m making commitments that have to do with me and my role, here.) And yes, of COURSE you may lose your client. Actually that’s a lot of what “Agile” has done: fired the testers. In their place, we often find quasi-tester/quasi-programmers who are more concerned with fitting in and poking around tools than doing testing.

Testing is not programming. Programmers are not testers. If you mix that, you do a sorry job of both. In other words: I am a programmer and a tester, but not a good programmer at the same time that I’m a good tester.

Please meditate on the difference between service and subservience. I am a servant and I am proud of that. I am support crew. I spent my time as a production programmer and I’m glad I don’t do that any more. I don’t like that sort of pressure. I like to serve people who will take that pressure on my behalf.

This doesn’t make me a doormat. Nobody wipes their feet on me– I clean their feet. There’s a world of difference. Good mothers know this difference better than anyone.

Personally, when I work on a software project, I want to feel like I’m part of the team. I need the developer, and the developer wants me. We work together to bake quality in from the start, and not try and sprinkle it on at the end as part of a “service” that I offer.

What strange ideas you have about service. You think people on the same team don’t serve each other? You think services are something “sprinkled” on the end? Please re-think this.

I want to be on the same team, too. The difference, maybe, is that I require my colleagues to choose me. I don’t rap on the door and say “let me in I belong here! I like to be on the team!” (You can’t just “join” a team. The easier it is to join a team, the less likely that it is actually a team, I think. Excellent teams must gel together over time. I want to speed that process– hence my commitments) Instead, I am invited. I insist that it be an invitation. And how I get invited quickly is by immediately dissolving any power struggle that the programmers may worry about, then earning their respect through the fantastic quality of my work.

You can, of course, demand respect. Then you will fail. Then five years later, you will realize true respect cannot be demanded. (That’s what I did. Ask anyone about what a terror I was when I worked at Apple Computer. If you are less combative or more intelligent than I am, you may make this transition in less time.)

I may have actually agreed with your points before I was exposed to Continuous Integration, because that’s how my relationship used to be; hence me interpreting this as a light hearted piece. However I know that this kind of relationship still exists today (its here at this place). When I began working with continuous integration, the traditional role of the tester, with a discrete testing phase became blurred, as the test effort is pushed back up stream towards the source. If the tester and developer share a conversation about “how can we be sure we build the right thing, and how can we ensure we built it right” before the code gets cut, and the resulting tests from that are used to help write the code (TDD, BDD, SBE), then both of us can have a nice warm fuzzy feeling about the code being of good quality. The automation removes the repetition (and reduces the feedback) to maintain that assertion.

First, your experience has nothing to do with “continuous integration.” Continuous integration is a concept. Your experience is actually related to people. Specific people are collaborating with you in specific satisfying ways, probably because it is their pleasure to do so. I have had wonderful and satisfying professional relationships on a number of projects– mostly at Borland, where we were not doing continuous integration, and where testers and programmers cooperated freely, despite being organized into separate administrative entities.

(Besides, I have to say… Testing cannot be automated. Automation doesn’t remove repetition, it changes it. Continuous integration, as I see it, is another way of forcing your customers do the testing, while placing naive trust in the magic of bewildering machinery. I have yet to hear about continuous integration from anyone who seemed to me to be a student of software testing. Perhaps that will happen someday.)

In any case, nothing I’m saying here prevents or discourages collaboration. What it addresses are some of the unhealthy things that discourage collaboration between a tester and a programmer. I’m not sure if in your role you are doing testing. If you want to be a programmer or a designer or manager or cheerleader or hanger-on, then do that. However, testing is inherently about critical thinking, and therefore it always carries a risk (in the same way that being a dentist does) that we may touch a nerve with our criticism. This is the major risk I’m trying to mitigate.

[Note: Michael Bolton pointed out to me the "warm fuzzy" line. I can't believe I let that go by when I wrote this reply, earlier. Dude! Stuart! If you are in the warm fuzzy business, you are NOT in the testing business. My goal is not anything to do with warm fuzzy thinking. My goal is to patrol out in the cold and shine my bright bright torches into the darkness. On some level, I am never satisfied as a tester. I'm never sure of anything. That's what it MEANS to be a professional tester, rather than an amateur or a marketer. Other people think positively. Other people believe. The warmth and fuzziness of belief is not for us, man. Do not pollute your testing mentality with that.]

My confusion over this being a tongue in cheek post is further compounded by point 2. Too many testers do believe that they own quality. They become the quality gate keeper, and i think they enjoy it. The question “can this go live” is answered by the tester. The tester may be unwilling to relinquish that control, because they perceive that as a loss of power.

Just as excellent testers must understand they provide a service, they must also understand that they are not “quality” people. They simply hold the light so that other people can work. Testers do not create quality. If you create quality, you are not being a tester. You are being an amateur programmer/designer/manager. That’s okay, but my commitment list is meant for people who see themselves as testers.

I guess what i’m fumbling about with here, is that some testers will read this, and sign up to it. Maybe even print it out and put it up on the wall.

I hope they do. This list has served me well.

Either way, you have written another thought provoking piece James, but i cant help wondering about the back story. What prompted you to write it?

Stuart - sat in the coffee shop, but with laptop and wifi, not a notepad

This piece was originally prompted when I took over a testing and build team for SmartPatents, in 1997. I was told that I would decide when the software was good enough. I called a meeting of the programmers and testers and distributed the first version of this document in order to clarify to the programmers that I was not some obstacle or buzzing horsefly, but rather a partner with them in a SERVICE role.

That was also the first time I gave up my office and opted to sit in a low cubicle next to the entrance that no one else wanted– the receptionist’s cubicle. It was perfect for my role. Programmers streamed by all day and couldn’t avoid me.

I think this piece is also a reaction to the Tester’s Bill of Rights (the one by Gilb, not the one by Bernie Berger, although I have my concerns about that one, too), which is one of the most arrogant pieces of crap I’ve ever seen written on behalf of testers. I will not link to it.

And now that I think of it, I wrote this a year after I hired a personal assistant, and two months later had her promoted to be my boss, even though she retained almost the same duties. The experience of working for someone whose actual day-to-day role was to serve me– but who felt powerful and grateful as my manager– helped me understand the nature of service and leadership in a new way.

Finally, a current that runs through my life is my experience as a husband and father. I am very powerful. I can make my family miserable if I so choose. I choose to serve them, instead.

10 steps to a successful testing career

2011-11-15T01:18:48+00:00

This is a presentation that I gave earlier this year to group of new testers, hoping to guide them on how to gave a great testing career. 10 steps for a successful testing careerView more presentations from teknologika.

Journey to 1000 tests a day

2011-11-14T23:36:48+00:00

To help, give something back to the community, I recently gave a presentation at the 2011 ignite software testing conference in Sydney. Here is an updated version of that presentation. Journey to 1000 tests a day View more presentations from teknologika

Smoke Testing vs. Sanity Testing: What You Really Need to Know

2011-11-08T16:31:35+00:00

If you spend any time in forums in which new testers can be found, it won’t be long before someone asks “”What is the difference between smoke testing and sanity testing?”

“What is the difference between smoke testing and sanity testing?” is a unicorn question. That is, it’s a question that shouldn’t be answered except perhaps by questioning the question: Why does it matter to you? Who’s asking you? What would you do if I gave you an answer? Why should you trust my answer, rather than someone else’s? Have you looked it up on Google? What happens if people on Google disagree?

But if you persist and ask me, here’s what I will tell you:

The distinction between the smoke and sanity testing is not generally important. In fact, it’s one of the most trivial aspects of testing that I can think of, offhand. Yet it does point to something that is important.

Both smoke testing and sanity testing refer to a first-pass, shallow form of testing intended to establish whether a product or system can perform the most basic functions. Some people call such testing “smoke testing”; others call it “sanity testing”. “Smoke testing” derives from the hardware world; if you create an electronic circuit, power it up, and smoke comes out somewhere, the smoke test has failed. Sanity testing has no particular derivation that I’m aware of, other than the common dictionary definition of the word. Does the product behave in some crazy fashion? If so, it has failed the sanity test.

Do you see the similarity between these two forms of testing? Can you make a meaningful distinction between them? Maybe someone can. If so, let them make it. If you’re talking to some person, and that person want to make a big deal about the distinction, go with it. Some organizations make a distinction between the smoke and sanity testing; some don’t. If it seems important in your workplace, then ask in your workplace, and adapt your thinking accordingly while you’re there. If it’s important that you provide a “correct” answer on someone’s idiotic certification exam, give them the answer they want according to their “body of knowledge”. Otherwise, it’s not important. Don’t worry about it.

Here’s what is important: wherever you find yourself in your testing career, people will use language that has evolved as part of the culture of that organization. Some consultancies or certification mills or standards bodies claim the goal of providing “a common worldwide standard language for testing”. This is as fruitless and as pointless a goal as a common worldwide standard language for humanity. Throughout all of human history history, people have developed different languages to address things that were important in their cultures and societies and environments. Those languages continue to develop as change happens. This is not a bad thing. This is a good thing.

There is no term in testing of which I am aware whose meaning is universally understood and accepted. There’s nothing either wrong or unusual about that. It’s largely true outside the testing world too. Pick an English word at random, and odds are you’ll find multiple meanings for it. Examples:

Pick (choose, plectrum for a guitar)

English (a language, spin on a billiard ball)

word (a unit of speech, a 32-bit value)

random (without a definite path, of equal probability)

odds (probability, numbers not divisible by two)

multiple (more than one, divisible by)

meaning (interpretation, significance)

Never mind the shades and nuances of interpretation within each meaning of each word! And notice that “never mind”, in this context, is being used ironically. Here, “never mind” doesn’t mean “forget” or “ignore”; here, it really means the opposite: “also pay attention to”!

Not only is there no universally accepted term for anything, there’s no universally accepted authority that could authoritatively declare or enforce a given meaning for all time. (Some might point to law, claiming that there are specific terms which have solid interpretations. If that were true, we wouldn’t need courts or lawyers.)

If you find yourself in conversation (or in an interview) with someone who asks you “Do you do X?”, and you’re not sure what X is by their definition, a smart and pragmatic reply starts with, “I may do X, but not necessarily by that name.” After that,

You can offer to describe your notion of X (if you have one).

You can describe something that you do that could be interpreted as X. That can be risky, so offer this too: “Since I don’t know what you mean by X, here’s something that I do. I think it sounds similar to X, or could be interpreted as X. But I’d like to make sure that we both recognize that we could have different interpretations of what X means.”

You can say, “I’d like to avoid the possibility that we might be talking at cross-purposes. If you can describe what X means to you, I can tell you about my experiences doing similar things, if I’ve done them. What does X mean to you?” Upon hearing their defintion of X, then truthfully describe your experience, or say that you haven’t done it.

If you searched online for an answer to the smoke vs. sanity question, you’d find dozens, hundreds of answers from dozens, hundreds of people. (Ironically, the very post that introduces the notion of the unicorn question includes, in the second-to-last paragraph, a description of a smoke test. Or a sanity test. Whatever.) The people who answer the smoke vs. sanity question don’t agree, and neither do their answers. Yet many, even most, of the people will seem very sure of their own answers. People will have their own firm ideas about how many angels can fit on the head of a pin, too. However, there is no “correct” definition for either term outside of a specific context, since there is no authority that is univerally accepted. If someone claimed to be a universally accepted authority, I’d reject the claim, which would put an instant end to the claim of universal acceptance.

With the possibile exception of the skills of memorization, there is no testing skill involved in memorizing someone’s term for something. Terms and their meanings are slippery, indistinct, controversial, and context-dependent. The real testing skill is in learning to deal with the risk of ambiguity and miscommunication, and the power of expressing ourselves in many ways.

MORE BATS – Or Build Your Own Test Report Heuristic

2011-11-07T18:18:00+00:00

When someone walks up to your desk and asks, “How’s the testing going?”, a good answer depends on remembering to tell that person the right things.

After reading Michael Kelly’s post on coming up with a heuristic (model), I felt inspired to tweak his MCOASTER test reporting heuristic model. I tried using it but it felt awkward. I wanted one that was easier for me to remember, with slightly different trigger words, ordered better for my context. It was fun. Here is what I did:

I listed all the words that described things I may want to cover in a test report. Some words were the same as Michael Kelly’s but some were different (e.g., “bugs”).
Then I took the first letter of each of my words and plugged them into Wordsmith.org’s anagram solver.
Finally, I skimmed through the anagram solver results, until I got to those starting with ‘M’ (I knew I wanted ‘Mission’ to be the first trigger). If my mnemonic didn’t jump out at me, I tweaked the input words slightly using synonyms and repeated step 2.

Two minutes later I settled on MORE BATS. After being an avid caver for some 13 years, it was perfect. When someone asks for a test report, I feel slightly lost at first, until I see MORE BATS. Just like in caving; when deep into the cave I feel lost, but seeing “more bats” is a good sign the entrance is in that general direction, because more bats tend to be near the entrance.

Here’s how it works (as a test report heuristic):

Mission

Obstacles

Risks

Environment

Bugs

Audience

Techniques

Status

“My mission is to test the new calendar control. My only obstacle is finding enough data in the future for some of my scenarios. The risks I’m testing include long range future entries, multiple date ranges, unknown begin dates, and extremely old dates. I’m using the Dev environment because the control is not working in QA. I found two date range bugs. Since this is a programmer (audience) asking me for a test report I will describe my techniques; I’m checking my results in database tableA because the UI is not completed. I think I’ll be done today (status).”

Try to make your own and let me know what you come up with.