Test Automation Planet

Well…yes.  I would.

The most prolific bug finder on my team is struggling with this question.  The less the team decides to fix her bugs, the less interested she grows in reporting them.  Can you relate?

There is little satisfaction reporting bugs that nobody wants to hear about or fix.  In fact, it can be quite frustrating.  Nevertheless, when our stakeholders choose not to fix certain classes of bugs, they are sending us a message about what is important to them right now.  And as my friend and mentor, Michael Bolton likes to say:

If they decide not to fix my bug, it means one of two things:

• Either I’m not explaining the bug well enough for them to understand its impact,
• or it’s not important enough for them to fix.

So as long as you’re practicing good bug advocacy, it must be the second bullet above.  And IMO, the customer is always right.

Nevertheless, we are testers.  It is our job to report bugs despite adversity.  If we report 10 for every 1 that gets fixed, so be it.  We should not take this personally.  However, we may want to:

• Adjust our testing as we learn more about what our stakeholders really care about.
• Determine a non-traditional method of informing our team/stakeholders our bugs.
• Individual bug reports are expense because they slowly suck everyone’s time as they flow through or sit in the bug repository.  We wouldn’t want to knowingly start filling our bug report repository with bugs that won’t be fixed.
• One approach would be a verbal debrief with the team/stakeholders after testing sessions.  Your testing notes should have enough information to explain the bugs.
• Another approach could be a “supper bug report”; one bug report that lists several bugs.  Any deemed important can get fixed or spun off into separate bug reports if you like.

It’s a cliché, I know.  But it really gave me pause when I heard Jeff “Cheezy” Morgan say it during his excellent STAReast track session, “Android Mobile Testing: Right Before Your Eyes”.  He said something like, “instead of looking for bugs, why not focus on preventing them?”.                 

Cheezy demonstrated Acceptance Test Driven Development (ATDD) by giving a live demo, writing Ruby tests via Cucumber, for product code that didn’t exist.  The tests failed until David Shah, Cheezy’s programmer, wrote the product code to make them pass. 

(Actually, the tests never passed, which they later blamed on incompatible Ruby versions…ouch.  But I’ll give these two guys the benefit of the doubt. )

Now back to my blog post title.  I find this mindshift appealing for several reasons, some of which Cheezy pointed out and some of which he did not:

• Per Cheezy’s rough estimate 8/10 bugs involve the UI.  There is tremendous benefit to the programmer knowing about these UI bugs while the programmer is writing the UI initially.  Thus, why not have our testers begin performing exploratory testing before the Story is code complete?
• Programmers are often incentivized to get something code completed so the testers can have it (and so the programmers can work on the next thing).  What if we could convince programmers it’s not code complete until it’s tested?
• Maybe the best time to review a Story is when the team is actually about to start working on it; not at the beginning of a Sprint.  And what do we mean when we say the team is actually about to start working on it?
• First we (Tester, Programmer, Business Analyst) write a bunch of acceptance tests.
• Then, we start writing code as we start executing those tests.
• Yes, this is ATDD, but I don’t think automation is as important as the consultants say.  More on that in a future post.
• Logging bugs is soooooo time consuming and can lead to dysfunction.  The bug reports have to be managed and routed appropriately.  People can’t help but count them and use them as measurements for something…success or failure.  If we are doing bug prevention, we never need to create bug reports.

Okay, I’m starting to bore myself, so I’ll stop.  Next time I want to explore Manual ATDD.

Recently, I wrote about Using Storytelling Games in Software Testing, and pointed you to a paper by Martin Jansson and Greger Nolmark. Now I want to give you some tips on creating great storytelling for your testing projects.

First of all, check out Cem Kaner’s work on Scenario Testing: An Introduction to Scenario Testing. I want you to pay special attention to the CHAT (cultural, historical activity theory) model that he talks about. For more on CHAT and testing, read this paper: Putting the Context in Context-Driven Testing (an
Application of Cultural Historical Activity Theory).Pay special attention to the descriptions of networks of activity, and tensions. These are vital to help construct variations and different forces within our storytelling. Both of these pieces are foundational and worth the effort to dig into.

Now, I want you to read Hans Buwalda’s article on Soap Opera Testing. This is a nice variation on scenario testing. Buwalda uses television soap operas as inspiration for a story arcs, for structure, and for variation. Remember, there are lots of variations on a theme in testing, as well as real life! Further to that, look into testing tours. Cem Kaner has a blog post with a link or two to help get some background info: Testing tours: Research for Best Practices?.

Soap Opera tests, Testing Tours and Test Scenarios are a great place to start creating good testing stories.

Next, read up on personas in user experience work. Jenny Cham has a really nice description, with lots of helpful links on creating personas here: Creating design personas. Remember to explore her links in this blog, she has great advice here. I wrote a position paper about using UX personas in testing years ago (I will have to dig it up, there’s a dead link) in this blog post. Elisabeth Hendrickson introduced me to this idea, but she recommended using extreme personas such as cartoon characters. I prefer the standard UX methods pioneered by people like Alan Cooper, but the cartoon or other characters are a great place to start, especially if you feel stuck. Personas are a great way to start developing characters for your story that are relevant. What are their motivations when they use our software? What are their fears? What are their cares and worries and distractions?

Next, I want you to read this piece on telling a great story by a famous author: Kurt Vonnegut at the Blackboard. (I am getting to the gamification side of this project, and I asked Andrzej Marczewski for good references on storytelling in games, and this was the first link he sent me. Thanks Andrzej!) Notice the different options for structuring a good story. In testing, we can use different ones for the same scenario, if we think about activity patterns, tensions, characters, and variations during real life product use. Several versions of one story will yield different kinds of important information and observations. Vonnegut provides a simple framework for story creation that we can easily adapt and apply.

Finally, I want you to look at story telling in games. Andrzej talks about it here: I want to experience games not just play them. Notice that within a game context, of a well designed game, he has a sense of cause and effect: decisions made here can impact things in other areas of the game. That’s just like real life, and it is important to add dimensions to storytelling in games for testing. Variation and dimensions have different effects in a system, and they are rewarding to exercise. Now read this piece on Gamasutra The Designer’s Notebook: Three Problems for Interactive Storytellers, Resolved by Ernest Adams. The points about character amnesia, internal consistency and narrative flow are pure gold for testers. We often arrive into a system without really knowing what is going on, especially at first. However, our customers are also starting from scratch when they use our app for the first time. These problems are areas we should also address when creating stories to test around.

There is also a lot of really useful information here: Environmental Storytelling: Creating Immersive 3D Worlds Using Lessons Learned from the Theme Park Industry by Don Carson, particularly with regards to environmental conditions being so important to incorporate (particularly for you mobile testers!) and the idea of an all-encompassing world, rather than one, linear story.

Andrzej also recommends reading Uncle Computer, Tell Me A Story, and Story Structure 104: The Juicy Details.

As testers, we can incorporate more than a linear scenario into our work. We can add so much more depth to our test approach using stories and worlds. Story development in games is incredibly similar to the story telling we need to do in testing. There is a lot to be learned about creating virtual worlds and stories within them to help change our perspective, explore variations and make important discoveries about the software and systems we test. We can leverage these various works that have been provided with us to create something new and powerful.

Some final points to put this all together:

• Combine the elements from each of the areas I asked you to study above to create a great story, or even better, sets of stories
• Use structure to create real life conditions: different people, motivations, different environmental conditions, and change.
• Add plot twists, surprises and ulterior motives, and look for unintended consequences in systems and people
• Don’t stop at one scenario – create variations on a theme, and change the setting, or the entire world you have created to help change your perspective
• Introduce different characters – are they interrupting? Helping?
• Create a beginning, middle and an end
• Move beyond all happy endings – also try to leave things unresolved, or end on a bad note

I have compiled several foundational concepts to help influence your storytelling, so now the rest is up to you. How you combine them to create something useful is up to you and your team. You have an opportunity to create rich perspectives to kickstart your testing efforts.

Happy storytelling!

public void testSortNumbers() {
  NumberSorter numberSorter = new NumberSorter(quicksort, bubbleSort);
  // Verify that the returned list is sorted. It doesn't matter which sorting
  // algorithm is used, as long as the right result is returned.
  assertEquals(
      new ArrayList(1, 2, 3),
      numberSorter.sortNumbers(new ArrayList(3, 1, 2)));
}

public void testSortNumbers_quicksortIsUsed() {
  // Pass in mocks to the class and call the method under test.
  NumberSorter numberSorter = new NumberSorter(mockQuicksort, mockBubbleSort);
  numberSorter.sortNumbers(new ArrayList(3, 1, 2));
  // Verify that numberSorter.sortNumbers() used quicksort. The test should
  // fail if mockQuicksort.sort() is never called or if it's called with the
  // wrong arguments (e.g. if mockBubbleSort is used to sort the numbers).
  verify(mockQuicksort).sort(new ArrayList(3, 1, 2));
}