Test Automation Planet

This poor girl from Ghana inherited many kilograms of gold and can’t get at it because she can’t pay the back due rent on the gold storage unit. She has reached out to me, a total stranger, for help. I use my testing skills to help her understand that she’s being scammed. But you know, sometimes it’s hard to help people see the truth when fantasy seems less painful…

A stranger approaches on Skype…

[12:29:22 AM] Irene Kusi: Hello

[12:29:33 AM] James Bach: hi

[12:29:44 AM] Irene Kusi: How are you doing ?

[12:29:54 AM] James Bach: Why are you contacting me?

[12:30:10 AM] Irene Kusi: I am very sorry about that and sorry again if i disturb you. I am Irene Kusi from the UK but i am now in Ghana and i would like to know more about you.

[12:30:51 AM] James Bach: Why are you contacting ME, though?

[12:31:39 AM] Irene Kusi: Like i just told you .sorry if i disturb you.

[12:32:06 AM] James Bach: You haven’t told me why you came to me instead of someone else. Is there any reason?

[12:32:10 AM] Irene Kusi: Can we get to know each other personally? What do you mean ?

[12:32:27 AM] James Bach: Look, strangers contact me often for business purposes. That’s normal. I’m trying to figure out if this is for business. Usually people tell me why they are contacting me.

[12:33:08 AM] Irene Kusi: Yes is not strictly business. I just would like to get to know you

[12:33:18 AM] James Bach: So, you don’t know who I am.

[12:33:54 AM] Irene Kusi: Yes i really dont know you but i found your profile on skype and decided to get to know you if i can work with you.

[12:34:23 AM] James Bach: You just chose me at random? Seems like a lot of people from Ghana are doing that.

[12:35:13 AM] Irene Kusi: Why have some people from ghana done something like that to you ?

[12:35:18 AM] James Bach: Yes you are not the first person from Ghana to randomly contact me

[12:35:40 AM] Irene Kusi: And what did they want ??

[12:35:51 AM] James Bach: They wouldn’t tell me… Just like you aren’t telling me.

(She cuts the small-talk and tells me her problem)

[12:36:30 AM] Irene Kusi: My main motive here is to find someone who will help me inherit my property thats why i am on skype .

[12:36:40 AM] James Bach: oh, a 419 thing. I understand

[12:38:27 AM] Irene Kusi: Oh who said is a 419 thing

[12:39:03 AM] James Bach: I did, “oh, a 419 thing” See, that’s where I said it.

[12:39:23 AM] Irene Kusi: what shows is a 419 thing

[12:39:41 AM] James Bach: You said “My main motive here is to find someone who will help me inherit my property thats why i am on skype.” If you haven’t heard of 419, that’s okay.

[12:40:54 AM] Irene Kusi: Ok

[12:41:12 AM] James Bach: Usually it’s Nigerians who need this kind of help. I didn’t know it was happening in Ghana, too.

[12:41:34 AM] Irene Kusi: Ok .so are you going to help me or not ?

[12:41:46 AM] James Bach: What do you need?

[12:42:06 AM] Irene Kusi: I will like you to help me pay with the bills so that i can get the gold

[12:43:07 AM] James Bach: Oh, you want money. Why didn’t you say so at the start?

[12:43:20 AM] Irene Kusi: I dont know why

(With my tester skills, I immediately comprehend what’s going on: she’s too innocent; she sees only the good in people)

[12:44:26 AM] James Bach: Okay, but, why would you trust me? You don’t know me. I might be a bad guy.

[12:45:11 AM] Irene Kusi: Nothing i have just gotten to know that you a very giid guy and i will trsut you .

[12:46:39 AM] James Bach: I think you have to be careful, though. There are many people in the world who may try to take advantage of you. Especially if you have inherited gold. Let me ask you this: If I help you, will you share part of the gold with me?

[12:47:51 AM] Irene Kusi: I will do exactly that not just half but i will let you get the whole gold that is i will bring it to the USA and then we will sell it there and share the money. But before we share the money i will give you 5kilograms of the gold for free just for helping me get it . [Editors note: that's about $300,000]

[12:48:30 AM] James Bach: See? You were just about to trust me and then I tried to take your gold! Irene, you are WAY too trusting. People will want to take from you and not give you anything.

[12:49:14 AM] Irene Kusi: Oh so will you do that to me ?

[12:49:28 AM] James Bach: I just proved that you would LET me if I TRIED to. That was a test. I do testing for a living. And you showed that you are too innocent and good. This is a hard world, full of cheaters.

[12:50:07 AM] Irene Kusi: Oh okay thank you so much for telling me this .

[12:50:13 AM] James Bach: They may try to contact you and fool you into giving up your money. So be more careful.

[12:50:26 AM] Irene Kusi: Thank you

(Now I go into consulting mode… First, learn about the problem. Often the real problem is not the one they come to you with.)

[12:50:31 AM] James Bach: Why don’t you keep your money? You will need it.

[12:51:16 AM] Irene Kusi: Ok so are you going to help me pay for the bills so that i get the gold ??

[12:51:46 AM] James Bach: I thought you said the gold belonged to you.

[12:51:52 AM] Irene Kusi: Yes

[12:52:04 AM] James Bach: Then you have much much more money than I have. You can pay them yourself, can’t you?

[12:52:21 AM] Irene Kusi: Oh no not right now. You know my dad left the Gold as a form of inheritance for me and since he could keep all that in his house he had to give it to his lawyer.

[12:53:14 AM] James Bach: Okay, so the lawyer has it.

[12:53:42 AM] Irene Kusi: Yes and what the lawyer is also saying is that he gave it to a security house here in ghana

[12:53:55 AM] James Bach: That’s good. Then you know where it is. What happened when you went to get it?

[12:54:42 AM] Irene Kusi: They told me that i have to pay for the number of years that it has been there so that they can give it to me . And the lawyer told me that i also need to get a man who will stand by me just to help me

[12:55:42 AM] James Bach: Is this place a bank?

[12:56:03 AM] Irene Kusi: No is not bank but a storage house .

[12:56:19 AM] James Bach: Is it licensed to store gold?

[12:56:29 AM] Irene Kusi: Yes. Is a licensed company. It has the sole mandate to store gold for people

(Use my knowledge of local laws and customs to reveal the true culprit. A consultant should have deep knowledge in useful domains.)

[12:56:57 AM] James Bach: Then they are violating Ghanan law by charging you to release it. Your lawyer should have told you that. Do you trust that lawyer? Do you know him well?

[12:57:27 AM] Irene Kusi: Yes he is the family lawyer .

[12:57:51 AM] James Bach: Then why is he telling you something that is against the laws of your own country? I think you need to talk to a different lawyer.

[12:58:59 AM] Irene Kusi: When we went there they told me that is not like they are charging me for the gold which is mine said by my late dad but for the number of years they have kept the gold for safe keeping thats what they mean. The lawyer has got nothing to do with this at all.

[1:01:34 AM] James Bach: That’s called an “advance fee scam” and they are trying to trick you. You need to go to the police. But I think your lawyer may be in on this. He may even have paid off the police.

(My kind of consulting involves tough, direct speech, on occasion. Sometimes that’s hard for clients to hear.)

[1:02:07 AM] Irene Kusi: You cant talk to my lawyer like that

[1:02:12 AM] James Bach: Your lawyer, if he knows Ghanan law, knows that they aren’t ALLOWED to charge you. They can’t charge you! You are being tricked!

[1:02:26 AM] Irene Kusi: Ok

[1:02:39 AM] James Bach: I told you that you have to be careful. Gold is VERY valuable. People want to steal it.

[1:03:03 AM] Irene Kusi: so now are you going to help me pay the bills or not ?

[1:03:46 AM] James Bach: I’m telling you that you don’t owe any bills. It’s a trap. They want you to pay them and then they won’t give you anything. You need to wake up Irene. You are a crime victim.

[1:04:05 AM] Irene Kusi: Stop what you are saying

[1:04:06 AM] James Bach: That gold is yours. You deserve it.

[1:04:08 AM] Irene Kusi: i hate it.  what do you take me for ? you are not ready to pay fpor the bills thats why you are doing this

[1:04:40 AM] James Bach: I think you put too much belief in people who are now trying to hurt you.  Now you are upset and you want to blame me. But I am just the messenger.

[1:05:08 AM] Irene Kusi: do you think the storage house is an orphanage home where they keep things for free?

[1:05:59 AM] James Bach: Stand up for your rights, Irene. Even in Ghana, there are laws and people have to follow them. They can’t just steal your gold and say you can’t have it because it’s been “sitting there for a long time”

[1:06:24 AM] Irene Kusi: Stop what you are saying

[1:06:26 AM] James Bach: Under GHANA LAW a licensed gold storage house CANNOT charge for storage AFTER the fact. They take payment UP FRONT. Your father already paid. Your lawyer knows that. You have to fight for what is right.

[1:06:49 AM] Irene Kusi: no he didnt pay

[1:07:06 AM] James Bach: THAT’S WHAT YOUR LAWYER TOLD YOU? I can’t believe what a liar this guy is. Do you think you can just bring gold to a storage house and not pay them to store it? He must have paid. So now you are asking strangers to help you pay a bill that YOU DON’T EVEN HAVE.

[1:08:23 AM] Irene Kusi: Bye

[1:08:29 AM] James Bach: I hope you get your gold. Think about what I said.

(If the client is not ready, it doesn’t matter what you tell them: they won’t benefit. I like to think Irene is taking my words to heart. But I have to accept that she probably won’t. Meanwhile I can only hope that her lawyer has invested his ill-gotten gains in an American solar company.)

Announcing the Rapid Testing Intensive seminar.

This is something new. From July 24 to 28, my brother and I are going to run a short, extreme testing project over five days. We have a nice big space, good Internet, and we’ll be cut off from all distractions.

The most interesting element, from my point of view, is that students can attend onsite OR online. Onsite attendance is more trouble, more work, more expense, but also a completely vivid immersive testing experience where you will work shoulder to shoulder with two of the guys who created the Rapid Testing methodology that is now taught all over the world. Online participants will also test and attend optional webinars twice a day during the week. We will have a dedicated coordinator and an online forum so that the experience will be exactly like working as a remote tester on a busy project team.

The outcome of the week will be a test report that will become a part of each participant’s professional portfolio.

I’m so glad Jon is doing this with me. We complement each other well. Jon is a day-to-day test manager who has applied Rapid Testing in his work for nearly all his career since the mid-90’s. I’m a consultant who’s been teaching Rapid Testing since I invented it. Besides, it gives me an excuse to use this graphic…

This may look like George Lucas and Vladimir Lenin, but it’s me and Jon…

I’ve been testing this darn thing all morning and I haven't found a single bug, or even an issue.  My manager probably thinks I’m not testing well enough.  My other tester colleagues keep finding bugs in their projects.  Maybe I’m not a very good tester.  My next scrum report is going to be lame.  This sucks, man.

Wrong!  It probably doesn’t suck.  Not finding bugs may be a good thing.  Your team may be building stuff that works.  And you get to be the lucky dude who delivers the good news. 

If there is lots of stuff that works and no bugs, you have even more to report than testers who keep finding bugs.  Testers who keep finding bugs are probably executing fewer tests than you so they know less about their products than you.  Instead of figuring out what works, they are stuck investigating what doesn’t work.  They’ll still need to figure out what works eventually, it’s just going to take them a while to get there.  And that sucks.

My manager is probably looking at my low bug count metric, thinking I’m not doing anything.  Logging bugs makes me feel like a bad ass.  There must be something I can log…hmmm…I know, I’ll log a bug for this user message; it’s not really worded as well as it could be…it has been like that for the last four years.

No!  No!  No!  That’s gaming the system.  It’s not going to work.  You’re going to get a reputation as a tester who logs trivial bugs.  Your manager is only counting bugs because you’re not giving her anything else.  She just wants to know what you’re doing.  Help your manager.  Show her where to find your test reports, session sheets, or test execution results.  Invite her to your scrum meetings. Tell her how busy you’ve been knocking out tests and how bad ass your entire project team is.

Think about it. 

Reporting what works may be better than reporting trivial bugs.

Software Testing resources that are helpful to development community

[[ This is a content summary only. Visit my website for full links, other content, and more! ]]

No, my muse is not quite as strong as to inspire me to write an ode, but I still want to emphasize a few points about the goodness of running a database on bare metal.

At Evite, we use sharded MySQL for our production database. We designed the current architecture in 2009, when NoSQL was still very much in its infancy, so MySQL seemed a solid choice, a technology that we could at least understand. As I explained elsewhere, we do use MySQL in an almost non-relational way, and we sharded from the get-go, with the idea that it's better to scale horizontally than vertically.

We initially launched with the database hosted at a data center on a few Dell PE2970 servers, each with 16 GB of RAM and 2 quad-core CPUs. Each server was running 2 MySQL instances. We didn't get a chance to dark launch, but the initial load testing we did showed that we should be OK. However, there is nothing like production traffic to really stress test your infrastructure, and we soon realized that we have an insufficient number of servers for the peak traffic we were expecting towards the end of the year.

We decided to scale horizontally in EC2, with one MySQL instance per m1.xlarge EC2 instance. At the time we also engaged Percona and they helped us fine-tune our Percona XtraDB MySQL configuration so we could get the most out of the m1.xlarge horsepower. We managed to scale sufficiently enough for our high season in 2010, although we had plenty of pain points. We chose to use EBS volumes for our database files, because at the time EBS still gave people the illusion of stability and durability. We were very soon confronted with severe performance issues, manifested as very high CPU I/O wait times, which were sometimes so high as to make the instance useless.

I described in a previous post how proficient we became at failing over from a master that went AWOL to a slave. Our issues with EBS volumes were compounded by the fact that our database access pattern is very write-intensive, and a shared medium such as EBS was far from ideal. Our devops team was constantly on the alert, and it seemed like we were always rebuilding instances and recovering from EC2 instance failures, although the end-user experience was not affected.

Long story short, we decided to bring the database back in-house, at the data center, on 'real' bare-metal servers. No virtualization, thanks. The whole process went relatively smoothly. One important point I want to make here is that we already had a year's worth of hard numbers at that point regarding the access patterns to our database, iops/sec, MySQL query types, etc, etc. So it made it easy to do proper capacity planning this time, in the presence of production traffic.

We started by buying 2 Dell C2100 servers, monster machines, with dual Intel Xeon X5650 processors (for a total of 24 cores), 144 GB RAM, and 12 x 1 TB hard disks out of which we prepared a 6 TB RAID 10 volume which we further divided in LVM logical volumes for specific types of MySQL files.

We put 2 MySQL instances on each server, and we engaged Percona again to help us fine-tune the configuration, this time including not only MySQL, but also the hardware and the OS. They were super helpful to us, as usual. Here are only some of the things they recommended, which we implemented:

• set vm.swappiness kernel setting to 0 in /etc/sysctl.conf
• set InnoDB flush method to O_DIRECT because we can rely on the RAID controller to do the caching (we also mounted XFS with the nobarrier option in conjunction with this change)
• disable MySQL query cache, which uses a global mutex that can cause performance issues when used on a multi-core server
• various other optimizations which were dependent on our setup, things like tweaking MySQL configuration options such as key_buffer_size and innodb_io_capacity

"After a few discussions we decided to build on four of our core technologies: MySQL/InnoDB for storage and replication, Multifeed (the technology that powers News Feed) for ranking, Thrift for communications, and memcached for caching. We chose well-understood technologies so we could better predict capacity needs and rely on our existing monitoring and operational tool kits."

A fellow named Oren Reshef writes in response to my post on Worthwhile Documentation.

Let me be the devil’s advocate for a post.

Not having fully detailed test steps may lead to insufficient data in bug reports.

Yup, that could be a risk (although having fully detailed steps in a test script might also lead to insufficient data in bug reports; and insufficient to whom, exactly?).

So what do you do with a problem like that? You manage it. You train the tester, reminding her of the heuristic that each problem report needs a problem description; an example of something that shows the problem; and why she thinks it’s a problem (that is, the oracle; the principle or mechanism by which the tester recognizes the problem). Problem, example, and why; PEW. You praise and reward the tester for producing reports that follow the PEW heuristic; you critique reports that don’t have them. You show the tester lots of examples of bug reports, and ask her to differentiate between the good ones and the bad ones, why each one might be consider good or bad, and in what ways. If the tester isn’t getting it, you have the tester work with and be coached by someone who does get it. The coach talks the tester through the process of identifying a problem, deciding why it’s a problem, and outlining the necessary information. Sometimes it’s steps and specific data; sometimes the steps are obvious and it’s only the data you need to specify; sometimes the problem happens with any old data, and it’s the steps that are important. And sometimes the description of the problem contains enough information that you need supply neither steps nor data. As a tester under time pressure, she needs to develop the skill to do this rapidly and well—or, if nothing works, she might have to find a job for which she is better suited.

You can argue that a good tester should include the needed information and steps in her bug report, but this raise (at least) two problems:

- The same information may be duplicated across many bugs, and even worst it will not be consistent.

As a manager, I can not only argue that a tester should include the needed information; I can require that a tester include the needed information. Come on, Mr. Advocate… this is a problem that a capable tester and a capable test manager (and presumably your client) can solve. If “the same” information is duplicated across many bugs, might that be an interesting factor worth noting? A test result, if you will? Will this actually persist for long without the test manager (or test leads, or the test team) noticing or managing it?

And in any case, would a script solve the problem that you post above? If you can solve that problem in a script, can you solve it in a (set of) bug report(s)?

Writing test steps is not as trivial as it sounds (for example due to cognitive biases, or simply by overlooking steps that seems obvious to you), and to be efficient they also need to be peer reviewed and tested. You don’t want that to happen in a bug report.

“Writing test steps is not as trivial as it sounds.” I know. It’s non-trivial in terms of time, and it’s non-trivial in terms of skill, and it’s non-trivial in terms of cost. That’s why I write about those problems. That’s why James Bach writes about them.

Again: how do you solve problems like testers providing inefficient repro steps? You solve it with training, practice, coaching, review, supervision, observation, interaction… that is, if you don’t like the results you’re getting, you steer the testers in the direction you want them to go, with leadership and management.

The tester may choose the same steps over and over, or steps that are easier for her but does not represent real customers.

Yes, I often hear things like this to justify poor testing. “Real customers” according to whom? It seems as though many organizations have a problem recognizing that hackers are real; that people under pressure are real; that people who make mistakes are real; that people who can become distracted are real. That people who get up and go away from the keyboard, such that a transaction times out are real.

Is it the role of testers to behave always like idealized “real” customers? That’s like saying that it’s the role of airport security to assume that all of the business class customers are “real” business people. I’d argue that it’s nice for testers to be able to act like customers, but it’s far more important for testers to act like testers. It’s the tester’s role to identify important vulnerabilities in the product. Sometimes that involves behaving like a typical customer, and sometimes it involves behaving like an atypical customer, or and sometimes it involves behaving like someone who is not a customer at all. But again, mostly it involves behaving like a tester.

Again you may argue that a good tester should take all that into account, but it’s not that simple to verify it especially for tests involving many short trivial steps.

Maybe it isn’t that simple. If that’s a problem, what about logging? What about screen capture tools? Such tools will track activities far more accurately than a script the tester allegedly followed. After all, a test script is just a rumour of how something should be done, and the claim that the script was followed is also a rumour. What about direct supervision and scrutiny? What about occasional pairing? What about reviewing the testers’ work? What about providing feedback to testers, while affording them both freedom and responsibility?

And would scripts solve that problem when (for example) you’re a recording bug that you’ve just discovered (probably after deviating from a script)? How, exactly? What happens when a problem identified by a script is fixed? Does the value of the script stay constant over time?

Detailed test steps (at least to some extent) might important if your test activity might be transferred to another offshore team someday (happened to me a few weeks ago, I sent them a test document with only high level details and hoped for the best), or your customer requires in-depth understanding of your tests (a multi-billion Canadian telecommunication company insisted on getting those from us during the late 90’s, we chose the least readable TestDirector export format and shipped it to them…).

Ah, yes. “I sent them a test document with only high level details and hoped for the best.” What can I say about “hope” as a management approach? Does a pile of test scripts impart in-depth understanding? Or are they (as I suspect) a way of responding to a question that you didn’t know how to answer, which was in fact a question that the telco didn’t know how to ask?

Going through some set of actions by rote is not a test. A test script is not a test. A test is what you think and what you do. It is a complex, cognitive activity that requires the presence or the development of much tacit knowledge. Raw data or raw instructions at best provide you with a miniscule fraction of what you need to know. If someone wanted in-depth understanding of how a retail store works, would you send them a pile of uncontextualized cash register receipts?

The Devil’s Advocate never seems to have a thoughtful manager for a client. I would suggest that a tester neither hire nor work for the devil.

Thank you for playing the devil’s advocate, Oren.

Dale Emery has taught me an enormous amount about using resistance as a resource.

I’m grateful. I use his ideas every time I set foot in a classroom or start consulting with a new client.

In particular, I channel my inner Dale whenever discussing any of the various controversial things I advocate, such as:

The whole team is responsible for testing and quality, not just QA or the designated testers.

If the regression tests aren’t automated and the team is having a hard time finishing all the testing within a sprint, have programmers help execute them.

Do away with traditional (and often punitive) defect metrics like % missed by phase. Focus instead on metrics related to accomplishments: story points completed, cycle time, and test coverage.

In many organizations these suggestions fly in the face of their accepted “best practices.” Such ideas also tread on political toes. So one response I hear a lot is: “That’s a nice theory, but it won’t work here.”

Before learning techniques for reframing that resistance into a resource, I would end up in a position based argument that amounted to the professional equivalent of “Will too!” “Will not!” “Will too!” “Will not!” Not useful.

Dale’s nicer and wiser than I am, so even when I don’t handle the interaction as well as I would like, by leaning on Dale’s techniques, I certainly handle the conversation better than I otherwise would have. (Although I’m far from perfect at this. Sometimes people succeed in pushing my buttons in such a way that I forget everything I know about how to communicate effectively.)

The first thing I need to know is whether the results of doing whatever it is would be useful. So I ask something like:

Do you think this practice could help improve things?

If I hear “No,” as the reply, we have a fundamental and possibly insurmountable difference in perspective. Nothing I can say will make them try the practice if they do not believe there is any value in it.

I can explore their reasoning. I can say, “That’s interesting. Why not?” But if someone flat out does not believe that a practice I advocate will help, and I disagree even after listening to their reasoning, there is a good chance that I will not be able to help them. Further discussion will cause more harm than good. The best thing for me to do is to stop.

On the other hand if I hear “Yes…but…” then we have a different conversation. First I have to understand what follows the “but…” Often it’s:

It won’t work here.

At this point I am tempted to ask, “Why not?”

But I don’t.

“Why not?” won’t get us anywhere. We’ll end up running down a rathole of excuses starting with “our context is different.” (And of course, they’re right. Their context is different. Every context is different.)

So instead of asking “Why not?” I flip it around. I ask:

What would have to change for this practice to work here?

Now we get a list of objections, but each one is framed as a neat little impediment statement.

It would need to allow for this inherent complexity in our situation.

We’d need to allow time for it.

We’d need executive support.

We’d need money in the training budget.

We’d need to get the programmers to buy in.

We’d need the QA manager to agree.

And we can work on each of those impediments in much the same way, following the trail of reasons why this is a nice theory that can’t possibly work in their real world context all the way down to the bottom.

In what way would it have to accommodate the complexity?

What would have to happen in order to make time for it?

What would have to happen in order to get executive support?

What would have to happen in order to get budget money?

What would have to happen in order for the programmers to buy in?

What would have to happen in order for the QA manager to agree?

The answers usually reveal perfectly practical steps. We can talk to the people in a position of authority or influence who can get us resources, training, budget money. We can try a small pilot. We can experiment with variations.

The simple reframe from “Why not?” to “What would have to change?” opens up possibilities. What could have become an argument becomes instead a brainstorming session. The result is a chain of steps we can take to go from where we are now to where we want to be.

This article will be published in a future addition of the Software Test Professionals Insider – community news.  I didn’t get a chance to write my blog post this week so I thought I would cheat and publish it on my own blog first.

I will also be interviewed about it on Rich Hand’s live Blog Talk Radio Show on Tuesday, January 31st at 1PM eastern time. 

My article is below.  If it makes sense to you or bothers you, make sure you tune in to the radio show to ask questions…and leave a comment here, of course.

 

Don’t Test It

As testers, we ask ourselves lots of questions:

• What is the best test I can execute right now?
• What is my test approach going to be?
• Is that a bug?
• Am I done yet?

But how many of us ask questions like the following?

• Does this Feature need to ever be tested?
• Does it need to be tested by me?
• Who cares if it doesn’t work?

In my opinion, not enough of us ask questions like the three above.  Maybe it’s because we’ve been taught to test everything.  Some of us even have a process that requires every Feature to be stamped “Tested” by someone on the QA team.  We treat testing like a routine factory procedure and sometimes we even take pride in saying...

“I am the tester.  Therefore, everything must be tested...by me...even if a non-tester already tested it...even if I already know it will pass...even if a programmer needs to tell me how to test it...I must test it, no exceptions!”

This type of thinking may be giving testers a bad reputation.  It emphasizes testing importance because of a thoughtless process rather than a service to provide the most valuable information to someone. 

James Bach came up with the following test execution heuristic:

Basic Heuristic:  “If it exists, I want to test it”

I disagree with that heuristic, as it is shown above and often published.  However, I completely agree with the full version James published when he introduced it in his 7/8/2006 blog post:

“If it exists, I want to test it. (The only exception is if I have something more important to do.)”

The second sentence is huge!  Why?  Because often we do have something more important to do, and it’s usually another test!  Unfortunately, importance is not always obvious.  So rather than measuring importance, I like to ask the three questions above and look for things that may not be worth my time to test.  Here are eight examples of what I’m talking about:

1. Features that don’t go to production -  My team has these every iteration.  These are things like enhancements to error logging tables or audit reports to track production activity.  On Agile teams these fall under the umbrella of Developer User Stories.  The bits literally do not go to production and by their nature cannot directly affect users. 
2. Patches for critical production problems that can’t get worse - One afternoon our customers called tech support indicating they were on the verge of missing a critical deadline because our product had a blocking bug.  We had one hour to deliver the fix to production.  The programmer had the fix ready quickly and the risk of further breaking production was insignificant because production was currently useless.  Want to be a hero?  Don’t slow things down.  Pass it through to production.  Test it later if you need to.
3. Cosmetic bug fixes with timely test setup - We fixed a spelling mistake that had shown up on a screen shot of a user error message.  The user was unaware of the spelling mistake but we fixed it anyway; quick and easy.  Triggering said error message required about 30 minutes of setup.  Is it worth it?
4. Straight forward configuration changes - Last year our product began encountering abnormally large production jobs it could not process.  A programmer attempted to fix the problem with an obvious configuration change.  There was no easy way to create a job large enough to cross the threshold in the QA environment.  We made the configuration change in production and the users happily did the testing for us.
5. Too technical for a non-programmer to test - Testing some functionality requires performing actions while using breakpoints in the code to reproduce race conditions.  Sometimes a tester is no match for the tools and skills of a programmer with intimate knowledge of the product code.  Discuss the tests but step aside.
6. Non-tester on loan - If a non-tester on the team is willing to help test, or better yet, wants to help test a certain Feature, take advantage of it.  Share test ideas and ask for test reports.  If you’re satisfied, don’t test it.
7. No repro steps - Occasionally a programmer will take a stab at something.  There are often errors reported for which nobody can determine the reproduction steps.  We may want to regression test the updated area, but we won’t prevent the apparent fix from deploying just because we don’t know if it works or not.
8. Inadequate test data or hardware - Let’s face it.  Most of us don’t have as many load balanced servers in our QA environment as we do in production.  When a valid test requires production resources not available outside of production, we may not be able to test it.

Many of you are probably trying to imagine cases where the items above could result in problems if untested.  I can do that too.   Remember, these are items that may not be worth our time to test.  Weigh them against what else you can do and ask your stakeholders when it’s not obvious.

If you do choose not to test something, it’s important not to mislead.  Here is the approach we use on my team. During our Feature Reviews, we (testers) say, “we are not going to test this”.  If someone disagrees, we change our mind and test it.  If no one disagrees, we “rubber stamp” it. Which means we indicate nothing was tested (on the work item or story) and pass it through so it can proceed to production.  The expression “rubber stamping” came from the familiar image of an administrative worker rubber stamping stacks of papers without really spending any time on each.  The rubber stamp is valuable, however.  It tells us something did not slip through the cracks.  Instead, we used our brains and determined our energy was best used elsewhere.

So the next time you find yourself embarking on testing that feels much less important than other testing you could be doing, you may want to consider...not testing it.  In time, your team will grow to respect your decision and benefit from fewer bottlenecks and increased test coverage where you can actually add value.

This morning I woke up with a lingering thought on my mind that was left over from recent conversations. In the technical community we often get so invested in our work that rather than talk about the simple building blocks that build our success we talk about the huge breakthroughs we make. The problem however is that our breakthroughs most often aren’t accessible to someone who wants to just get started. So today I will give an intro tutorial to using node.js, coffeescript and mongodb to build a simple blog. It builds off the concept in a tutorial I first used to learn node.js more than a year ago, but with a completely from scratch approach. In this tutorial I will also cover practicing Behavior Driven Development using Mocha.

Getting Started

Quite obviously, you’re going to need node.js and mongodb installed. I recommend downloading and installing from the node.js website and following the instructions. I will note that this tutorial covers 0.6.7 so if you come across this post a year from now the API might have changed significantly since then. You will also need mongodb installed, you can download it here. If you use Ubuntu (or some other Debian derivative) you should consider installing from their apt repository. OSX? No problem, you can also install it via homebrew or macports.

Finally, since we’ll be using coffeescript for this tutorial, run npm -g install coffee-script (you might need to sudo) to install coffeescript. Run coffee from the commandline to access the coffeescript REPL. If all works well, install these additional packages listed below via npm that we’ll be using throughout the tutorial.

• express
• mocha

Now let’s bootstrap our project structure. Type express coffeepress to generate a skeleton express project structure. You should see output similar to the following:

Notice how at the end it says to cd to the directory and type npm install? Let’s follow those instructions. Let’s run what we have so far by typing node app.js and navigating to http://localhost:3000. This is the default structure that gives a good starting point. Feel free to investigate the files under the directory before moving on. I even suggest poking around by changing the view a bit and changing the title from express to “My Coffeepress Blog”.

Porting to Coffeescript

At this point, let’s port our backend to coffeescript. I used to copy and paste files into the js2coffee website but you can also install js2coffee via npm. So run the following:

Now you can run coffee app.coffee to run the same app, but now in coffeescript. Take a look at the resulting files to get a feel for what has changed. New to coffeescript? Then I recommend taking a gander at coffeescript.org before moving on. Here is the project structure so far.

Basic Navigation

I like to try and work my way from the outside in while developing a site or feature, materializing components into existence as I need them. So let’s start by working on the initial navigation of the site with some simple in-memory storage of blog posts. This is a good time to get our test framework setup and write a few simple tests against our routes. Normally I prefer to not write tests against my routes, shoving logic into heavy models or services. However I have come to learn that untested components of a system serve as a gravity well for untested code that eventually leads to clients calling you about broken applications. What follows serves as both an introduction to Mocha as well as express’ routing mechanism.

Let’s edit our package.json to include our test framework dependencies.

We include should so that we can use BDD style assertions (more on this in a bit). Write a simple test case located at test/routes-test.coffee with the following code to get us started with mocha

Now run this by typing mocha from the root of the project directory. It should pass. Let’s go ahead and make it fail by changing 4 to 5 and rerunning it. Hopefully this gives you a good feel for our test framework before we move on and change this test to reflect our existing index route. Swap the code in this test out with the following.

Here we fake our requests and response in order to capture what is passed into the response. We fake the render method and verify that our rendered view is “index” and that the variable title is equal to what we expect to be passed in. Run the tests and make changes to your route to make it pass.

Now let’s add a post variable that will be an array of posts we’ll display on the front page. Add the following assertion right after the title assertion:

Run the tests to see it fail and change the route to have a posts array variable available in the template.

Unfortunately you’ll notice that the test fails. This is due to a subtle difference between equal and eql. The former enforces strict equality while the latter is a bit looser, so we change our assertion to use eql. Take a look at the should documentation for more information.

Next let’s write tests for the “new post” route.

Run it, see the failure, and rework our routes.coffee file to include the route (with no implementation yet)

You’ll notice our test passes. That’s not good. Why? Because we put our assertion in our req.render callback, which never gets executed. Doh! How can we make absolutely sure it gets called during our test run? Old school thinking would have you assign a local variable outside the scope of the callback that gets assigned during execution and then can be verified against later on. However we have no guarantee that the routing logic will be synchronous!

Thankfully mocha has a feature that allows for easy testing in these situations. In the method declaration of our test specify a parameter named done. This is a callback that we can call anywhere to indicate the test is done. Basically the test will wait up to a default of 2000ms for it to be called. With this in mind, let’s modify our tests with the following:

If we run this via mocha now we’ll notice that we have one failure. Let’s go ahead and implement the route and connect it into our router.

And connecting it up in the app.coffee

Modifying the Views

This code is useless without views, so let’s modify our views a bit. Let’s modify our layout.jade to link to the new posts page. This layout also makes use of twitter bootstrap because I’m too lazy to design something for this tutorial.

And create our add_post view at views/add_post.jade. An interesting thing to note here that I’ll touch on in a bit is that I prefix the input names with post.

Now let’s add another route to handle the post. This time I’m going to kind of skip delving into the details of writing the test, but you can look at what I have so far here if you’d like to see it.

For now, we’re just going to store each post in an array. Nothing fancy yet. We also add a new route to app.coffee. We could refactor or use some express-mvc plugin to reduce adding each route by hand, but I think it’s good to do it like this to get a feel for express’ low level routing mechanisms.

Finally, we’ll add one last view for viewing a single post:

Go ahead and start the application up and navigate to http://localhost:3000. Post a few posts and play around a bit. You can see the finished application we have so far here.

Mongoose

Whew. I hope I haven’t lost you yet. Especially with the tests against the routes… I know those are always a bit painful! Now that we have functional blog let’s make it work by storing posts in mongodb using Mongoose.

Let’s add a dependency on mongoose to our project and freeze it at version 2.4.10. As always, run npm install to bring it in. Now we’ll create an initial test to just test mongoose out.

Here we import both mongoose and the model object that we’re going to create. Since we want our test to start with a clean slate, we use the before hook (which runs once before anything else in the test runs) to both connect to the database and then remove all of the Post objects from mongodb. We pass the done callback to the remove call so that tests don’t run until all Posts have been removed.

Now we create a new Post instance. You can pass an object literal in to set properties on the model, so we do that here. Finally, in our post.save callback we look the post back up and verify certain attributes have been set. It’s a dumb test (and in fact I rarely test mongoose’s behavior like this), but it does verify that we’ve configured our model correctly.

Now let’s implement our model to make the test pass.

Pretty simple. Now let’s refit our routes to use the Post model instead of an in memory array.

That’s all good and dandy, but one last hiccup is our tests for our routes now fail. Chalk this one up to not having any abstraction or dependency injection in place, but that is fine for now, we’ll live with it and change the tests.

Finally we need our app to actually connect to mongoose when we run it. I like to do this based on the express configuration. This is immensely important if you have mongodb running on servers separated from your application. For this example we’ll just use the databases coffeepress-dev and coffeepress-prod.

Run it and write a few posts. Restart the app and you’ll see the posts still there. Woot!

Conclusion

Well, that about wraps it up… you can see this tutorial in it’s finished glory on the finished branch of the repository. There’s a bit missing out here that we’d implement in the real world. Obviously some kind of authentication would be in order if we took this further, possibly using mongoose-auth. We’d also want to add some validation when posting. These are all excellent topics for future posts but for now I hope this was enough to help you get going!

I asked a friend of mine (who isn’t a coder) if she could try out the amazing Codecademy for me.

She is a smart person, but again, she isn’t a programmer. I thought Codecademy would be a great way to introduce her to programming for the web. As a programmer, I thought it was extremely well made and assumed it would work for anyone.

She landed on the homepage and read the instructions.

She wrote her name (“Elizabeth”), then decided however to change it to the shortened form of her name, “Liz”. To do so she highlighted the whole string she had just typed and tried to press the backspace key to delete it. The whole page disappeared and the browser redirected her to the new tab.

Liz: “Why did it do that?”

Me: “You can’t highlight a word like you’d do in Microsoft Word. You’re in a browser, so the backspace button tells the browser to go back to the previous page you visited in that particular tab. You need to delete the string one letter at the time.”

She reopened the page and tried again, this time things were successful.

Next, she had to measure the length of her name. But she forgot the quotes, so she got this:

Liz: “What happened?”

Me: “What do you think happened?”

Liz: “Some kind of error, but what does ‘Liz is not defined’ mean?”

Me: “You accidentally forgot the quotes.”

Liz: “Oh, OK, let me try again.”

She wrote the first quote but pressed enter before immediately realizing she had forgotten the closing one. As such she encountered the following scenario:

Liz: “I messed it, let me try again.”

Liz: “I typed it right this time, why didn’t it work?”

Me: “Since you didn’t close the first string, it thinks that the second line is still part of the first command. Do you know how to get out of this?”

Liz: “No, but let me try.”

No, she didn’t close the string and pressed enter twice or deleted a bunch of characters (this web REPL allows it, kudos to Codecademy). She simply reloaded the page. Thankfully the same exercise (#1) was reloaded. After completing it successfully though, instead of exercise #2, exercise #0 reappeared, asking her to enter her first name again.

After raising an eyebrow at that point, she went along with it. Next she was asked to try some math.

She can’t be faulted for thinking that “for instance” meant she could try something else. Yet, she got an “Oops, try again.” message.

At that point I suspended this tiny hallway test and I asked her what she thought of this system. She found the site to be a bit unintuitive, but above all felt like she didn’t know how to get unstuck when she made an error.

Now, before you all jump in and tell me that my friend is dumb, that she’ll never be a programmer, or that you can’t idiot-proof a design, keep in mind the very point of this post.

Usability is hard because real users use our UIs and software in different and unexpected ways. Codecamedy did a very good job, but even they left plenty of room for improvement. As web developers it’s our duty to give usability and user experience a significant amount of attention. The UI is the software to the end user.

No related posts.