Put the “User” Back in User Acceptance Testing

I think it's about time we really focus on making the lives of the users better in our User Acceptance Testing (UAT).

Here's how I propose to do that, and gain the advantages within an iterative Agile framework.

First off, let's settle on definitions. How exactly is UAT defined, and what is its purpose? Most definitions of UAT on the web offer a variant of "having actual users test the application just prior to delivery, to ensure it meets requirements." Some organizations hand users the detailed requirements to test against. Some provide the users with actual test cases or scripts. To these definitions and methods, I would say, respectfully…hogwash.

In UAT, I am interested in whether or not the product makes the users' lives better. Does it make their jobs easier? Does it accurately reflect and support the way they actually work, and not whether what we captured in requirements reflects the way they work? Testing the product against requirements, or test cases, is the job of the team…not the users. What if the requirements are wrong? Maybe we missed some need, or some use cases. Or what if, even though the requirements reflect what the users asked for, once they get their hands on the product, they realize it's not what they need?

Here is my proposed solution to the issue.

First, my Utopian scenario. In an Agile framework, my ideal scenario for UAT is to have direct access to users during the iteration, to gather their feedback as the feature is actually being developed. Show them a screen; let them spend some time with an isolated module. If we have that luxury, the cost of making changes based on their feedback is far less.

Absent that, here's my process framework for incorporating UAT into an iterative development process. NOTE: this process is tailored for a multi-iteration release cycle:

1. Iteration review/demo: We have finished iteration 1. We have the Product Owner identify and invite interested users (our UAT group) to the demo at the end of the iteration. Let them meet the team, and use the demo of the features developed in that iteration as an introduction to the UAT group of what they will be testing. Gather their initial feedback from the demo and review it with the Product Owner to see if the P.O. wants it in the Product Backlog.
   
2. UAT Cycle Start: Use the product demo as a launching pad for a UAT cycle. The UAT group has seen the features in action, they've been able to ask questions and get an initial impression. Now, during iteration 2, it's time to get their hands on it. Structure this in such a way that:
   1. They are using the product in a way actually reflective of the way they work,
      
   2. It has minimal impact on their busy schedules,
      
   3. They aren't necessarily testing the same areas,
      
   4. You get periodic feedback from them in a group setting.
      
3. UAT standup: While the team is developing in iteration 2, your Agile team lead (or Scrum Master or business analyst), maybe with the Product Owner, is having quick feedback-gathering sessions with the UAT team…maybe 2x/week. Do this as a group, so members can hear each other's observations. Ask questions such as "How does this help make your job easier?" or "What would you change about these features?" Notice these are NOT 'yes' or 'no' questions. Open-ended questions elicit discussion and feedback…and that's what you want here.
   
4. Cycle feedback into the Product Backlog: The Scrum Master or business analyst can now review that feedback with the Product Owner, for possible inclusion in the backlog for a future iteration. Create user stories from the feedback, and have the P.O. prioritize them.
   
5. Bring feedback stories into future sprints: So now, let's say iteration 3 is beginning. And feedback from the UAT held during iteration 2 has made it to the top of the backlog and is brought into iteration 3. At the demo for this iteration, your UAT group will see their feedback brought in as actual working product before the release.
   
6. Lather. Rinse. Repeat.
   

Iterative UAT

What are the benefits we're likely to see from this process?

1. UAT users become invested parties: They are providing valuable feedback, helping shape the product as it's being developed;
   
2. Product better aligned to user needs: By having UAT testers interact with the product according to their actual work patterns, instead of a script, we are more likely to get realistic, usable feedback;
   
3. UAT users become community advocates: Imagine your reaction to this process if you are one of the UAT group. "I saw the product in development, tried it, gave feedback, and they actually listened and made the changes I suggested before they released the product!" These users will become strong SMEs on the product, and advocates for your process within the user community. And THAT is a big win in an Agile adoption.
   

 
I would be very interested to hear your thoughts and observations on UAT in iterative development. So as always, I welcome your comments.