Modules and Wholes

I'm looking forward to discussion at next month's UPA conference on incorporating usability into agile software development methods (BTW: if you'll be in Denver and would like to meet up, drop me a line.)

One key question that needs a good discussion is how much user testing is needed in an agile development. How much testing is enough? Is testing always necessary?

These questions may seem to be a matter of opinion at the moment, but there is no reason why the usability community can't work to develop some data to answer these questions concretely.

At least one prominent advocate of agile usability believes that traditional user centered design methods involve an over-reliance on user testing.

I am concerned that agile software development can result in an under-reliance on user testing. By under-reliance, I mean that flaws in software design that affect users go undetected because they weren't tested during the development process.

While I am interested in agile methods, I can't claim special expertise. Insofar as I understand agile usability approaches, there is no unified approach people commonly follow. There have been some process-oriented methods that have been developed, as well as numerous improvisational attempts to find a balance that works. From what I have gleaned, these different approaches have not been explicit on how much user testing to conduct, and under what circumstances to conduct it, and at what point to conduct it. Some have suggested that successful results are possible with agile methods with less reliance on less user testing, because the design process is more robust, so problems are resolved before users ever see them. Others don't make this claim, though they admit that agile methods typically involve less user testing simply because the timelines and project structure limit the testing element.

What the usability community needs to know is whether doing a more light-weight version of user testing under an agile process is a net benefit, or simply a sacrifice to accommodate usability to the agile framework. Obviously, if there is no need for extensive usability testing in an agile process, and quality for users is as good as if more extensive testing was done, this is a big positive.

I'd like to see some comparisons of agile processes with limited user testing, compared with more extensive user testing. If agile methods can result in better designs that need little user testing, there will be few changes to the original design as a result of user testing. We would also expect that a very small sample of users would be adequate to catch any issues, so that larger numbers of test subjects would yield no new information. Once changes have been made to the design following the first round of user tests, any subsequent test would yield no further information. After one iteration, making any changes to the design would be superfluous.

Usability has a strong empirical tradition, where we look at and debate topics vital to our effectiveness, such as how many test subjects is optimal. Perhaps we can start developing some data about the effectiveness of testing with agile methods. We need to move beyond talking about over-reliance and under-reliance of user testing without clear definitions of criteria, and concrete data measuring the achievement of these criteria.