Friday, December 30, 2005
agile usability by committee
You know something is brewing when the end-of-year issues of both SIGCHI's Interactions and the UPA's User Experience have articles on so-called "agile usability." I welcome the exploration of new approaches, especially hearing about real world experiences with these. Agile usability is an attempt to incorporate usability (sometimes loosely defined) with agile software development methods. It addresses a screaming problem in software development: the complete lack of an explicit role for UCD input in the standard frameworks of software development methods, particularly the "Rational Unified Process" that has such a stranglehold on development these days. IBM should be taken to court for promoting RUP as good practice when UCD at most is considered an optional bolt-on accessory (supply your own bolts and hope you can find the right size.)
First the good news: agile software development methods are generally better than RUP in reflecting user needs. The bad news: agile is not that much better. Can fusing usability into agile programming make the product truly reflect user needs? The jury is still out, the experiment is still unfolding. But I am wary. Agile usability seems like a band-aid solution to a trauma wound.
Agile programmers seem like cool people. They experiment, listen to others, and dislike stuffy paperwork. Not only are they cool, they even use a few words used by the UCD community, notably "iteration." (Alas, what agile programmers consider an iteration and what UCD folks consider an iteration differ widely, a faux ami. )
While discussion is an endearing quality of agile methods, an important party seems to be missing from the discussion: those anonymous folks known as users. Agile advocates will protest that I exaggerate here: they invite a person variously called a "customer representative" or a "user surrogate" to chat with the programmers. But it is important not to let the conversation get too big, otherwise stealthy character of agileness is lost.
At the heart of agile methods is meetings, generally very small meetings (we want to be agile, after all), but meetings all the same. At these meetings, programmers try to model what should be happening with the program. The two or three people meeting decide what to do next with the program. What do they base their decisions on? I see two major sources of feedback for agile programmers: how the code is performing in meeting perceived needs, and conceptual models that the programmers create to think through what users need from the program.
There are some significant risks associated with using functional prototypes as a foundation for the final end product. You get invested in your solution when you choose not to throw away alternatives, which is the beauty of paper prototyping. Your UI can get enmeshed in your functional domain model, making it difficult to change. You are prisoner to the fundamental problem associated with any form of incremental design, namely that you invariably develop a solution that "satisfies" (is the first solution to minimally meet needs at hand), rather than a solution that looks at broader and longer term issues, and seeks to find the best alternative given the wider trade-offs. Sometimes you end up a blind alley, as your evolving solution fails to scale to the growing complexity of needs.
I find troubling the notion that conceptual models can serve as adequate proxy of user needs. Programmers are smart people, and love models, especially high level abstract ones. It should be no surprise that techniques that promise to model user needs are the techniques of choice embraced by agile programmers. The models favored are variants of either scenario-based models, or usage-based models. What are these user models based on? Sometimes they are just based on a conversation between a pair of programmers. If more elaborate, the programmer pair might call a meeting to get input from some other stakeholders, such as the customer representative. But what these models aren't based upon is proper user research. The scenarios and "usage" reflects what a bunch of people sitting around a conference table said, and no more. All kinds of assumptions are made, and never verified, in such scenario and usage modeling. Models reflect the tunnel vision of their creators. They lack the peripheral vision gained by widespread consultation with users before design and during development.
What is most lacking from agile usability is a formal role for user testing. There may be a grudging acknowledgement that user testing is useful is limited circumstances, and a few UCD consultants have managed to sneak-in testing on an agile development project. But generally agile programmers see usability testing as a time waster, and unless and until that attitude changes, agile usability will be only be agile without the usability. Some agile advocates, particularly Larry Constantine, claim you really don't need to test to attain usability. "Our view [of usability testing] is not uncritically positive," he writes:
That quote might even sound reasonable out of context, until you see that Constantine devotes only a few pages to usability testing in his 500 page book (Software for Use) that is supposedly about usability. Constantine is a critic of usability testing, considering it too expensive and inefficient (if it weren't for the fact that usability testing "plays such a prominent role in the business of software development", I wonder if he would even acknowledge the limited benefits he concedes it offers). He proposes to "upgrade usability" through is own methodology of usage centered design, which which in his mind effectively eliminates the need for testing. Somehow Constantine fashions himself as a usability expert, but he dismisses what 99% of other usability experts consider the foundation of usability: usability testing. How Constantine can call usability testing "a specialized tool", as though it was on the fringes of common use, escapes me.
I happen to think Constantine's usage centered design (to quote his own phrasing) "can be an important and useful tool in service of enhanced usability so long as it is recognized as only one specialized tool among many." But Constantine would have you believe his approach is the only one that matters (the book's list of references contain mainly his own writings). We are back to the old days, when checking real world usability is an afterthought, merely tidying up a few minor details. If only things were that simple.
The hubris of usage centered design is the conceit that a select few can know the needs of many through enlightened processes. Constantine does briefly speak about the need to get information from real users, but he devotes most of this discussion to how to get information about users at arm's length (from surveys, for example, rather from realistic settings.) Whenever users are mentioned, the discussion is short (not enough to act on), perfunctory (by acknowledging that true, some people do direct user research, which in limited circumstances might be useful for some people, if interested look elsewhere for details) and ambivalent (dealing with users involves "chaos"; a lack of enthusiasm for UCD abounds.)
Constantine wants to "move away from purely user centered approaches to software design." I'm all in favor of improving design methods, and reducing the amount of testing needed, even iterative testing. There is too much software to test properly, so testing needs to be prioritized. But while Constantine has identified a valid problem, and even offered some additional tools to deal with the problem (mostly task modeling), it is highly grandiose to imagine he has solved the problem. In Constantine's view, modeling will produce mostly usable software (what he calls "built-in usability"). Any remaining problems can be addressed through "collaborative usability inspections", in other words, more people chatting while sitting around a conference table.
I may be entirely wrong: perhaps one can design completely usable software without doing either user research or user testing. One can simply rely on design methods and usability rules, and presto, a usable software system emerges. But even though I respect the power of methods and rules to improve products for users, I am unaware of any combination of methods/rules that would guarantee fully usable software. Best practice is useful, but insufficient. There are too many variations for best practice to address, too many unknowns about users, too much innovation happening, too little certainty about how all these factors interact. Perhaps years from now, when user research has uncovered its last discovery and when technology has evolved to a point of standing still, we will have a science that won't require users to offer their inputs into requirement and show their performance during testing. Until then, modeling and inspections seem like a recipe for missed requirements, unforeseen interaction problems, and confused people.
I focus on Constantine's views in particular because for many people in the agile programming world, he is the face of usability. [Disclosure: I've never met Constantine or even know anyone who has. My criticisms of are the methods he advocates, not of him as a person.] Constantine is a major writer on the Yahoo agile usability list, a list more dominated by programmers than usability professionals. The people-free "usability solution" offered by usage centered design is no doubt appealing to some programmers. But if agile programmers are going to learn what usability is about, they need to get a representative presentation of usability, especially the importance of user testing.
First the good news: agile software development methods are generally better than RUP in reflecting user needs. The bad news: agile is not that much better. Can fusing usability into agile programming make the product truly reflect user needs? The jury is still out, the experiment is still unfolding. But I am wary. Agile usability seems like a band-aid solution to a trauma wound.
Agile programmers seem like cool people. They experiment, listen to others, and dislike stuffy paperwork. Not only are they cool, they even use a few words used by the UCD community, notably "iteration." (Alas, what agile programmers consider an iteration and what UCD folks consider an iteration differ widely, a faux ami. )
While discussion is an endearing quality of agile methods, an important party seems to be missing from the discussion: those anonymous folks known as users. Agile advocates will protest that I exaggerate here: they invite a person variously called a "customer representative" or a "user surrogate" to chat with the programmers. But it is important not to let the conversation get too big, otherwise stealthy character of agileness is lost.
At the heart of agile methods is meetings, generally very small meetings (we want to be agile, after all), but meetings all the same. At these meetings, programmers try to model what should be happening with the program. The two or three people meeting decide what to do next with the program. What do they base their decisions on? I see two major sources of feedback for agile programmers: how the code is performing in meeting perceived needs, and conceptual models that the programmers create to think through what users need from the program.
There are some significant risks associated with using functional prototypes as a foundation for the final end product. You get invested in your solution when you choose not to throw away alternatives, which is the beauty of paper prototyping. Your UI can get enmeshed in your functional domain model, making it difficult to change. You are prisoner to the fundamental problem associated with any form of incremental design, namely that you invariably develop a solution that "satisfies" (is the first solution to minimally meet needs at hand), rather than a solution that looks at broader and longer term issues, and seeks to find the best alternative given the wider trade-offs. Sometimes you end up a blind alley, as your evolving solution fails to scale to the growing complexity of needs.
I find troubling the notion that conceptual models can serve as adequate proxy of user needs. Programmers are smart people, and love models, especially high level abstract ones. It should be no surprise that techniques that promise to model user needs are the techniques of choice embraced by agile programmers. The models favored are variants of either scenario-based models, or usage-based models. What are these user models based on? Sometimes they are just based on a conversation between a pair of programmers. If more elaborate, the programmer pair might call a meeting to get input from some other stakeholders, such as the customer representative. But what these models aren't based upon is proper user research. The scenarios and "usage" reflects what a bunch of people sitting around a conference table said, and no more. All kinds of assumptions are made, and never verified, in such scenario and usage modeling. Models reflect the tunnel vision of their creators. They lack the peripheral vision gained by widespread consultation with users before design and during development.
What is most lacking from agile usability is a formal role for user testing. There may be a grudging acknowledgement that user testing is useful is limited circumstances, and a few UCD consultants have managed to sneak-in testing on an agile development project. But generally agile programmers see usability testing as a time waster, and unless and until that attitude changes, agile usability will be only be agile without the usability. Some agile advocates, particularly Larry Constantine, claim you really don't need to test to attain usability. "Our view [of usability testing] is not uncritically positive," he writes:
Our own view of usability testing is that it can be an important and useful tool in service of enhanced usability so long as it is recognized as only one specialized tool among many. Particularly in the absence of good models or methods of design, usability testing is indispensable. Testing, however, is never sufficient in itself to
deliver highly usable software. [my emphasis]
That quote might even sound reasonable out of context, until you see that Constantine devotes only a few pages to usability testing in his 500 page book (Software for Use) that is supposedly about usability. Constantine is a critic of usability testing, considering it too expensive and inefficient (if it weren't for the fact that usability testing "plays such a prominent role in the business of software development", I wonder if he would even acknowledge the limited benefits he concedes it offers). He proposes to "upgrade usability" through is own methodology of usage centered design, which which in his mind effectively eliminates the need for testing. Somehow Constantine fashions himself as a usability expert, but he dismisses what 99% of other usability experts consider the foundation of usability: usability testing. How Constantine can call usability testing "a specialized tool", as though it was on the fringes of common use, escapes me.
I happen to think Constantine's usage centered design (to quote his own phrasing) "can be an important and useful tool in service of enhanced usability so long as it is recognized as only one specialized tool among many." But Constantine would have you believe his approach is the only one that matters (the book's list of references contain mainly his own writings). We are back to the old days, when checking real world usability is an afterthought, merely tidying up a few minor details. If only things were that simple.
The hubris of usage centered design is the conceit that a select few can know the needs of many through enlightened processes. Constantine does briefly speak about the need to get information from real users, but he devotes most of this discussion to how to get information about users at arm's length (from surveys, for example, rather from realistic settings.) Whenever users are mentioned, the discussion is short (not enough to act on), perfunctory (by acknowledging that true, some people do direct user research, which in limited circumstances might be useful for some people, if interested look elsewhere for details) and ambivalent (dealing with users involves "chaos"; a lack of enthusiasm for UCD abounds.)
Constantine wants to "move away from purely user centered approaches to software design." I'm all in favor of improving design methods, and reducing the amount of testing needed, even iterative testing. There is too much software to test properly, so testing needs to be prioritized. But while Constantine has identified a valid problem, and even offered some additional tools to deal with the problem (mostly task modeling), it is highly grandiose to imagine he has solved the problem. In Constantine's view, modeling will produce mostly usable software (what he calls "built-in usability"). Any remaining problems can be addressed through "collaborative usability inspections", in other words, more people chatting while sitting around a conference table.
I may be entirely wrong: perhaps one can design completely usable software without doing either user research or user testing. One can simply rely on design methods and usability rules, and presto, a usable software system emerges. But even though I respect the power of methods and rules to improve products for users, I am unaware of any combination of methods/rules that would guarantee fully usable software. Best practice is useful, but insufficient. There are too many variations for best practice to address, too many unknowns about users, too much innovation happening, too little certainty about how all these factors interact. Perhaps years from now, when user research has uncovered its last discovery and when technology has evolved to a point of standing still, we will have a science that won't require users to offer their inputs into requirement and show their performance during testing. Until then, modeling and inspections seem like a recipe for missed requirements, unforeseen interaction problems, and confused people.
I focus on Constantine's views in particular because for many people in the agile programming world, he is the face of usability. [Disclosure: I've never met Constantine or even know anyone who has. My criticisms of are the methods he advocates, not of him as a person.] Constantine is a major writer on the Yahoo agile usability list, a list more dominated by programmers than usability professionals. The people-free "usability solution" offered by usage centered design is no doubt appealing to some programmers. But if agile programmers are going to learn what usability is about, they need to get a representative presentation of usability, especially the importance of user testing.
However broken existing software development processes may be, with its inability to reflect user needs, I hope we can develop a meaningful solution to the problem, and not a lesser of two evils solution. For the moment, agile usability is just somewhat better than RUP. Let's hope we can get a real UCD solution embedded in the software development process, before agile usability gets entrenched, with everyone believing the problem has been solved.
Link to this article. posted by Michael Andrews @ 9:44 AM 
