© UX Research, 2008

Usability in Agile development

John Eklund and Ciaran Levingston

UX Research
info@uxresearch.com.au

ABSTRACT now enjoys fully online banks, mobile internet and ecommerce
Agile methods of development prescribe rapid prototyping and on a scale of change that is unprecedented in our modern world.
minimal documentation, and are particularly suitable for These changes are driven by the demands of users and the rapid
creating web interfaces for businesses, encouraging throw-away pace of technological change. In the last ten years we have also
prototypes and allowing designers the freedom to be more seen dramatic changes in software development processes. Our
creative. There are additional challenges in including usability once immature industry is changing and it is no longer
techniques in Agile development, as it has multiple iterations acceptable nor advisable for development agencies to ignore
and often substitutes the client as the user-advocate. This paper sound development practices. Clients demand solid project
examines some of the ways Agile methods are being practiced management as well as an assurance that final deliverables will
in web development and discusses approaches to user centred meet their expectations as well as those of their customers. The
methods that meet the challenges of rapid development to usability industry has taken a key role in software quality
assure the usability of systems and ultimately customer assurance. It is now generally accepted that some form of user
satisfaction. Some of these methods are part of a more recent testing must be build into development lifecycles. Usability and
trend sometimes called Usability 2.0, which aims to make user user centered design are understood, frequently used and it’s
centred methods more accessible, practical, affordable and of business benefits recognized by developers and business
demonstrated value to the strategic directions of a business. As stakeholders.
we shall see, none of these methods are really new, but useful In development circles, timeframes are contracting with clients
adaptations of a well established approach that is proven to demanding implemented solutions to be delivered in less
work well. elapsed time. These systems are often web-based as the suite of
technologies available for rapid web development has increased
Keywords and more self-service functionality is being made available to
HCI, User Centred Design, Agile, Usability, Software the end user. Websites are becoming far more the engine-rooms
Development Methods. of businesses, no longer the static brochure-ware meant to
attract customers, but attract and transact with them, making
business easier, less costly and more globally delivered. All this
1. INTRODUCTION has driven changes in software development, as more
sophisticated software companies enter the marketplace,
1.1 About this paper merging media and digital offerings, and seeking to distinguish
This paper examines modern approaches to Agile development, themselves with solid methodologies and time-driven successes.
its theory and common practices and examines ways that
usability practitioners can effectively integrate their services The key question for companies like UX Research, providers of
into Agile development lifecycles. The research is a result of outsource usability services, is “How can we adapt to meet this
discussions with a number of software development changing environment?”
organisations that provided a picture of some of the particular
practices they use as part of their Agile approaches. The authors 2. THE EMERGENCE OF AGILE
themselves are not developers, but usability practitioners and
their aim is to give an overview of some Agile methods in 2.1 The meaning of Agile development
practice. The paper briefly covers usability in the traditional Agile appears to be one of those terms that get used sometimes
software development lifecycle, and some of the considerations where it shouldn’t, and it is being applied in a great variety of
that will need to be addressed to allow usability to successfully settings and in many different ways. Possibly this is acceptable,
integrate with Agile development approaches. Finally, we after all it really means “flexible”, “adaptable”, “nimble”, and
propose some tangible models for the ways that outsourced this can also be applied to its definition. How agile development
usability providers can engage with development teams that are is done in practice depends on many factors, including the
following Agile approaches to make usability services more particular skills of the development team, the particular project
accessible and understandable. that it is being applied to, and the needs of the client. Some may
see it as a way of shortcutting expensive processes early in
1.2 A climate of change development, such as specification, in order to save budget. For
We are in a period of significant change with the way others it’s an exciting new way to get things done faster and
technology is affecting our lives. Our children are growing up getting deliverables to the client earlier. It requires quite an
with access to new tools for business and recreation beyond organisational shift within development teams that needs the co-
anything we could have imagined a decade ago. Our society operation of project managers, designers and importantly, the
client. One clear outcome of our investigation in Agile is that

1
© UX Research, 2008

clients need to be onside and committed to the process for it to doesn’t approve of aspects of the interface design. It may be
work. that this model is suited to developing large-scale systems but it
also depends on stable requirements that are correct. There is
At its base level we can say that agile is a type of iterative
clear statistical evidence indicating the importance of getting
development. the requirements right and that the cost of modifying a system
increases exponentially as the product gets closer to release.
Establishing clear and accurate requirements is a difficult and
time consuming process. This means that in a more traditional
model such as the V model, a lot of the work is done in the
requirements analysis phase and in developing detailed
specifications, and unless these are correct and stable, the
system will fall short of expectation at some point.
Agile by contrast recognizes that requirements will change and
this stage is not removed but simplified and blended with the
production of high-level designs.

So Agile says “requirements will always change, we accept that

Figure 1. Agile is based on iterative development and structure our project planning to allow for it.

Agile aims to move forward the design phase in a development

lifecycle and replace or reduce the amount of time spent
documenting a system before prototypes are developed. We will
deal with that aspect of it, it’s impact on the design process, but
it is also relevant to ask “why it has emerged?”

2.2 History
It is worth remembering that the core of Agile is not really new,
and that there is a long established history of Incremental
Development (ID or iterative development). The earliest
documented instance of a project that used multiple iterations
was in the 1930s, and it seems it was in the 1990s that the
notion of iterative development superceded the Waterfall model
(Larman and Basili, 2003). Agile became a properly established Figure 2. An iterative development lifecycle
method when in 2001 an organisation called the Agile Alliance
was established, with goal to share information and techniques
on this method. Like most things, including iterative In an Agile world, the stages of gathering requirements and
development or Usability 2.0, Agile in the modern world is not designing overlap. Agile prescribes a process of gathering some
new, but it is different. It’s on the web; it’s developing business basic requirements, start designing, and as you do so the
critical systems for Small to Medium Enterprises (SME); it’s remaining requirements will emerge.
done faster and with new tools.
When you deconstruct Agile development, at its base level it is
Agile methods have emerged as a response to a changing an iterative development methodology. The basic structure of
environment for development. New technologies allow designs Agile development is simple, cycle through the specification,
to be created and experimented with easily, later stages of design and evaluate stages several times refining requirements
development can be entered earlier, greater access to a range of at each iteration. In the evaluation stage, whether it is an
prototyping tools and developers to be more creative. It informal review of designs or formal usability testing, it is
acknowledges that web-based business critical systems must be important that it be performed regularly and done with a certain
developed rapidly but with sound processes, and it allows degree of independence.
changing requirements to be discovered through designs
(Eklund & Lowe, 2002). If we say that Analysis includes eliciting requirements, namely
information gathering activities, then we could almost remove
all but the high-level requirements from the process. In some
2.3 Development models ways that’s what Agile does: De-ephasise requirements, and
Development models like Waterfall (Parida, 2006) depend on embed them in Analysis and Design.
building a detailed specification on which design is based.
These models are linear in that they have no feature to return to Iterative development can be done by creating successively
the designs and make changes, nor validation that requirements deeper designs, starting with flat screens and moving to more
are complete. Crunch-time comes on delivery of the finished functional prototypes, until the final system is delivered. This is
system, when its typically the case that the client recognises that a standard Agile approach and has the benefit of allowing
requirements are missing, introduces new requirements, or

2
© UX Research, 2008

clients and users to provide feedback on prototypes as 2.5 Benefits

development progresses. There are several important advantages of this approach: As
Each time we perform an iteration from concept to functional requirements are usually expressed as documents, which are
system there is some form of evaluative process, which would often inaccessible to clients, Agile means designers spend less
appear acceptable in theory but there is a potential problem with time creating documents and more time building (sometimes
this approach. disposable) prototypes that can be used to elicit requirements.
Many of us will have been on one side or the other of this
scenario. It’s not until designs are presented that the client is
able to articulate what it is they wanted. No matter how
2.4 Key differences in Agile Iterative carefully a specification is done, until the tangible designs are
produced, the client has trouble providing feedback that is
development valuable in gathering system specifications. This is often due to
In traditional development the system is built in layers, with all the density of specification documentation which clients and
the functionality delivered in the resulting product. A good business stakeholders find difficult to interpret.
analogy is building a house. You start with the foundation that
encompasses a complete set of functionality and create An advantage of this approach is that it reflects practices in the
successive layers from the ground up. real world. Many businesses are unsure of what functionality
they require in a system and they do not have a clear idea of
what the User Interface will ultimately look like, so Agile
attempts to overlay time consuming requirements gathering
with designing in order to elicit such system specifications as
may be required.
This gives clients a feeling of security in that they can see
system designs early on in the project and thus provide more
tangible input throughout the life of the project. Clients get
greater a sense of involvement in the project and a higher level
of control of the direction of the project as a whole.
Agile also has the benefit of allowing the designers more
freedom to experiment with their designs as they are less bound
Figure 3. Layered approach in traditional development by a specification. This means there is more scope to be
creative, allowing multiple sketch solutions to be put forward.

3. AGILE IN PRACTICE
3.1 Agile and team dynamics
Ultimately a successful project is about a deliverable piece of
software which is created by people and key to that is
successful people management. Everyone has a role to play, and
Agile methods prescribe some significant changes to traditional
team dynamics.

The client seeks the expertise to create a system that will meet
Figure 4. Functional approach to Agile development his business goal. He has a view of what he wants and what
users will do in his system, and starts his project with
enthusiasm and energy. Then there is the project manager who
Agile iterates differently, selecting core functionality to have represents the best interests of the development organisation,
several stand-alone deliverables, allowing the client to adjust making sure the deliverables are met by those on his team,
scope at each iteration based on what has been delivered. In this managing client expectations of scope, and generally
way it aims to deliver something that’s working and complete distributing project plans as they are revised. He understands
early and often. the business and technical constraints and is key to managing
relationships. They are joined by Creative people, designers
So in Agile we start designing early and spend less time who are responsible for developing the user experience and
documenting. The most important aspect is that the creating exciting and dynamic visuals that meet client needs.
Specification (or Requirements) and Design phases are
overlayed, they co-exist. Specifications themselves become less
formal, easier to produce, are often presented as designs The team also consists of an outsource usability practitioner or
themselves, and develop as the designs become more detailed. interaction designer who provides an independent analysis of
designs and makes structured recommendations based on user
responses and against best practice and usability standards. He

3
© UX Research, 2008

is an advocate for the user and is frequently commissioned by usability analyst will have had exposure to and interacted with
the client. In addition there is the technical team without whom many systems and can advocate for best practice and standards
we wouldn’t have any HTML code or a functioning server. The throughout the design process.
key point is that all these people have different skills, goals and
attitudes and getting them all to work together can sometimes 4.1 A working definition
pose quite a challenge.
In summary, Agile methods are ones that organise development
In traditional Development, the project manager oversees the teams differently to promote open communication and involve
work of the designer and is in charge of the relationship with multiple development iterations. It anticipates that clients are
the client. Communication is principally done through the uncertain of their requirements and that requirements will
project manager. change throughout the life of the project. In Agile methods
scope creep does not mean a poorly specified project, but is
built into the development methodology. The process of
designing earlier allows requirements to emerge, change or be
abandoned. A key differentiator is in the way the system is
iterated in groups of functionality. This allows the delivery of a
working system, not just a prototype, which contains core
functionality. This allows projects to be signed off in stages and
scoping of the project to be simplified. This gives us a working
definition:
Agile is a term used to describe a type of iterative development
that relies on designing early as a means of discovering
requirements, producing usable systems in stages of
functionality designed to meet the uncertain needs of modern
business systems.

5. ASPECTS OF UCD IN AGILE

Figure 5. New team dynamics specified by Agile This section discusses the role of usability in Agile
development. Usability as a word has many interpretations – it’s
a discipline, an industry, a method, but it can be thought of as a
In Agile, the client is in closer contact with the designer, and the statement of quality about a system. We can say something is
project manager becomes the overseer of the process, less highly usable if users can complete their tasks with ease. In
directly involved in the detail of the design. This structure other words, it’s a measure of quality that is judged by the
affords greater client involvement and gives them the extent to which the design and functionality of a system meet
opportunity to take the role of advocate for the user group. The user requirements, whether these requirements are explicit or
challenges for the project manager in this situation are to ensure not.
the designer and client work well together and stay on scope.
5.1 Usability in theory
In User Centered Design, the evaluation process clearly
4. USER CONSULTATION MODELS involves an evaluation of designs against user requirements, and
this is typically done by an independent analyst.
In some organizations user consultation is done by the
designer/developer by gathering business requirements from the When we undertake usability testing there are two main sets of
client by eliciting “stories” which are the various tasks a user benchmarks against which the designs are judged. The first is
would undertake on the system. These are often written on post- the needs, requirements and perceptions of users. They are
it notes and become a sort of defacto set of user and functional given common tasks to perform and the degree to which they
requirements that can be embodied in designs. Clearly in this can complete their goal, and their satisfaction in doing so, forms
scenario it is still client conjecture as to what a user would want much of the basis of a study. It mixes observation on tasks with
from the system which forms the basis of user requirements. interview and questionnaire, and this provides a highly
contextual view of the system and how users respond to it.
A better method is to include direct consultation with users, a
sort of participatory design process, where the client as well as a However, usability testing is not all about what users say and
group of users review designs and provide feedback. Now we do, there is a significant component of expert analysis,
have direct involvement of users, however two key things are particularly when it comes to making recommendations. The
missing from this scenario - an independent person to interpret usability analyst will have undertaken many usability studies
and moderate the user feedback, and feedback from best and be aware of best practice and well as guiding principles
practice perspective. relating to consistency, feedback and other principles. Issues are
identified and analysed against a set of criteria, described in
The final and best method for organising the team to produce
standard terminology such as navigation, page layout, aesthetic
high quality designs is to have an independent usability analyst
design and so forth. It is knowledge and understanding of these
provide input to the designs throughout and at key stages of the
heuristics by the outsource usability analyst which highlights
design process. The benefit of this is to ensure the maintenance
their value.
of independence and usability standards. An experienced

4
© UX Research, 2008

One way to look at usability in practice is in terms of These are the challenges for usability in Agile. Now there is a
perspectives and benchmarks. Perspectives are like layers of changed team dynamic, traditional team roles have shifted, and
quality. Yes the system needs to be usable, but also users must the client, occasionally supported by consultation with real
be motivated to use it. This is called user acceptance. users, acts as the user advocate. Since user consultation is now
more embedded in the development process, it is more difficult
Benchmarks are an essential part of any usability evaluation. to recognize the benefit of external usability consultation. It
These are the standards by which the quality of the design is becomes even more problematic to decide when to undertake
judged. These are informed by design principles and usability external usability testing, as there are many iterations, in what
principles first stated in Neilson’s 10 usability heuristics - can appear to be an ill-defined process.
things like the visibility of system status. And of course, best
practice is derived from these benchmarks and conducting many
studies and seeing what works in practice.
6. STRUCTURING USABILITY
SERVICES TO BE SUITABLE FOR
5.2 Usability in the real world AGILE
Usability Companies such as UX Research typically offer a
range of quality assurance services across the development 6.1 Principles and assumptions
lifecycle, from the requirements gathering phase through to In this section we discuss ways that usability practitioners can
design and build. This gives the impression that they are busily respond to this new development environment. We begin with
engaged across the whole project doing various good things to an analysis of the characteristics of usability services that will
make designs more usable and ultimately serve customers. fit with Agile development methods. Firstly, as the process in
its initial stages is creative and flexible, it will be important to
The reality is somewhat different - our experience is that be able to call on usability analysts in a flexible way, which is
external usability testing tends to be at one, possibly two places dependant on the needs of the project and the issues that arise.
in development lifecycles. The larger, higher profile projects we This means having an arrangement with a usability provider
have worked on have tended to increase the numbers of users in already in place, knowing their process and methods and costs,
testing, rather than the number of test cycles. In addition,
and being able to commission them to undertake usability work
despite the acknowledgement that usability evaluation is quickly.
invaluable in the early design phase, in our experience it is
placed much later in the process when most of the design is Secondly, such advice will need to be at a detail level, frequent
locked down. This reduces the usability study’s focus to detail and as needed by the team. This advice will be related to
elements at a page level, rather than more fundamental aspects principles and standards and draw on the expertise of the
such as functionality or information architecture. analyst who has a knowledge of current and best practice.
The reasons for this are twofold. It’s partly that some see it as a Further, due to the fact that Agile methods involve multiple
validation exercise, which is fine if user centred design has been iterations, its likely that usability advice will be required
undertaken in development. It is also that project development multiple times, and testing will be executed quite quickly and
teams have found it difficult to engage external usability people. for less cost. Project teams will not have the luxury of time to
It is seen as quite a chunk of time taken away from activities wait for a detailed report, and will seek input more directly into
directly related to development, and developers don’t want a design decisions.
critique of their work until they are certain they are ready for it.
If the purpose of usability testing is validation, this attitude In forming a response to how the usability services of an
makes some sense. This mindset is however slowly changing external organisation can fit into Agile development methods,
and developers are starting to see the value of external usability we start with the basic assumptions that external usability
at different stages of development. Depending on the stage at testing must be employed at some point as it’s a fundamental
part of User Centered Design. We know it works, as it is a
which it takes place, usability evaluation is different. Early in
the development lifecycle it will be more exploratory, and method that has been used for decades though it appears in
contribute more directly to designs, while later on its more a various guises such as Usability 2.0. We know as providers of
matter of validating the quality of the user experience for these services that we can add value in each and every instance
business owners. It will almost always have a strategic but in making external usability testing faster and cheaper we
component, offering advice to business owners that will cannot use machines or substitute junior staff as the quality of
improve market positioning or customer retention. usability insights gained through testing is always going to rely
on the experience of the analyst to make the appropriate
judgments in each case. We think that the provision of usability
services in an Agile environment is a matter of making usability
5.3 Usability in Agile is a challenge testing more efficiently delivered.
It is no surprise that organisations are struggling to see how
usability services can fit into Agile development lifecycles.
Many were just becoming comfortable with incorporating
usability into normal development methods and with the
6.2 A UCD approach for Agile
With a limited budget for usability testing on any project,
tendency to perform larger external testing later in
development, it is not surprising that development companies traditional development typically included one or two rounds of
are having difficulty fitting external testing into their project user testing. In Agile development we aim to include expert
plans and budgets. advocacy and more rounds of smaller test events as required.

5
© UX Research, 2008

Expert advocacy is being available as a usability expert to a button is poorly labeled we wont spend any time discussing
comment on designs, solve design problems, be an advocate for that with user, but create tasks that direct them to other areas of
the user, and ensure standards are being met. This is best done the site to explore new problems. Including business objectives
in short meetings usually without any report or making a in user testing is also important. If a site’s business goal is to
comment or opinion by email. This has the benefit of solving generate signups the focus of the testing is to find the points
simple problems, which do not require user testing to be that cause most users to drop out of the flow.
identified and addressed. It also provides the usability analyst
with more involvement in the design process allowing them to The test sessions in Agile must be hosted in a flexible manner
better plan for usability test events. also, sometimes with a very consultative style that treats the
user as a partner in discussing issues, which is a less pure
A good example of how advice can reduce time spent in formal hosting protocol that challenges users to confirm or deny mental
testing is recently one of our consultants was meeting with a models. In some testing protocols you might say that such
client to discuss what to include in the next round of testing. questioning is leading, however the justification is that you only
There was debate around some screen elements, but to the have a short time with the participants and you need to get as
consultant the answer was clear, as the same issue had much feedback from them as possible.
presented itself in a previous study for a different client. We
convinced them to adopt these suggestions, several of them, In Agile, often there will be multiple designs to test, and so
feeling confident that we knew what would work. This occurred comparative testing becomes part of the process. In fact
with a number of design elements, until it was realised that comparative testing is an important component as it allows the
many of the issues on the list that were thought to require users to judge designs against each other rather than some ill-
defined absolute standard. It is advisable to include some
testing were now resolved. It was decided to delay user testing
until some other aspects of the design had progressed. This comparative components in all Agile testing, particularly in
example shows the value of time flexible advice closer to the early iterations.
design team. Finally, Agile demands that usability is closer to the design
process while remaining independent of it. This is so user
testing can add more direct value and have immediate input to
designs. This can be done in a number of ways: The usability
consultant as a design team member, called upon for occasional
consultation as an advocate; Faster testing with a participatory
design component aimed at eliciting requirements; testing on
client premises, and making usability outputs more ‘design
friendly’ with reports centred on recommendations supported
by observation.

7. CONCLUSION AND
RECOMMENDATIONS
In conclusion we offer a set of recommendations for those
organisations wishing to embark on Agile development projects
and concerned about how to include usability testing in what is
a challenging and exciting approach to development.
Figure 6. Models of UCD in traditional and Agile Firstly, it is important to constantly remind ourselves of the first
principle of User Centered Design – to involve users at several
stages of development. If the design process involves the
Usability testing itself has to be shorter and faster, and that designer working closely with the client, then the use of
means cheaper. The whole lifecycle for usability testing can be external users as a sounding board for designs will obtain
as short as a week or even less, including five days to recruit the valuable insights. Give users the tasks they would undertake in
participants. Testing can be done in one day with up to eight the system and gauge their reactions. User involvement can be
sessions, using a portable usability lab on the client premises, informal, but it must exist.
with a report delivered to the client two days later.
Secondly, we recommend the engagement of an external
Testing on the client site has its challenges but a test usability specialist to give advice as required. They can help
configuration can be created quite simply. The advantage this solve design issues, particularly minor interface issues, speed
has is that business people have more opportunity to participate development and save user testing time that can be used in areas
by watching the testing by dropping into the observation room that are more problematic. It will be important to consult with
when they can. Other organisations are making sessions more the usability consultant about the timing and scope of the
accessible through the use of various video streaming testing that is required rather than commissioning testing
technologies, such as those in Morae. against a set project plan
The usability testing itself is also different. It is more Further, if the project prescribes one round of testing,, ask for
exploratory at earlier stages with a more open test script and two smaller ones. If your usability provider suggests two rounds
incorporates usability standards. In many ways the users are with 12 users in each, ask for 3 rounds of 8.
there to help the analyst find problems - for instance if we know

6
© UX Research, 2008

Lastly, when usability testing is undertaken, ask your analyst to http://ezinearticles.com/?Essence-Of-Waterfall-

gather requirements as well as test designs. Expect a more open Model&id=352291
and exploratory sessions early in development, and more 3. Larman C & Bascili V R (2003) Iterative and Incremental
validation later in development. Work the test script through Development: A Brief History. Computer, June 2003
with the analyst and endeavor to involve the business in Available online at
observing sessions, which may be facilitated by testing on the http://www2.umassd.edu/SWPI/xp/articles/r6047.pdf
client premises, to help engage the business. A report should be
available two day after testing, which will allow designing to 4. Introduction to Agile Usability: User Experience Activities
proceed. The report will always offer both strategic and design on Agile Development Projects. Scott W.Ambler.
recommendations. http://www.agilemodeling.com/essays/agileUsability.htm
5. Agile Development Processes: An Interview with Jeff
Patton. By Jared M. Spool and Joshua Porter.
http://www.uie.com/articles/patton_interview/
8. ACKNOWLEDGMENTS 6. Usability and User Interface Design in XP by CC Pace
The research in this paper is based on a series of industry
Systems.
seminars about Usability and Agile methods that were given by
http://www.ccpace.com/Resources/documents/UsabilityinX
UX Research in May and June 2008, and at STANZ in August
P.pdf
2008. These were sponsored by The Web Industry
Professional’s Association (WIPA) and Software Testing 7. DIS 2002: Xtreme Programming makes Good Partner to
Australia New Zealand (STANZ) respectively. User-Centred Design. Source: UN, 13 August 2002
Submitted by Ann Light.
http://www.usabilitynews.com/news/article623.asp
8. Are Agile and Usability Methodologies Compatible?
9. REFERENCES Author: DÈsilets, A. Source: Hot Topics Newsletter
1. Eklund J & Lowe D (2002) Using Partial Designs to elicit (Carleton University). June 2005. NRC Publication
requirements in web development - a survey of commercial Number: NRC 48294
practice. Proceedings of Ausweb02.
2. Parida, P. (2006, November 9). Essence Of Waterfall
Model. Retrieved June 18, 2008, from