Usability studies are usually conducted in a compressed time scale (measured in hours) compared with a user's eventual experience with a product (often measured in years). For this reason, typical usability evaluations focus on success during initial interactions with a product (see for example Dumas & Redish, 1994 and Nielsen & Mack, 1994). Familiarity with similar products may predict success on initial use of a new product. Are what we call "intuitive" user interfaces really just familiar user interfaces? This familiarity effect can often swamp the usability differences between design alternatives. If usability evaluations continue to emphasize initial success with a product we may inhibit innovation in user interface design. There is a tension between initial usability (measured by success at first encounter) and efficiency of skilled performance. Initial learning of a product's user interface often results in quite rapid increases in efficiency of use whether it might be with a computer game or a business software application. The time spent with a product once up on a learning plateau typically greatly exceeds the time on the steeper part of the learning curve. Thus, traditional usability evaluation techniques that emphasize initial product use, may fail to capture usability problems that affect users the most. We do not question the importance of testing a products learnability, (see Usability Sciences Corporation, 1994) but feel that the human-computer interaction community should ensure usability throughout the product life-cycle. A narrow focus on initial usability elevates learnability above efficiency once up the learning curve. While this approach may be appropriate for products targeted primarily for casual or occasional users, it fails to capture the usability issues associated with power users (those with significant experience, training, or a professional orientation to their interaction with the product). Initial interactions with a product may affect the purchase decision, but usability over the longer term may determine whether a user will become a truly satisfied customer.
The goals of the workshop were to exchange and develop techniques to address usability testing of:
New features and incremental design improvements to an existing product with an established, highly-experienced, power user population. Example: a new version of a computer aided design program.
Innovative or novel products replacing older technologies used by an established power user population. Example: replacing a command line interface with a graphical user interface.
Entirely new technologies that may include a new and unfamiliar user interface and have no established user population. Example: the first Web TV appliance.
Thirteen highly experienced user interaction designers and usability engineers participated in the workshop (see Figure 2). In extensive pre-workshop e-mail communication, we shared our views of the critical issues in testing for power usability and each described in detail one real-world testing example.
From our electronic correspondence we learned that the products we design and test differ greatly, ranging from hand-held data entry devices to fighter aircraft. We come from many countries and work in different settings; from design groups in multinational corporations to consulting firms to academia. Despite these differences, we united with a desire to improve the state of the art in design and usability testing of products with an experienced user population and mission-critical requirements for error-free performance and productivity.
The workshop began with each participant presenting a real-world example of testing for power usability. These presentations included the nature of the task and work environment, description of the product (its revolutionary versus evolutionary nature, i.e., preexistence of a power user population), techniques used for usability testing, and lessons learned. During the presentations, all participants recorded brief descriptions of causes and potential solutions related to the problems of testing for power usability. Participants wrote these causes and solutions with large markers on color coded, self adhesive paper to facilitate later posting for group interaction.
From the presentations we agreed on a problem statement:
Power usability is not adequately addressed in product testing.
We then used an adaptation of the Ishikawa Root Cause Analysis technique (Ishikawa, 1982; also see Kimbler, 1995) to dissect the underlying causes of this problem and identify potential solutions. This analysis technique (also known as a Fish Bone Diagram) is a problem solving tool used in various quality management programs. Our adaptation elaborated on the fish bone analysis technique (see Figure 1) and included fish-related themes for the four steps in the analysis method:
Fish Bone: We posted the brief descriptions of the underlying causes of the stated problem that we had recorded during the presentations. We roughly clustered these causes on bones of a large model fish skeleton.
Name that Bone: With additional discussions and critiques we refined the arrangement and clustering of the causes and provided a name for each bone. This provided succinct statements of the underlying causes of the problem.
Filet-O-Fish: We posted solutions on the fish bone diagram; aligning these with associated problems.
Name that Fish: We reflected on the fish and identified conclusions and questions for further work
Simulating Scenarios & Tasks in Tests (how to duplicate the context of real work in a lab setting)
Identifying the Power User Profile (defining critical attributes)
Selecting Users for Testing (how to ensure that test subjects match the power user profile)
Designing Tests of Power Usability (subject training, metrics, data analysis, etc.)
The appendix presents the complete list of problems and solutions generated by the fishbone analysis. Note that the list is probably incomplete and the solutions may not be appropriate to every situation.
During the course of the discussion, workshop participants identified a partial set of power user attributes.
Experienced in the task domain
Experienced (and usually trained) with the product
Have a professional orientation to the product, typically in an occupational context
Emphasize efficiency and minimizing errors; frequently skilled at problem solving
Frequently in a state of flow -- at one with the machine, able to focus on task goal without thinking about the specifics of the tool.
In addition, we listed some system and environmental factors that often affect power users:
Context of use can be complex (e.g., complex task scenarios, multiple users, concurrent operation of multiple systems, collaborative work)
For some systems, power users are the only intended user group; but for other systems, they are a minority whose needs may conflict with a larger group of basic users.
Usability is often mission critical for systems designed for power users.
As with most areas of user-centered design, power usability must be considered in the initial phases of the design to have maximum impact. Here are some issues to consider in this regard.
Get the best data possible about the users, their tasks, flow behavior and the surrounding context
Test the current release or competitive product to identify problems before design starts
Set meaningful and measurable goals for power usability
Involve users in design (task domain experts, user review panels, focus groups, etc.)
Turn designers into users (train on prior or similar products)
Test early and often against the goals, making appropriate use of many testing tools
Validate user profiles, use scenarios and workflows at each opportunity
Tasks and subjects used in testing must be representative of real work and real users
Of course, no amount of laboratory testing will reveal all the usability issues with a product. To anticipate this, plan and budget for data collection after the product is introduced. Consider building data capture capabilities into the product for data collection in the user's environment. Focus on identifying long-term use problems to be fixed in the next version of the product. Remember that some problems may be architectural in nature, requiring more major changes and several releases to fix.
During discussions, we identified several questions for further inquiry:
Can the same product satisfy requirements of both power and non-power users?
Is power usability more "contextual" than normal usability or is the context more complex?
Is there a Power User "personality?"
When faced by a problem, do power users work collaboratively (ask for help) or try to solve the problem by themselves?
Do power users have a more complete, accurate mental model of a product?
Can "training wheels" facilitate novices becoming power users over time?
Operators working in a flow state often take sub-optimal paths. Does "good design" optimize efficiency in these situations?
Is "good design" adequate for skilled performance in unpredictable situations?
Can power use of a product be simulated when experienced users are not available -- either before power users exist or when they cannot participate? The emphasis is on how we should train users prior to participation in testing.
Neilson, J., Mack, R. (1994) Usability Inspection Methods. John Wiley & Son, NY
Shneiderman, B. (1990) User interface races: Sporting competition for power users. In Art of Human-Computer Interface Design. B. Laurel (ed.) Addison Wesley, Reading MA
All workshop participants (Figure 2) contributed to the content of this article. The workshop organizers and editors of this article were:
Keith Karn is a senior user interface designer at Xerox Corporation. His background is in human-machine interaction design and usability evaluation of airplane cockpits and office products. He has an M.S. in industrial engineering and Ph.D. in experimental psychology.
Tom Perry is a senior user interface designer at Xerox Corporation. He is a Certified Professional Ergonomist with an M.A. in Human Factors Psychology from California State University, Northridge.
Marc Krolczyk is a user interface designer at Xerox Corporation. He has worked in the field of graphic design and received his M.A.H. from SUNY at Buffalo in visual communication.
These are the results for the root-cause analysis: a list of causes
underlying the problem of inadequate testing for power usability
and related potential solutions. The headings evolved as the
bones of the fishbone diagram (see text for details on the
root-cause analysis technique). Problems are marked with a
bullet, solutions with a bullet and a "!".
No tool / technique to select critical portions of interface for testing
Tests lag behind critical design decision points (results are too late to impact current release)
! Realize it may be necessary to delay solution till future release (i.e., leap-frogging may be necessary)
! Find a champion at a senior level
! Advertise successes to increase support
! Cosponsor tests with marketing organization
! Integrate usability tests into iterative design process
! Focus testing on a critical subset of tasks or features
! Reserve some resources for longitudinal studies
! Hire users to generate something truly needed by the program team during the usability test (e.g., documentation, tutorial examples, sample output, etc.)
! Use early and frequent designer / peer evaluations and task walk-throughs as an inexpensive alternative to full tests
How to measuring power use with low-fidelity prototypes?
Difficult to test novel solutions for power users
How to handle rich amount of data?
Hard to do field tests in real contexts
Context of use varies dramatically for a multinational product
Designers blind spots
Users may become more like designers with repetitive experience in usability assessment
When is a sample size large enough?
Difficulty motivating participants in simulated tasks
How to avoid over-simplifying the product complexity and get good data?
How to avoid "over training" users so learnability and conceptual model problems are found in testing.
! Design test to include measures of goals criteria
! Do phased testing - refine for 2nd, (3rd) passes
! Test release x to inform design of release x+1
! Try everything - a mix of methods is very important
! Try anything (as opposed to nothing) then improve it
! Use tools for study of real work in context: interviews, artifact capture, data logging, sessions logging, thinking aloud, audio journals, user diaries
! Continuous context and data collection with key customers
! Get real customer jobs and use in tests
! Include an "own work" phase (test subjects bring in their own work to do with new system)
! Highlight "outside the box" practices
! "Critical incident" software
! Start analysis immediately to get changes into next release (e.g., when study is of extended use)
! Products can include data analysis / logging capability
! Separate the roles to minimize blind spots: designer, trainer for test, test administrator
! User interface races (Shneiderman, 1990) to motivate skilled subjects
! Use designers as confederates in multi-user test contexts
! Hands-on training prior to test session
! Let user explore prior to testing
! Wait a week between training and test
! Gather anecdotal data during training
! Collect learnability data during power user training sessions
Figure 1. Workshop participants clustering related statements of problem causes onto the fishbone diagram
Figure 2. Workshop Participants posing with the completed fish-bone cause and effect diagram. Left to right: Keith Karn, Xerox Corporation, USA; Tim White, Eastman Kodak Company, USA; Jill Drury, The MITRE Corporation, USA; Marc Krolczyk, Xerox Corporation, USA; Vibeke Jorgensen, Kommunedata, Denmark; Anette Zobbe, Kommunedata, Denmark; Gerard Jorna, Philips Corporate Design, The Netherlands; Tom Perry, Xerox Corporation, USA; Julianne Chatelain, Trellix Corporation, USA; Ronald Baecker, University of Toronto, Canada; Jose Madeira Garcia, Autodesk, USA; Kaisa Vaananen-Vainio-Mattila, Nokia, Finland; Stephanie Rosenbaum, Tec-Ed, Inc., USA; (not pictured: Judith Rattle, Philips Corporate Design, The Netherlands).
